| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-26 | |||
| 15:04:48 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: tests: Add reproducer for bug #1889633 https://review.opendev.org/748254 | |
| 15:05:20 | noonedeadpunk | I mean like `os_compute_api:servers:create:zero_disk_flavor` is already checking the flavor. Maybe somewhere near it's possible to check if flavor has zero size disk, but has no volume attached? | |
| 15:05:42 | noonedeadpunk | I'm not really familiar with nova codebase as it's so huge... | |
| 15:06:32 | sean-k-mooney | you proably need the block device mappings | |
| 15:07:17 | sean-k-mooney | that said i think we have a bfv flag somewhere | |
| 15:08:18 | noonedeadpunk | At least I hope I told userstory I'm trying to implement in an understandable way.... | |
| 15:08:44 | sean-k-mooney | you did | |
| 15:08:57 | sean-k-mooney | you want to restict who can boot a vm without bfv | |
| 15:09:04 | sean-k-mooney | using policy and roles | |
| 15:09:08 | sean-k-mooney | right | |
| 15:10:14 | noonedeadpunk | ideally yes. How to restrict it on whole cloud I guess you've already answered and it's with max_local_block_devices | |
| 15:10:26 | noonedeadpunk | eventually that will probably work for me as well | |
| 15:11:14 | sean-k-mooney | yes the config option will also block it much latere on the compute host rather then api | |
| 15:12:20 | noonedeadpunk | (in the worst case scenario would use isolate aggregates which will be also tricky because of the need to separate resources) | |
| 15:12:44 | noonedeadpunk | but yeah, checking that on api level might be more efficient | |
| 15:12:55 | noonedeadpunk | and produce more understandable output | |
| 15:14:04 | sean-k-mooney | an api check woudl allow us to retrun a specific api respoce saying why | |
| 15:14:11 | sean-k-mooney | currently it would jsut get a no valid host | |
| 15:14:19 | noonedeadpunk | yeah | |
| 15:14:24 | sean-k-mooney | which is not a great user experice | |
| 15:14:48 | gmann | zero_disk_flavor policy already control the zero disk boot if it is not bfv | |
| 15:15:28 | noonedeadpunk | zero_disk_flavor eventually can only prohibit spawning a VM when it's created with flavor with 0 disk | |
| 15:15:29 | sean-k-mooney | oh so the edgecase it did not fix was allowing bfz when disk >0 | |
| 15:15:46 | sean-k-mooney | there is an edgecase that it breaks | |
| 15:15:47 | noonedeadpunk | even if its bfv | |
| 15:17:12 | gmann | https://github.com/openstack/nova/blob/eef4b5435e7cdfe53ee9d9265d96c7dd278d9e93/nova/compute/api.py#L730 | |
| 15:17:51 | sean-k-mooney | ya htat is blocking using 0 disk flavor without bfv | |
| 15:17:58 | noonedeadpunk | I think it's pretty different | |
| 15:18:00 | noonedeadpunk | yeah | |
| 15:18:09 | sean-k-mooney | it does not prevent createing vms with non 0 disk flavors | |
| 15:18:25 | noonedeadpunk | eventually... it blocks 0 disk flavor with bfv as well. At least blocked on train for me... | |
| 15:18:44 | sean-k-mooney | it should not | |
| 15:19:03 | gmann | yeah, it should not block that | |
| 15:19:04 | sean-k-mooney | that check should only block non bfv vms with 0 disk flavors | |
| 15:19:24 | noonedeadpunk | oh, yeah, maybe you're right | |
| 15:19:27 | noonedeadpunk | sorry | |
| 15:20:07 | noonedeadpunk | actually, even when instance has 0 disk flavor it can be still created without bfv | |
| 15:20:19 | noonedeadpunk | which also can't be covered with that policy | |
| 15:20:19 | sean-k-mooney | not with that policy | |
| 15:20:32 | sean-k-mooney | that is what it block | |
| 15:20:41 | sean-k-mooney | but on train it might now be enabeld by default | |
| 15:20:57 | noonedeadpunk | oh, indeed, yeah, you're right | |
| 15:21:08 | sean-k-mooney | its relitivly new | |
| 15:21:16 | noonedeadpunk | so yeah, edgecase when flavor >0 | |
| 15:21:41 | noonedeadpunk | I just got it overriden after upgrade to train, so mixing a bit, sorry | |
| 15:21:49 | gmann | it is there before train and with admin only. before stein it was default with admin-or-owner | |
| 15:22:57 | noonedeadpunk | so like another option is to drop all non-0 flavors and use that policy | |
| 15:22:59 | gmann | ok so the case left here is to allow flavor.disk>0 in bfv case ? | |
| 15:23:17 | sean-k-mooney | noonedeadpunk: yes | |
| 15:23:38 | sean-k-mooney | noonedeadpunk: and create private flavor with disk if you need that and only give access to sepcific tenants | |
| 15:23:42 | gmann | or instead of drop, control the flavor access | |
| 15:24:19 | sean-k-mooney | the flavor access i tenatnt based rather then role based is the only wrinkel with that | |
| 15:24:28 | sean-k-mooney | but it would work | |
| 15:25:28 | noonedeadpunk | ok, thank you very much for your time. You gave me 2 options which is more than enough at this time for me:) | |
| 15:27:11 | noonedeadpunk | I just really thought that I had issues with creating even with volumes and 0 disk flavors when had os_compute_api:servers:create:zero_disk_flavor locked for admin | |
| 15:27:42 | sean-k-mooney | if you can repoduce it simply then file a bug if you hit it | |
| 15:27:44 | sean-k-mooney | as that should work | |
| 15:30:20 | noonedeadpunk | yeah, sure! thanks again! | |
| 15:53:39 | rmart04 | thanks @sean-k-mooney, graphing this in grafana now, still unsure what the best approach is here though to stop OOMs. Was StephenFin saying that huge pages may help me over come this? | |
| 15:58:57 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/ussuri: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/747189 | |
| 16:05:13 | bauzas | stephenfin: gave you a terrible -1 https://review.opendev.org/#/c/747744/2 | |
| 16:05:50 | stephenfin | rmart04: I was suggesting explicit page size allocations would help | |
| 16:06:00 | bauzas | stephenfin: I know I'm a mean person | |
| 16:06:04 | stephenfin | because then you can do per-node reserved memory amounts | |
| 16:06:43 | stephenfin | e.g. 'hw:mem_page_size=small' | |
| 16:07:52 | sean-k-mooney | that will force a singel numa node unless you also use hw:numa_nodes=X | |
| 16:08:34 | sean-k-mooney | rmart04: do you allow ram oversubscription | |
| 16:08:49 | sean-k-mooney | rmart04: and are you using train or newer? | |
| 16:09:00 | rmart04 | We're using Stein and no RAM oversubscription | |
| 16:09:11 | sean-k-mooney | you are using pinning yes | |
| 16:09:13 | rmart04 | Yes | |
| 16:09:23 | sean-k-mooney | ok so in that case you should use hugepages | |
| 16:09:48 | sean-k-mooney | using explcit small pages could help but there hugepages will improve performance | |
| 16:09:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent starting services with older than N-1 computes https://review.opendev.org/738482 | |
| 16:10:06 | sean-k-mooney | you are already using pinning so you cannot live migrete | |
| 16:10:15 | sean-k-mooney | adding hugepages wont make that worse in this case | |
| 16:10:30 | rmart04 | ok JohnGarbut also suggested this to me recently, i'll look into it and see how i get on. Does the host need any explicit configuration or is it just flavor metadata specs? | |
| 16:10:53 | sean-k-mooney | rmart04: the host needs to have memroy preaccloated as hugepages | |
| 16:11:02 | sean-k-mooney | you can do tha that on the kernel commandline | |
| 16:11:53 | sean-k-mooney | you have 768 GB and are running a 750GB guest right | |
| 16:12:06 | sean-k-mooney | so you would allocate 750GB of 1GB hugepages ideally | |
| 16:12:23 | sean-k-mooney | then set hw:mem_page_size=large in the flavor | |
| 16:12:58 | sean-k-mooney | default_hugepagesz=1G hugepagesz=1G hugepages=750 | |
| 16:13:10 | sean-k-mooney | that is what you need to add to the kernel command line in grub | |
| 16:13:47 | sean-k-mooney | you have only 1 numa node on the host currently right? | |
| 16:13:47 | rmart04 | ah OK amazing, thank you | |
| 16:13:53 | rmart04 | No two | |
| 16:14:14 | sean-k-mooney | and the vm has how many numa nodes? 2 | |
| 16:14:22 | rmart04 | yes | |
| 16:14:27 | sean-k-mooney | ok then that makes sense | |
| 16:14:37 | sean-k-mooney | otherwise i was wondering how the vm booted :) | |
| 16:15:03 | rmart04 | I managed to get over that hurdle :P | |
| 16:15:17 | sean-k-mooney | rmart04: you should see >20% incresse form hugepages in most workloads | |
| 16:15:27 | sean-k-mooney | more if its memory or io bound | |
| 16:15:35 | sean-k-mooney | less if it compute bound | |
| 16:15:41 | sean-k-mooney | or disk bound | |
| 16:17:50 | rmart04 | ok great, thanks for all your help | |
| 16:18:38 | sean-k-mooney | by the way the reason i ask about oversubsrcitip is hugepage cant be swapped out ot disk but they also will never be reap as a result of OOM | |
| 16:18:56 | sean-k-mooney | so it shoudl solve your issue although there is a very small change that the qemu process can still be selected | |
| 16:20:00 | sean-k-mooney | qemu will be using relitvly little non hugepage memeory in this config so it will have a low memery pressur score vs other process and memeory will be balance evenly across both host numa nodes for the vm | |