Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-26
15:12:20 noonedeadpunk (in the worst case scenario would use isolate aggregates which will be also tricky because of the need to separate resources)
15:12:44 noonedeadpunk but yeah, checking that on api level might be more efficient
15:12:55 noonedeadpunk and produce more understandable output
15:14:04 sean-k-mooney an api check woudl allow us to retrun a specific api respoce saying why
15:14:11 sean-k-mooney currently it would jsut get a no valid host
15:14:19 noonedeadpunk yeah
15:14:24 sean-k-mooney which is not a great user experice
15:14:48 gmann zero_disk_flavor policy already control the zero disk boot if it is not bfv
15:15:28 noonedeadpunk zero_disk_flavor eventually can only prohibit spawning a VM when it's created with flavor with 0 disk
15:15:29 sean-k-mooney oh so the edgecase it did not fix was allowing bfz when disk >0
15:15:46 sean-k-mooney there is an edgecase that it breaks
15:15:47 noonedeadpunk even if its bfv
15:17:12 gmann https://github.com/openstack/nova/blob/eef4b5435e7cdfe53ee9d9265d96c7dd278d9e93/nova/compute/api.py#L730
15:17:51 sean-k-mooney ya htat is blocking using 0 disk flavor without bfv
15:17:58 noonedeadpunk I think it's pretty different
15:18:00 noonedeadpunk yeah
15:18:09 sean-k-mooney it does not prevent createing vms with non 0 disk flavors
15:18:25 noonedeadpunk eventually... it blocks 0 disk flavor with bfv as well. At least blocked on train for me...
15:18:44 sean-k-mooney it should not
15:19:03 gmann yeah, it should not block that
15:19:04 sean-k-mooney that check should only block non bfv vms with 0 disk flavors
15:19:24 noonedeadpunk oh, yeah, maybe you're right
15:19:27 noonedeadpunk sorry
15:20:07 noonedeadpunk actually, even when instance has 0 disk flavor it can be still created without bfv
15:20:19 noonedeadpunk which also can't be covered with that policy
15:20:19 sean-k-mooney not with that policy
15:20:32 sean-k-mooney that is what it block
15:20:41 sean-k-mooney but on train it might now be enabeld by default
15:20:57 noonedeadpunk oh, indeed, yeah, you're right
15:21:08 sean-k-mooney its relitivly new
15:21:16 noonedeadpunk so yeah, edgecase when flavor >0
15:21:41 noonedeadpunk I just got it overriden after upgrade to train, so mixing a bit, sorry
15:21:49 gmann it is there before train and with admin only. before stein it was default with admin-or-owner
15:22:57 noonedeadpunk so like another option is to drop all non-0 flavors and use that policy
15:22:59 gmann ok so the case left here is to allow flavor.disk>0 in bfv case ?
15:23:17 sean-k-mooney noonedeadpunk: yes
15:23:38 sean-k-mooney noonedeadpunk: and create private flavor with disk if you need that and only give access to sepcific tenants
15:23:42 gmann or instead of drop, control the flavor access
15:24:19 sean-k-mooney the flavor access i tenatnt based rather then role based is the only wrinkel with that
15:24:28 sean-k-mooney but it would work
15:25:28 noonedeadpunk ok, thank you very much for your time. You gave me 2 options which is more than enough at this time for me:)
15:27:11 noonedeadpunk I just really thought that I had issues with creating even with volumes and 0 disk flavors when had os_compute_api:servers:create:zero_disk_flavor locked for admin
15:27:42 sean-k-mooney if you can repoduce it simply then file a bug if you hit it
15:27:44 sean-k-mooney as that should work
15:30:20 noonedeadpunk yeah, sure! thanks again!
15:53:39 rmart04 thanks @sean-k-mooney, graphing this in grafana now, still unsure what the best approach is here though to stop OOMs. Was StephenFin saying that huge pages may help me over come this?
15:58:57 openstackgerrit Vlad Gusev proposed openstack/nova stable/ussuri: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/747189
16:05:13 bauzas stephenfin: gave you a terrible -1 https://review.opendev.org/#/c/747744/2
16:05:50 stephenfin rmart04: I was suggesting explicit page size allocations would help
16:06:00 bauzas stephenfin: I know I'm a mean person
16:06:04 stephenfin because then you can do per-node reserved memory amounts
16:06:43 stephenfin e.g. 'hw:mem_page_size=small'
16:07:52 sean-k-mooney that will force a singel numa node unless you also use hw:numa_nodes=X
16:08:34 sean-k-mooney rmart04: do you allow ram oversubscription
16:08:49 sean-k-mooney rmart04: and are you using train or newer?
16:09:00 rmart04 We're using Stein and no RAM oversubscription
16:09:11 sean-k-mooney you are using pinning yes
16:09:13 rmart04 Yes
16:09:23 sean-k-mooney ok so in that case you should use hugepages
16:09:48 sean-k-mooney using explcit small pages could help but there hugepages will improve performance
16:09:53 openstackgerrit Balazs Gibizer proposed openstack/nova master: Prevent starting services with older than N-1 computes https://review.opendev.org/738482
16:10:06 sean-k-mooney you are already using pinning so you cannot live migrete
16:10:15 sean-k-mooney adding hugepages wont make that worse in this case
16:10:30 rmart04 ok JohnGarbut also suggested this to me recently, i'll look into it and see how i get on. Does the host need any explicit configuration or is it just flavor metadata specs?
16:10:53 sean-k-mooney rmart04: the host needs to have memroy preaccloated as hugepages
16:11:02 sean-k-mooney you can do tha that on the kernel commandline
16:11:53 sean-k-mooney you have 768 GB and are running a 750GB guest right
16:12:06 sean-k-mooney so you would allocate 750GB of 1GB hugepages ideally
16:12:23 sean-k-mooney then set hw:mem_page_size=large in the flavor
16:12:58 sean-k-mooney default_hugepagesz=1G hugepagesz=1G hugepages=750
16:13:10 sean-k-mooney that is what you need to add to the kernel command line in grub
16:13:47 sean-k-mooney you have only 1 numa node on the host currently right?
16:13:47 rmart04 ah OK amazing, thank you
16:13:53 rmart04 No two
16:14:14 sean-k-mooney and the vm has how many numa nodes? 2
16:14:22 rmart04 yes
16:14:27 sean-k-mooney ok then that makes sense
16:14:37 sean-k-mooney otherwise i was wondering how the vm booted :)
16:15:03 rmart04 I managed to get over that hurdle :P
16:15:17 sean-k-mooney rmart04: you should see >20% incresse form hugepages in most workloads
16:15:27 sean-k-mooney more if its memory or io bound
16:15:35 sean-k-mooney less if it compute bound
16:15:41 sean-k-mooney or disk bound
16:17:50 rmart04 ok great, thanks for all your help
16:18:38 sean-k-mooney by the way the reason i ask about oversubsrcitip is hugepage cant be swapped out ot disk but they also will never be reap as a result of OOM
16:18:56 sean-k-mooney so it shoudl solve your issue although there is a very small change that the qemu process can still be selected
16:20:00 sean-k-mooney qemu will be using relitvly little non hugepage memeory in this config so it will have a low memery pressur score vs other process and memeory will be balance evenly across both host numa nodes for the vm
16:20:53 yoctozepto morning nova folks - today's question is when nova compute host name can get different from nova hypervisor host name and how
16:21:43 sean-k-mooney yoctozepto: it will be different for ironic
16:22:00 yoctozepto sean-k-mooney: true, let's focus on nova+kvm :-)
16:22:16 sean-k-mooney out side of that it shoudl be the same unless you set teh HOST config element or you add a fqdn to you /etc/hostname
16:22:17 yoctozepto actually evaluating noonedeadpunk's https://review.opendev.org/728629 (for masakari)
16:23:03 stephenfin yoctozepto: You can manually override '[DEFAULT] host'
16:23:09 sean-k-mooney yoctozepto: this is how its set https://github.com/openstack/nova/blob/master/nova/conf/netconf.py#L50-L70
16:23:19 stephenfin it defaults to socket.gethostname()
16:23:24 stephenfin what sean-k-mooney just linked to
16:23:33 sean-k-mooney yep
16:23:42 sean-k-mooney so they should be the same unless you change that config option
16:24:04 sean-k-mooney if you use socket.fqdn() in your code it may not agreee
16:24:08 sean-k-mooney i fixed this in cyborg

Earlier   Later