| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-26 | |||
| 15:23:42 | gmann | or instead of drop, control the flavor access | |
| 15:24:19 | sean-k-mooney | the flavor access i tenatnt based rather then role based is the only wrinkel with that | |
| 15:24:28 | sean-k-mooney | but it would work | |
| 15:25:28 | noonedeadpunk | ok, thank you very much for your time. You gave me 2 options which is more than enough at this time for me:) | |
| 15:27:11 | noonedeadpunk | I just really thought that I had issues with creating even with volumes and 0 disk flavors when had os_compute_api:servers:create:zero_disk_flavor locked for admin | |
| 15:27:42 | sean-k-mooney | if you can repoduce it simply then file a bug if you hit it | |
| 15:27:44 | sean-k-mooney | as that should work | |
| 15:30:20 | noonedeadpunk | yeah, sure! thanks again! | |
| 15:53:39 | rmart04 | thanks @sean-k-mooney, graphing this in grafana now, still unsure what the best approach is here though to stop OOMs. Was StephenFin saying that huge pages may help me over come this? | |
| 15:58:57 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/ussuri: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/747189 | |
| 16:05:13 | bauzas | stephenfin: gave you a terrible -1 https://review.opendev.org/#/c/747744/2 | |
| 16:05:50 | stephenfin | rmart04: I was suggesting explicit page size allocations would help | |
| 16:06:00 | bauzas | stephenfin: I know I'm a mean person | |
| 16:06:04 | stephenfin | because then you can do per-node reserved memory amounts | |
| 16:06:43 | stephenfin | e.g. 'hw:mem_page_size=small' | |
| 16:07:52 | sean-k-mooney | that will force a singel numa node unless you also use hw:numa_nodes=X | |
| 16:08:34 | sean-k-mooney | rmart04: do you allow ram oversubscription | |
| 16:08:49 | sean-k-mooney | rmart04: and are you using train or newer? | |
| 16:09:00 | rmart04 | We're using Stein and no RAM oversubscription | |
| 16:09:11 | sean-k-mooney | you are using pinning yes | |
| 16:09:13 | rmart04 | Yes | |
| 16:09:23 | sean-k-mooney | ok so in that case you should use hugepages | |
| 16:09:48 | sean-k-mooney | using explcit small pages could help but there hugepages will improve performance | |
| 16:09:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent starting services with older than N-1 computes https://review.opendev.org/738482 | |
| 16:10:06 | sean-k-mooney | you are already using pinning so you cannot live migrete | |
| 16:10:15 | sean-k-mooney | adding hugepages wont make that worse in this case | |
| 16:10:30 | rmart04 | ok JohnGarbut also suggested this to me recently, i'll look into it and see how i get on. Does the host need any explicit configuration or is it just flavor metadata specs? | |
| 16:10:53 | sean-k-mooney | rmart04: the host needs to have memroy preaccloated as hugepages | |
| 16:11:02 | sean-k-mooney | you can do tha that on the kernel commandline | |
| 16:11:53 | sean-k-mooney | you have 768 GB and are running a 750GB guest right | |
| 16:12:06 | sean-k-mooney | so you would allocate 750GB of 1GB hugepages ideally | |
| 16:12:23 | sean-k-mooney | then set hw:mem_page_size=large in the flavor | |
| 16:12:58 | sean-k-mooney | default_hugepagesz=1G hugepagesz=1G hugepages=750 | |
| 16:13:10 | sean-k-mooney | that is what you need to add to the kernel command line in grub | |
| 16:13:47 | rmart04 | ah OK amazing, thank you | |
| 16:13:47 | sean-k-mooney | you have only 1 numa node on the host currently right? | |
| 16:13:53 | rmart04 | No two | |
| 16:14:14 | sean-k-mooney | and the vm has how many numa nodes? 2 | |
| 16:14:22 | rmart04 | yes | |
| 16:14:27 | sean-k-mooney | ok then that makes sense | |
| 16:14:37 | sean-k-mooney | otherwise i was wondering how the vm booted :) | |
| 16:15:03 | rmart04 | I managed to get over that hurdle :P | |
| 16:15:17 | sean-k-mooney | rmart04: you should see >20% incresse form hugepages in most workloads | |
| 16:15:27 | sean-k-mooney | more if its memory or io bound | |
| 16:15:35 | sean-k-mooney | less if it compute bound | |
| 16:15:41 | sean-k-mooney | or disk bound | |
| 16:17:50 | rmart04 | ok great, thanks for all your help | |
| 16:18:38 | sean-k-mooney | by the way the reason i ask about oversubsrcitip is hugepage cant be swapped out ot disk but they also will never be reap as a result of OOM | |
| 16:18:56 | sean-k-mooney | so it shoudl solve your issue although there is a very small change that the qemu process can still be selected | |
| 16:20:00 | sean-k-mooney | qemu will be using relitvly little non hugepage memeory in this config so it will have a low memery pressur score vs other process and memeory will be balance evenly across both host numa nodes for the vm | |
| 16:20:53 | yoctozepto | morning nova folks - today's question is when nova compute host name can get different from nova hypervisor host name and how | |
| 16:21:43 | sean-k-mooney | yoctozepto: it will be different for ironic | |
| 16:22:00 | yoctozepto | sean-k-mooney: true, let's focus on nova+kvm :-) | |
| 16:22:16 | sean-k-mooney | out side of that it shoudl be the same unless you set teh HOST config element or you add a fqdn to you /etc/hostname | |
| 16:22:17 | yoctozepto | actually evaluating noonedeadpunk's https://review.opendev.org/728629 (for masakari) | |
| 16:23:03 | stephenfin | yoctozepto: You can manually override '[DEFAULT] host' | |
| 16:23:09 | sean-k-mooney | yoctozepto: this is how its set https://github.com/openstack/nova/blob/master/nova/conf/netconf.py#L50-L70 | |
| 16:23:19 | stephenfin | it defaults to socket.gethostname() | |
| 16:23:24 | stephenfin | what sean-k-mooney just linked to | |
| 16:23:33 | sean-k-mooney | yep | |
| 16:23:42 | sean-k-mooney | so they should be the same unless you change that config option | |
| 16:24:04 | sean-k-mooney | if you use socket.fqdn() in your code it may not agreee | |
| 16:24:08 | sean-k-mooney | i fixed this in cyborg | |
| 16:24:14 | yoctozepto | then I would be changing the thing for nova-compute but not for the hypervisor, right? | |
| 16:24:32 | sean-k-mooney | yes | |
| 16:24:46 | sean-k-mooney | if you set [default]/host to something | |
| 16:25:03 | yoctozepto | ok, so where does the name for the hypervisor spawn and how's it kept linked? :D | |
| 16:25:03 | sean-k-mooney | hyperviour_hostname and host will not be the same for libvirt | |
| 16:25:08 | sean-k-mooney | or other virt dirvers | |
| 16:25:54 | sean-k-mooney | host is the compute service host and is set by that config and hypervisor_hostname is set by the virt driver | |
| 16:26:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Don't inherit from 'ProviderUsageBaseTestCase' https://review.opendev.org/748271 | |
| 16:26:13 | sean-k-mooney | the libvirt driver get it form libvirt | |
| 16:26:25 | sean-k-mooney | and it calls socket.gethostname internally in c | |
| 16:26:29 | stephenfin | bauzas: https://review.opendev.org/748271 | |
| 16:26:52 | sean-k-mooney | socket.gethostname is a fucntion provided by libc and pythons socket module jsut call the libc version | |
| 16:27:19 | sean-k-mooney | socket.gethostname() reads /etc/hostname if it exits on linux | |
| 16:27:23 | yoctozepto | sean-k-mooney: ah, so it's consulted each time with libvirt? I guess that's why I had a hard time figuring this one out | |
| 16:27:32 | sean-k-mooney | yes | |
| 16:27:37 | yoctozepto | yeah, the rest makes sense | |
| 16:27:40 | stephenfin | sean-k-mooney: https://review.opendev.org/#/c/748250/ | |
| 16:27:49 | yoctozepto | I do wonder how noonedeadpunk (and some others) ended up desynchronizing them | |
| 16:28:05 | bauzas | stephenfin: I don't get your comment, you're saying you can't just use the InstanceMixin ? | |
| 16:28:08 | noonedeadpunk | I have situnation where socket.gethostname and socket.getfqdn are different things | |
| 16:28:13 | bauzas | stephenfin: directly I mean ? | |
| 16:28:31 | yoctozepto | noonedeadpunk: hmm, ah | |
| 16:28:40 | yoctozepto | but then sean-k-mooney was only about gethostname | |
| 16:28:52 | yoctozepto | you mean libvirt is actually doing getfqdn? | |
| 16:28:52 | bauzas | stephenfin: of course, you'd have to specify the latest microversion but is that terrible to do ? | |
| 16:29:01 | stephenfin | bauzas: Not really. I'd have to copy-paste all of this https://github.com/openstack/nova/blob/master/nova/tests/functional/integrated_helpers.py#L1107-L1132 | |
| 16:29:10 | sean-k-mooney | yoctozepto: no its getting the hostname | |
| 16:29:23 | stephenfin | bauzas: Which seems worse than having functions available that I'm not using | |
| 16:29:51 | noonedeadpunk | sean-k-mooney: but I think that libvirt checks for the socket.getfqdn... | |
| 16:29:52 | sean-k-mooney | stephenfin: lol ok | |
| 16:29:54 | stephenfin | '_IntegratedTestBase' is basically what you're asking for but with all that boilerplate inline | |
| 16:29:59 | yoctozepto | then there is a conflict between your testimonies, noonedeadpunk and sean-k-mooney :-) | |
| 16:30:00 | sean-k-mooney | noonedeadpunk: it does not | |
| 16:30:05 | yoctozepto | investigation ongoing! | |
| 16:30:07 | yoctozepto | :D | |
| 16:30:12 | sean-k-mooney | noonedeadpunk: unless they changed it wich would be a breaking change | |
| 16:30:20 | stephenfin | there are some differences around policy but I'm resolving those in separate patches | |