| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-04 | |||
| 17:41:59 | stephenfin | so we need to somehow have two views into memory - one with NUMA context and another without it | |
| 17:42:19 | sean-k-mooney | so making all numa guest have hw:mem_page_size=any we trun on the memroy tracking | |
| 17:42:29 | stephenfin | and somehow still be able to get a useful "how much free memory do I have" answer for both NUMA and non-NUMA guests | |
| 17:42:31 | sean-k-mooney | it will use smallpages unless the image asks | |
| 17:42:46 | sean-k-mooney | we can then extend that to do over subsiption if we want | |
| 17:42:49 | stephenfin | that's doable though | |
| 17:43:33 | sean-k-mooney | claudiub: anyway sorry for the info overload | |
| 17:43:50 | claudiub | nono, thanks, this is useful. :) | |
| 17:43:56 | sean-k-mooney | claudiub: just being trying to fix this for 6+ years | |
| 17:43:59 | stephenfin | for example, in the NUMA case, free memory for NUMA N could be seen as (NUMA N total memory * overcommit) - (NUMA N used memory) - (non-NUMA used memory / NUMA count) | |
| 17:44:16 | stephenfin | i.e. just evenly divide the non-NUMA memory usage across all NUMA nodes | |
| 17:44:41 | sean-k-mooney | we cant do that | |
| 17:44:42 | stephenfin | my point being, there are other ways to solve this that don't involve the hw:mem_page_size=any hammer :) | |
| 17:45:03 | sean-k-mooney | that will cause oom issue | |
| 17:45:28 | sean-k-mooney | when a vm has a numa toplogy we tell teh kernel and restict its meory allcoations and the cores it can float over to the numa node | |
| 17:45:54 | stephenfin | by the memory isn't locked | |
| 17:45:58 | stephenfin | so it can be swapped out | |
| 17:46:07 | sean-k-mooney | it can be swapped yes | |
| 17:46:14 | sean-k-mooney | in the case of 4k pages at least | |
| 17:46:20 | stephenfin | and the memory for non-NUMA hosts will move across various NUMA nodes as needed | |
| 17:46:22 | sean-k-mooney | hugepages are not swapabel | |
| 17:46:36 | stephenfin | yeah, I'm only focusing on small pages for now | |
| 17:47:05 | sean-k-mooney | stephenfin: so there are 3 times we shoudl do. 1 numa blanceing, 2 hw:mem_page_size=any 3, make smallpages oversubscibale per numa node | |
| 17:47:15 | sean-k-mooney | well do 2 after 3 | |
| 17:47:39 | stephenfin | I'm still not sure why 2 is needed, but agreed on the other two | |
| 17:47:51 | sean-k-mooney | we do to turn on the memroy tracking | |
| 17:47:55 | stephenfin | I'll try to take a look at this the week after next | |
| 17:48:04 | sean-k-mooney | we dont do the claim otherwise in the hsot numna toplogy object | |
| 17:48:05 | stephenfin | since despite all this talk, I'm not going to start on it today :) | |
| 17:49:14 | sean-k-mooney | ok either way i think we need to disucss this in the ptg again and we can talk about it on irc again before | |
| 17:49:19 | stephenfin | agreed | |
| 17:49:31 | stephenfin | wanna add it to the agenda if you haven't already? | |
| 17:49:35 | stephenfin | if not I can | |
| 17:49:59 | sean-k-mooney | i didnt readded it since i was going to proceed based on what we agreed last ptg | |
| 17:50:13 | sean-k-mooney | i just didnt get time to wrok on this this cycle because of vdpa | |
| 17:50:24 | claudiub | I'll try the hw:numa_pages thing as soon as possible, since it's a bit of a burning issue for us. :) | |
| 17:51:22 | sean-k-mooney | claudiub: setting it to hw:mem_page_size=any or hw:mem_page_size=small should resolve your current issue | |
| 17:51:38 | sean-k-mooney | at the cost of oversubsciption of memory beign blocked | |
| 17:51:44 | lyarwood | stephenfin: do you have a link to you libvirt secure boot bug to hand? | |
| 17:52:01 | lyarwood | your* | |
| 17:52:13 | stephenfin | lyarwood: https://bugzilla.redhat.com/show_bug.cgi?id=1929357 | |
| 17:52:14 | openstack | bugzilla.redhat.com bug 1929357 in libvirt "UEFI: Provide a way how to configure different combinations of secure boot enabled/disabled and keys enrolled/not enrolled" [Medium,New] - Assigned to phrdina | |
| 17:52:25 | lyarwood | thanks | |
| 17:52:44 | sean-k-mooney | claudiub: it that is not an option for you and you can carry a downstream only patch you would add a random shuffle here https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/virt/hardware.py#L2276 of host_cells | |
| 17:54:41 | claudiub | will try both. :) | |
| 17:56:42 | sean-k-mooney | the balancing without over subctiption or randomisation with it likely could be backproted as a workaround bugfix. with proper feature in the future | |
| 18:24:45 | sean-k-mooney | stephenfin:related to the previous topic it look like libvirt is locking the guest memroy any time a guest has vfio(pci passthough/sriov), mdev or nvme devices assigned nova was previously not aware of that and them means many guests we previously assumed were swapable are not. i still have to file a bug for this but its emrging forlowing the memory locking disucssion we had for vdpa | |
| 18:26:29 | sean-k-mooney | im not sure how to fix that yet without requiring all guest with any kind of passthoug device to be numa guests or otherewise use hw:mem_page_size in some form | |
| 18:27:13 | sean-k-mooney | we can mitagate the problem for q35 guests by enabling the viommu | |
| 18:27:36 | sean-k-mooney | but i suspect this is the cause of many OOM bug that have been filed in the past | |
| 18:29:54 | sean-k-mooney | oh and more fun file backed memroy does not seam to work the way we tought either | |
| 18:30:36 | sean-k-mooney | i should do some more testing but usign it i was not actully able to allocate more vms then i had memory for without OOM issues killing the running vms | |
| 18:31:08 | sean-k-mooney | so it looks like instead of mmaping the guest memory form the files as the qemu/libvirt docs impleis | |
| 18:31:39 | sean-k-mooney | qemu just malloc the memroy normally and then also create a mapping of the memory to a file | |
| 18:32:22 | sean-k-mooney | that might be because we are usign the legacy api for this or it might be for a different reason but either way it makes me sad :( | |
| 18:49:25 | sean-k-mooney | actully i think i know why its broken if we want vms to only have file backed memory which is what we inteded we need to explitly set the normal memory to 0 i belive and then add the file using the memory hotplug feature so ya it likely a qemu bug in the old api. | |
| 19:43:05 | openstackgerrit | Merged openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/c/openstack/nova/+/673790 | |
| 20:17:26 | openstackgerrit | Merged openstack/nova master: nova-manage: Add libvirt get_machine_type command https://review.opendev.org/c/openstack/nova/+/769548 | |
| 23:03:14 | melwitt | lyarwood: could you pls take a look at these stable/victoria backports? they've disabled tests in tripleo ci to workaround intermittent failures due to the bug https://review.opendev.org/c/openstack/nova/+/777121 and https://review.opendev.org/c/openstack/nova/+/777209 | |
| 23:22:20 | openstackgerrit | Takashi Kajinami proposed openstack/nova master: WIP: Clean up allocations left by evacuation https://review.opendev.org/c/openstack/nova/+/778696 | |
| 23:51:50 | openstackgerrit | melanie witt proposed openstack/nova master: Add functional test for bug 1837995 https://review.opendev.org/c/openstack/nova/+/775449 | |
| 23:51:52 | openstack | bug 1837995 in OpenStack Compute (nova) ""Unexpected API Error" when use "openstack usage show" command" [Undecided,In progress] https://launchpad.net/bugs/1837995 - Assigned to melanie witt (melwitt) | |
| 23:51:52 | openstackgerrit | melanie witt proposed openstack/nova master: Dynamically archive FK related records in archive_deleted_rows https://review.opendev.org/c/openstack/nova/+/773834 | |
| #openstack-nova - 2021-03-05 | |||
| 00:06:21 | openstackgerrit | melanie witt proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/695012 | |
| 00:06:23 | openstack | bug 1853009 in OpenStack Compute (nova) ussuri "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard) | |
| 00:06:23 | openstackgerrit | melanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/695187 | |
| 00:06:23 | openstackgerrit | melanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/695188 | |
| 00:06:24 | openstackgerrit | melanie witt proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802 | |
| 00:06:24 | openstackgerrit | melanie witt proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189 | |
| 04:53:48 | openstackgerrit | Yongli He proposed openstack/nova master: Smartnic support - cyborg drive https://review.opendev.org/c/openstack/nova/+/771362 | |
| 04:53:49 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support - new vnic type https://review.opendev.org/c/openstack/nova/+/771363 | |
| 04:53:49 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 07:04:15 | openstackgerrit | Yongli He proposed openstack/nova master: Smartnic support - cyborg drive https://review.opendev.org/c/openstack/nova/+/771362 | |
| 07:04:16 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support - new vnic type https://review.opendev.org/c/openstack/nova/+/771363 | |
| 07:04:17 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 08:24:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 08:27:25 | lyarwood | elod: https://review.opendev.org/c/openstack/nova/+/777209/1 - would you mind hitting this and the regression test below it for melwitt please. | |
| 08:47:29 | elod | lyarwood: sure, looking | |
| 08:48:30 | yonglihe | alex_xu: gibi: test cases fixed, resolve concerns. | |
| 08:57:49 | lyarwood | elod: thanks | |
| 09:23:16 | bauzas | gibi: stephenfin: the whole api db series is now +Wd | |
| 09:23:26 | bauzas | but, | |
| 09:23:56 | bauzas | have we forgotten to add wallaby db placeholders like we did for Rocky ? :scared_emoji_you'd_like_have: | |
| 09:24:19 | stephenfin | Do we need to? | |
| 09:24:27 | stephenfin | We can just use the Victoria placeholders | |
| 09:24:39 | stephenfin | We don't have any DB migrations afaik | |
| 09:25:47 | bauzas | stephenfin: well, in general, we create DB placeholders even if we don't have existing DB things | |
| 09:25:52 | bauzas | https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions | |
| 09:26:03 | stephenfin | I know. I'm questioning that :) | |
| 09:26:54 | bauzas | stephenfin: the point is, if someone writes a DB migration in Xena and then we need to backport it in Wallaby but not in Victoria, how to do it ? | |
| 09:27:17 | bauzas | surely, it's a corner case, but that's why we have placeholders for every release | |
| 09:27:30 | stephenfin | Hmm, that's a good point | |
| 09:28:03 | stephenfin | I really doubt we'd want/need them but you can add them if you think it's a good idea? | |
| 09:28:09 | bauzas | I can try | |
| 09:28:17 | stephenfin | They'll all be deleted when we switch to alembic in Xena anyway | |
| 09:28:19 | bauzas | it's maybe just a bikeshed | |
| 09:28:35 | bauzas | but in case we have issues, then... | |
| 09:29:05 | bauzas | stephenfin: have we already agreed on accepting to use alembic for nova db migrations in Xena ? :D | |
| 09:29:22 | stephenfin | I'm going on the assumption that we have, yes :D | |
| 09:29:35 | openstackgerrit | Merged openstack/nova stable/victoria: Add regression test for bug 1914777 https://review.opendev.org/c/openstack/nova/+/777121 | |