Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-12-18
08:37:33 opendevreview Sylvain Bauza proposed openstack/nova-specs master: Proposes mdev live-migration support in libvirt https://review.opendev.org/c/openstack/nova-specs/+/900636
12:38:57 opendevreview sean mooney proposed openstack/nova master: add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015
13:24:55 opendevreview sean mooney proposed openstack/nova master: add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015
13:24:55 opendevreview sean mooney proposed openstack/nova master: add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844
13:24:56 opendevreview sean mooney proposed openstack/nova master: add healthcheck tracker to nova context https://review.opendev.org/c/openstack/nova/+/829468
13:39:50 opendevreview sean mooney proposed openstack/nova master: add healthcheck utils and constants https://review.opendev.org/c/openstack/nova/+/829469
13:39:51 opendevreview sean mooney proposed openstack/nova master: add healthcheck endpoint to proxy commands https://review.opendev.org/c/openstack/nova/+/830703
14:02:44 bauzas sean-k-mooney: gibi: fwiw, I'm about to start writing the series for the mdev live-migration RFE
14:04:55 gibi bauzas: good luck!
14:11:17 opendevreview Artom Lifshitz proposed openstack/nova master: Allow live migrate paused instance when post copy is enabled https://review.opendev.org/c/openstack/nova/+/444517
14:38:35 artom melwitt, sean-k-mooney, gibi, btw, I updated https://review.opendev.org/c/openstack/nova/+/883682/ to hopefully reflect what was agreed on in the review comments
14:38:48 artom So now there's the notifications best effort patch on top: https://review.opendev.org/c/openstack/nova/+/903807/3
15:07:29 gibi artom: thanks. Both looks good to me
15:08:28 sean-k-mooney artom: ack ill look at it shortly
15:09:30 artom Thank _you_!
16:29:02 gibi sean-k-mooney: if you have a moment later, then I verified https://review.opendev.org/c/openstack/nova/+/444517 and it solves the live-migration of paused instance when post-copy is configured in nova.conf.
16:29:13 gibi artom: ^^ \o/
16:33:14 kashyap gibi: Also the "double live migration of a paused instance" thing - did you see that bz too?
16:33:30 kashyap Although, why would one want to run a double LM, though
16:34:11 artom gibi, ~~o~~
16:34:19 kashyap "If a VM is in paused, and it live-migrated twice, it is lost"
16:34:21 kashyap https://bugs.launchpad.net/nova/+bug/1947725
16:34:31 kashyap "Lost"?
16:35:04 artom Here instance instance, here boy!
16:46:47 gibi kashyap: I can try
16:55:33 gibi kashyap: nope, that but is still reproducible https://paste.opendev.org/show/bswQVIMJddVkrUtC1Pk0/
16:56:55 gibi kashyap: but it make sens as the related qemu bug is also open https://gitlab.com/qemu-project/qemu/-/issues/686
16:58:49 kashyap gibi: Ohh, I missed the attached QEMU bug
17:06:17 artom sean-k-mooney, so, tracing our snapshot upload call, we actually call it with a file handler for image_file/data
17:07:00 artom https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L3182
17:07:09 artom https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L3176
17:09:00 artom The thing is, on the sender side (Nova) passing a file handler to Python requests results in a streaming upload: https://docs.python-requests.org/en/latest/user/advanced.html#streaming-uploads
17:09:33 artom For chunks we have to use a generator or iterator: https://docs.python-requests.org/en/latest/user/advanced.html#chunk-encoded-requests
17:10:44 sean-k-mooney ack so we likely need a [glance]uplod_type=streaming|chunked config option
17:11:11 sean-k-mooney default to streamign since that is ths current behavior and allow opting into chunked
17:11:19 sean-k-mooney and we can consider changing the default in the future
17:20:36 sean-k-mooney melwitt: bauzas gibi can ye look at https://review.opendev.org/c/openstack/nova/+/903530/2 and the patch below to unblock mypy in teh requiremetns job for stephenfin
17:22:00 sean-k-mooney seperatly this https://review.opendev.org/c/openstack/nova/+/897218/2 and the 3 patches below it will finish the codespell and sphinx-lint series
17:22:35 sean-k-mooney if we can get the mypy codespell and sphinx-lint serise merged this week it would be great
17:27:06 sean-k-mooney gibi: how did you test https://review.opendev.org/c/openstack/nova/+/444517 in the openstack-k8s-operators CI
17:27:11 sean-k-mooney did you submit a pr?
17:27:28 sean-k-mooney i know they were working on makign the conent provide supprot buildign with upstream patches
17:27:34 sean-k-mooney btu i didnt think that was done yet
17:28:21 melwitt sean-k-mooney: sure, I can look (if others don't get to it first)
17:29:01 opendevreview Artom Lifshitz proposed openstack/nova master: POC: attempting glance chunked uploads https://review.opendev.org/c/openstack/nova/+/903611
17:42:13 gibi sean-k-mooney: sorry I was missleading I applied the patch locally
17:42:27 sean-k-mooney oh ok
17:42:40 sean-k-mooney that is something we will be able to do eventually
17:42:43 sean-k-mooney with a depends on
17:42:55 sean-k-mooney but not for a few months yet
17:43:40 sean-k-mooney its on the todolist and somethign ill be supprotign with rhex-ci too eventually
18:14:54 melwitt sean-k-mooney: hrm.. I notice that artom's patch is going to fail CI for a guest kernel panic 🫤 and it looks to be using the split image
18:14:54 melwitt sean-k-mooney: hrm.. I notice that artom's patch is going to fail CI for a guest kernel panic 🫤 and it looks to be using the split image
18:15:26 artom :(
18:17:41 opendevreview Artom Lifshitz proposed openstack/nova master: POC: attempting glance chunked uploads https://review.opendev.org/c/openstack/nova/+/903611
18:18:22 sean-k-mooney melwitt: i guess we can reviwe the logs when it finishes
18:18:35 melwitt and it seems like it's often (always?) this volume backed server resize test(s). so I wonder if there's something about it that's different than everything else. I'll see if I can find anything
18:18:41 sean-k-mooney is that in nova-next where we are not using split image or one of the other jobs
18:19:02 melwitt no it's in tempest-integrated-compute-rbac-old-defaultsf
18:19:08 sean-k-mooney well the volume test obviouly has some pci hotplug events
18:19:36 sean-k-mooney so maybe we can mitigate that by adding hw_disk_bus=scisi
18:19:40 melwitt logs are at https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb
18:19:53 melwitt hm ok
18:20:10 sean-k-mooney i.e. chagne from virtio-blk to virtio-scisi as that will not be a pci hotplu and instead be a scsi attach/detach
18:20:37 melwitt gotcha
18:21:34 sean-k-mooney the sttack trace looks like its in the page fault handeler
18:21:45 sean-k-mooney so this looks like the same failure we saw before
18:22:02 sean-k-mooney i wonder if this is bfv
18:22:11 sean-k-mooney and if we replace the image in that case
18:23:14 melwitt it is bfv
18:23:58 sean-k-mooney ok so we shoudl double check that tempest used the correct cirros image in that case
18:25:18 sean-k-mooney its calling into the share create_server https://github.com/openstack/tempest/blob/ab3686d28d2728001e3bd2fd543575087bf00137/tempest/api/compute/servers/test_server_actions.py#L491
18:25:56 sean-k-mooney which end up using https://github.com/openstack/tempest/blob/ab3686d28d2728001e3bd2fd543575087bf00137/tempest/common/compute.py#L204
18:26:01 sean-k-mooney image_id = CONF.compute.image_ref
18:26:20 sean-k-mooney melwitt: have we confirued that to be the split image?
18:26:50 sean-k-mooney its https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb/log/controller/logs/tempest_conf.txt#24
18:27:00 sean-k-mooney 86a0308f-80a0-4582-8a64-f2b54e487232
18:27:17 sean-k-mooney i guess we coudl check the devstack log and conrim what image that is
18:27:55 sean-k-mooney https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb/log/controller/logs/devstacklog.txt#15755-15773
18:28:07 sean-k-mooney ok so that is the uec image
18:29:54 sean-k-mooney so ya my best suggestion is lets try adding hw_disk_bus=scsi in devstack adn see if that helps or disabling https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.libvirt_disable_apic
18:31:05 melwitt ok, I can try that
18:32:12 sean-k-mooney thinking about this its failign in the really early boot
18:32:26 sean-k-mooney so im not sure that will help
18:32:51 sean-k-mooney the other thing we coudl tyr is adding swap to the tempest flavors to see if that help with the OOM issues
18:34:47 melwitt ah right
19:17:25 artom sean-k-mooney, bleah, glanceclient chunks it for us: https://opendev.org/openstack/python-glanceclient/src/branch/master/glanceclient/common/http.py#L110
19:17:57 sean-k-mooney ack streaming might still be good to enable
19:17:59 artom Which... I guess I learned stuff today, but then I'm no closer to understanding why g-api OOM'ed :P
19:18:24 sean-k-mooney ya not sure
19:59:31 opendevreview Merged openstack/nova master: Allow live migrate paused instance when post copy is enabled https://review.opendev.org/c/openstack/nova/+/444517
21:20:30 opendevreview Merged openstack/nova master: [codespell] fix final typos and enable ci https://review.opendev.org/c/openstack/nova/+/897214
21:20:38 opendevreview Merged openstack/nova master: Bump hacking version https://review.opendev.org/c/openstack/nova/+/903529
22:22:52 JayF sean-k-mooney: https://blueprints.launchpad.net/nova/+spec/ironic-guest-metadata is updated; if you have a preliminary review before tomorrow's meeting I'm happy to do a round trip of feedback ahead of it :)
23:05:14 opendevreview Jay Faulkner proposed openstack/nova master: [ironic] Partition & use cache for list_instance* https://review.opendev.org/c/openstack/nova/+/900831
23:05:15 opendevreview Jay Faulkner proposed openstack/nova master: Limit nodes by ironic shard key https://review.opendev.org/c/openstack/nova/+/903915
23:05:15 opendevreview Jay Faulkner proposed openstack/nova master: Add nova-manage ironic-compute-node-move https://review.opendev.org/c/openstack/nova/+/903916
23:05:16 opendevreview Jay Faulkner proposed openstack/nova master: Make compute node rebalance safter https://review.opendev.org/c/openstack/nova/+/903917
23:29:35 opendevreview Merged openstack/nova master: [codespell] ignore codespell in git blame https://review.opendev.org/c/openstack/nova/+/897215
#openstack-nova - 2023-12-19
01:13:04 opendevreview melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313

Earlier   Later