| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-12-18 | |||
| 16:34:31 | kashyap | "Lost"? | |
| 16:35:04 | artom | Here instance instance, here boy! | |
| 16:46:47 | gibi | kashyap: I can try | |
| 16:55:33 | gibi | kashyap: nope, that but is still reproducible https://paste.opendev.org/show/bswQVIMJddVkrUtC1Pk0/ | |
| 16:56:55 | gibi | kashyap: but it make sens as the related qemu bug is also open https://gitlab.com/qemu-project/qemu/-/issues/686 | |
| 16:58:49 | kashyap | gibi: Ohh, I missed the attached QEMU bug | |
| 17:06:17 | artom | sean-k-mooney, so, tracing our snapshot upload call, we actually call it with a file handler for image_file/data | |
| 17:07:00 | artom | https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L3182 | |
| 17:07:09 | artom | https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L3176 | |
| 17:09:00 | artom | The thing is, on the sender side (Nova) passing a file handler to Python requests results in a streaming upload: https://docs.python-requests.org/en/latest/user/advanced.html#streaming-uploads | |
| 17:09:33 | artom | For chunks we have to use a generator or iterator: https://docs.python-requests.org/en/latest/user/advanced.html#chunk-encoded-requests | |
| 17:10:44 | sean-k-mooney | ack so we likely need a [glance]uplod_type=streaming|chunked config option | |
| 17:11:11 | sean-k-mooney | default to streamign since that is ths current behavior and allow opting into chunked | |
| 17:11:19 | sean-k-mooney | and we can consider changing the default in the future | |
| 17:20:36 | sean-k-mooney | melwitt: bauzas gibi can ye look at https://review.opendev.org/c/openstack/nova/+/903530/2 and the patch below to unblock mypy in teh requiremetns job for stephenfin | |
| 17:22:00 | sean-k-mooney | seperatly this https://review.opendev.org/c/openstack/nova/+/897218/2 and the 3 patches below it will finish the codespell and sphinx-lint series | |
| 17:22:35 | sean-k-mooney | if we can get the mypy codespell and sphinx-lint serise merged this week it would be great | |
| 17:27:06 | sean-k-mooney | gibi: how did you test https://review.opendev.org/c/openstack/nova/+/444517 in the openstack-k8s-operators CI | |
| 17:27:11 | sean-k-mooney | did you submit a pr? | |
| 17:27:28 | sean-k-mooney | i know they were working on makign the conent provide supprot buildign with upstream patches | |
| 17:27:34 | sean-k-mooney | btu i didnt think that was done yet | |
| 17:28:21 | melwitt | sean-k-mooney: sure, I can look (if others don't get to it first) | |
| 17:29:01 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: attempting glance chunked uploads https://review.opendev.org/c/openstack/nova/+/903611 | |
| 17:42:13 | gibi | sean-k-mooney: sorry I was missleading I applied the patch locally | |
| 17:42:27 | sean-k-mooney | oh ok | |
| 17:42:40 | sean-k-mooney | that is something we will be able to do eventually | |
| 17:42:43 | sean-k-mooney | with a depends on | |
| 17:42:55 | sean-k-mooney | but not for a few months yet | |
| 17:43:40 | sean-k-mooney | its on the todolist and somethign ill be supprotign with rhex-ci too eventually | |
| 18:14:54 | melwitt | sean-k-mooney: hrm.. I notice that artom's patch is going to fail CI for a guest kernel panic 🫤 and it looks to be using the split image | |
| 18:14:54 | melwitt | sean-k-mooney: hrm.. I notice that artom's patch is going to fail CI for a guest kernel panic 🫤 and it looks to be using the split image | |
| 18:15:26 | artom | :( | |
| 18:17:41 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: attempting glance chunked uploads https://review.opendev.org/c/openstack/nova/+/903611 | |
| 18:18:22 | sean-k-mooney | melwitt: i guess we can reviwe the logs when it finishes | |
| 18:18:35 | melwitt | and it seems like it's often (always?) this volume backed server resize test(s). so I wonder if there's something about it that's different than everything else. I'll see if I can find anything | |
| 18:18:41 | sean-k-mooney | is that in nova-next where we are not using split image or one of the other jobs | |
| 18:19:02 | melwitt | no it's in tempest-integrated-compute-rbac-old-defaultsf | |
| 18:19:08 | sean-k-mooney | well the volume test obviouly has some pci hotplug events | |
| 18:19:36 | sean-k-mooney | so maybe we can mitigate that by adding hw_disk_bus=scisi | |
| 18:19:40 | melwitt | logs are at https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb | |
| 18:19:53 | melwitt | hm ok | |
| 18:20:10 | sean-k-mooney | i.e. chagne from virtio-blk to virtio-scisi as that will not be a pci hotplu and instead be a scsi attach/detach | |
| 18:20:37 | melwitt | gotcha | |
| 18:21:34 | sean-k-mooney | the sttack trace looks like its in the page fault handeler | |
| 18:21:45 | sean-k-mooney | so this looks like the same failure we saw before | |
| 18:22:02 | sean-k-mooney | i wonder if this is bfv | |
| 18:22:11 | sean-k-mooney | and if we replace the image in that case | |
| 18:23:14 | melwitt | it is bfv | |
| 18:23:58 | sean-k-mooney | ok so we shoudl double check that tempest used the correct cirros image in that case | |
| 18:25:18 | sean-k-mooney | its calling into the share create_server https://github.com/openstack/tempest/blob/ab3686d28d2728001e3bd2fd543575087bf00137/tempest/api/compute/servers/test_server_actions.py#L491 | |
| 18:25:56 | sean-k-mooney | which end up using https://github.com/openstack/tempest/blob/ab3686d28d2728001e3bd2fd543575087bf00137/tempest/common/compute.py#L204 | |
| 18:26:01 | sean-k-mooney | image_id = CONF.compute.image_ref | |
| 18:26:20 | sean-k-mooney | melwitt: have we confirued that to be the split image? | |
| 18:26:50 | sean-k-mooney | its https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb/log/controller/logs/tempest_conf.txt#24 | |
| 18:27:00 | sean-k-mooney | 86a0308f-80a0-4582-8a64-f2b54e487232 | |
| 18:27:17 | sean-k-mooney | i guess we coudl check the devstack log and conrim what image that is | |
| 18:27:55 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb/log/controller/logs/devstacklog.txt#15755-15773 | |
| 18:28:07 | sean-k-mooney | ok so that is the uec image | |
| 18:29:54 | sean-k-mooney | so ya my best suggestion is lets try adding hw_disk_bus=scsi in devstack adn see if that helps or disabling https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.libvirt_disable_apic | |
| 18:31:05 | melwitt | ok, I can try that | |
| 18:32:12 | sean-k-mooney | thinking about this its failign in the really early boot | |
| 18:32:26 | sean-k-mooney | so im not sure that will help | |
| 18:32:51 | sean-k-mooney | the other thing we coudl tyr is adding swap to the tempest flavors to see if that help with the OOM issues | |
| 18:34:47 | melwitt | ah right | |
| 19:17:25 | artom | sean-k-mooney, bleah, glanceclient chunks it for us: https://opendev.org/openstack/python-glanceclient/src/branch/master/glanceclient/common/http.py#L110 | |
| 19:17:57 | sean-k-mooney | ack streaming might still be good to enable | |
| 19:17:59 | artom | Which... I guess I learned stuff today, but then I'm no closer to understanding why g-api OOM'ed :P | |
| 19:18:24 | sean-k-mooney | ya not sure | |
| 19:59:31 | opendevreview | Merged openstack/nova master: Allow live migrate paused instance when post copy is enabled https://review.opendev.org/c/openstack/nova/+/444517 | |
| 21:20:30 | opendevreview | Merged openstack/nova master: [codespell] fix final typos and enable ci https://review.opendev.org/c/openstack/nova/+/897214 | |
| 21:20:38 | opendevreview | Merged openstack/nova master: Bump hacking version https://review.opendev.org/c/openstack/nova/+/903529 | |
| 22:22:52 | JayF | sean-k-mooney: https://blueprints.launchpad.net/nova/+spec/ironic-guest-metadata is updated; if you have a preliminary review before tomorrow's meeting I'm happy to do a round trip of feedback ahead of it :) | |
| 23:05:14 | opendevreview | Jay Faulkner proposed openstack/nova master: [ironic] Partition & use cache for list_instance* https://review.opendev.org/c/openstack/nova/+/900831 | |
| 23:05:15 | opendevreview | Jay Faulkner proposed openstack/nova master: Limit nodes by ironic shard key https://review.opendev.org/c/openstack/nova/+/903915 | |
| 23:05:15 | opendevreview | Jay Faulkner proposed openstack/nova master: Add nova-manage ironic-compute-node-move https://review.opendev.org/c/openstack/nova/+/903916 | |
| 23:05:16 | opendevreview | Jay Faulkner proposed openstack/nova master: Make compute node rebalance safter https://review.opendev.org/c/openstack/nova/+/903917 | |
| 23:29:35 | opendevreview | Merged openstack/nova master: [codespell] ignore codespell in git blame https://review.opendev.org/c/openstack/nova/+/897215 | |
| #openstack-nova - 2023-12-19 | |||
| 01:13:04 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313 | |
| 01:13:04 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 03:35:43 | opendevreview | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.opendev.org/c/openstack/nova/+/903427 | |
| 08:07:59 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.2: Allow live migrate paused instance when post copy is enabled https://review.opendev.org/c/openstack/nova/+/903834 | |
| 08:08:40 | gibi | sean-k-mooney: elodilles: a clean cherry pick backport ^^ | |
| 08:18:07 | opendevreview | Merged openstack/nova master: Fixed an error when caching multiple images in aggregate https://review.opendev.org/c/openstack/nova/+/894288 | |
| 10:40:58 | elodilles | gibi: +2'd | |
| 10:42:47 | gibi | thanks | |
| 10:43:02 | gibi | I will push it back to antelope as well when the bobcat one lands | |
| 10:43:44 | elodilles | ++ | |
| 11:42:12 | sean-k-mooney | gibi: you might as well bring it to zed to have it on all stable branches | |
| 11:58:01 | opendevreview | Takashi Kajinami proposed openstack/python-novaclient master: coveragerc: Fix wrong omitted directory https://review.opendev.org/c/openstack/python-novaclient/+/903960 | |
| 11:59:02 | opendevreview | Takashi Kajinami proposed openstack/os-vif master: coveragerc: Remove non-existent path https://review.opendev.org/c/openstack/os-vif/+/903961 | |
| 12:03:13 | gibi | sean-k-mooney: I can | |
| 12:04:52 | sean-k-mooney | for now thats going to be my default backport target until next it moves to unmaintened next cycle | |
| 12:07:43 | sean-k-mooney | ^ those two patches form takashi are trival and correct by the way if anyone wants some simple reviews | |
| 12:19:15 | gibi | done | |
| 13:03:16 | opendevreview | Elod Illes proposed openstack/nova stable/zed: Translate VF network capabilities to port binding https://review.opendev.org/c/openstack/nova/+/899229 | |
| 13:28:28 | opendevreview | Merged openstack/nova stable/2023.2: Allow live migrate paused instance when post copy is enabled https://review.opendev.org/c/openstack/nova/+/903834 | |
| 14:05:50 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.1: Allow live migrate paused instance when post copy is enabled https://review.opendev.org/c/openstack/nova/+/903840 | |
| 14:06:34 | gibi | elodilles: sean-k-mooney: ^^ moving it back to Antelope | |
| 14:54:40 | elodilles | +2'd | |
| 15:20:04 | opendevreview | Merged openstack/python-novaclient master: coveragerc: Fix wrong omitted directory https://review.opendev.org/c/openstack/python-novaclient/+/903960 | |