| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-06-30 | |||
| 11:03:21 | sean-k-mooney | or conductor in some cases | |
| 11:06:03 | sahid | hum I did not tried on conductor... on source the only error I have is Live Migration failure: 'fa:....' | |
| 11:06:22 | sahid | Migration operation has aborted | |
| 11:06:39 | sean-k-mooney | aborted is not the same as failing. that likely just hit the timeout | |
| 11:09:30 | sahid | sean-k-mooney: I think the Live Migration failure failure is comming frist from libvirt driver | |
| 11:09:39 | sahid | then the exception is propagaded | |
| 11:10:22 | sean-k-mooney | it can if it failed on the migrate step | |
| 11:10:39 | sahid | let me see if I find from where is comming the message Migration operation has aborted | |
| 11:14:19 | sahid | the event is mentionning, compute_rollback_live_migration_at_destination and there is no log at destination | |
| 11:14:33 | sahid | we are missing to log some important details | |
| 11:23:51 | opendevreview | Merged openstack/nova stable/2024.1: Fix detaching devices by alias with mdevs https://review.opendev.org/c/openstack/nova/+/953662 | |
| 11:24:29 | sean-k-mooney | just an fyi i repliekd to mikal email just now, neutron broken debian 12 support with https://github.com/openstack/neutron/commit/8990dd598f84b9da976d72672c6fd6037fc0d125 it seams so the nova hybrid plug job is broken until they fix the coalaiton type | |
| 11:55:09 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace utils.spawn_n with spawn https://review.opendev.org/c/openstack/nova/+/948076 | |
| 11:55:09 | opendevreview | Kamil Sambor proposed openstack/nova master: Use futurist for _get_default_green_pool() https://review.opendev.org/c/openstack/nova/+/948072 | |
| 11:55:10 | opendevreview | Kamil Sambor proposed openstack/nova master: Move ComputeManager to use spawn_on https://review.opendev.org/c/openstack/nova/+/948186 | |
| 11:55:10 | opendevreview | Kamil Sambor proposed openstack/nova master: Add spawn_on https://review.opendev.org/c/openstack/nova/+/948079 | |
| 11:55:11 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754 | |
| 12:47:23 | fungi | [repeating myself from yesterday, sorry] it's come up a few times in debian over recent years that other cloud platforms don't rely on ahci driver access for cdrom devices, and so their minimal "cloud" linux kernel config doesn't include support for it, which means users have to manually override to virtio to make configdrive work with debian's official "genericcloud" images. | |
| 12:47:25 | fungi | is there a reason not to switch the default in nova to virtio, as suggested in https://bugs.debian.org/1108403 ? (i see zigo just pushed change 953732 for this too) | |
| 12:48:33 | zigo | I didn't push for this, because I was fearing it would be too much of an aggressive change, but I'd basically agree with fungi that it'd be nice to have virtio by default. | |
| 12:49:52 | opendevreview | Thomas Goirand proposed openstack/nova master: Add a [libvirt]/force_virtio_cd option https://review.opendev.org/c/openstack/nova/+/953732 | |
| 12:50:39 | zigo | My patch looks like passing the CI, it just had a POST_FAILURE for nova-ovs-hybrid-plug. | |
| 12:50:39 | zigo | fungi: Thanks for spotting the typo btw! :) | |
| 12:53:12 | fungi | it's also obvious to us, but perhaps not to the debian kernel maintainers, that nova changing its default now will only slowly trickle down to public cloud providers, as there are some still running their services on decade-old versions of openstack | |
| 12:54:06 | zigo | I just believe that Waldi only cares about his customer in this case (ie: Microsoft and Azure) that's not impacted by the missing AHCI driver. | |
| 12:54:22 | zigo | IMO, it's a conflict of interest case. | |
| 12:55:05 | zigo | Never the less, virtio-cd is much nicer than ide... | |
| 12:59:32 | sean-k-mooney | fungi: so we use sata by default when using q35 machine type or ide for pc | |
| 12:59:54 | sean-k-mooney | fungi: we cant use virtio-scis because of window window si belive | |
| 13:00:16 | sean-k-mooney | so we dont really have an alternitive choice | |
| 13:02:03 | sean-k-mooney | zigo: we only use ide for the pc machien type if you defautl to q35 we use sata as i noted above, virtio-cd either iddnt exist or was not a option when we last chagned the defautl 4+ years ago | |
| 13:02:27 | opendevreview | Takashi Kajinami proposed openstack/nova master: Use built-in declarative https://review.opendev.org/c/openstack/nova/+/953751 | |
| 13:02:42 | sean-k-mooney | we coudl consider if that is now viablebased on our min libvirt version although we may need to restrict that to only linux machine unless windows ships with it by default | |
| 13:03:01 | zigo | sean-k-mooney: Oh, so activating the feature I'm proposing would break windows setup, right? | |
| 13:03:17 | sean-k-mooney | maybe we dont supprot virtio-cd at all | |
| 13:03:49 | sean-k-mooney | so we woudl first need to supprot that and deterim if it can be the default or contintioal default absed on OS_TYPE on the galnce image | |
| 13:04:12 | sean-k-mooney | im late for a meeting but ill be free in an hour and can look into this more | |
| 13:04:47 | zigo | We do, it just need a specific metadata. | |
| 13:05:18 | zigo | As much as my colleague is telling me, this would *not* break our windows image that has the virtio CD support. | |
| 13:05:38 | zigo | hw_cdrom_bus=virtio | |
| 13:05:38 | zigo | the metadata is: | |
| 13:05:39 | sean-k-mooney | will it break a vanilla windows install iso | |
| 13:05:47 | sean-k-mooney | zigo: so you can set that | |
| 13:05:55 | zigo | *we* wouldn't care. | |
| 13:05:56 | sean-k-mooney | but there used to be problems with useing it | |
| 13:06:03 | sean-k-mooney | well nova would | |
| 13:06:39 | zigo | sean-k-mooney: Yeah, *we* can, but some idiot customers have no idea about it, will take system image snapshot or backup, attempt to boot, and see it fail because of the CD being ide and the lack of AHCI support in the image. | |
| 13:06:45 | zigo | I'd like to avoid this ... | |
| 13:07:37 | sean-k-mooney | so we really try to not make things dynmaic based on the image in a lot of cases but its worth a dicussion at least | |
| 13:07:47 | opendevreview | Takashi Kajinami proposed openstack/nova master: sqlalchemy: Use built-in declarative https://review.opendev.org/c/openstack/nova/+/953751 | |
| 13:07:57 | sean-k-mooney | zigo: i know there was a reason we could not use virtio as the default at the time | |
| 13:08:02 | sean-k-mooney | but i dont recall the details | |
| 13:08:10 | sean-k-mooney | it may have been our min libnvirt versin at the time | |
| 13:09:35 | sean-k-mooney | zigo: i assume you woudl prefer that we use virio by default both for pc and q35 machine type? | |
| 13:12:15 | sean-k-mooney | zigo: the last time we chagne defautl was yoga https://blueprints.launchpad.net/nova/+spec/virtio-as-default-display-device | |
| 13:12:24 | sean-k-mooney | so we have precendet for doing this | |
| 13:13:29 | opendevreview | Takashi Kajinami proposed openstack/placement master: sqlalchemy: Use built-in declarative https://review.opendev.org/c/openstack/placement/+/953759 | |
| 13:54:29 | stephenfin | sean-k-mooney: gibi: ralonsoh: I assume you're aware that nova-ovs-hybrid-plug is permafailing on master since last Friday? https://zuul.opendev.org/t/openstack/builds?job_name=nova-ovs-hybrid-plug&project=openstack/nova | |
| 13:54:46 | ralonsoh | let me check | |
| 13:55:08 | stephenfin | I took a look at things that merged to neutron and neutron-lib between the last pass and first fail but didn't spot anything obvious | |
| 13:55:38 | stephenfin | ralonsoh: neutron-api is failing because there's a name field being set on a model that doesn't have the relevant column | |
| 13:56:00 | stephenfin | https://zuul.opendev.org/t/openstack/build/79bd7dae1b954169a77159ca53b85422/log/controller/logs/screen-neutron-api.txt | |
| 13:56:34 | stephenfin | (specifically the networksegments table as you'll se) | |
| 13:56:35 | stephenfin | *see | |
| 13:56:49 | ralonsoh | let me check first the last patches merged | |
| 14:02:06 | ralonsoh | stephenfin, this is because a stupid guy merged a patch last friday | |
| 14:02:15 | ralonsoh | a very stupid guy and a very stupid patch | |
| 14:02:20 | gibi | sean-k-mooney: dansmith: could you add the back the +A to https://review.opendev.org/c/openstack/nova/+/948072/ it was lost in a rebase | |
| 14:03:08 | dansmith | done | |
| 14:03:19 | ralonsoh | stephenfin, I've pushed the fix: https://review.opendev.org/c/openstack/neutron/+/953768. Check that the Neutron db migration is failing: https://zuul.opendev.org/t/openstack/build/46c6d08486c3461dbc6f3fcb3c2b832d/log/job-output.txt#13491 | |
| 14:03:24 | stephenfin | ralonsoh: :D | |
| 14:03:26 | gibi | dansmith: thanks | |
| 14:04:27 | stephenfin | ahh | |
| 14:04:35 | stephenfin | I was looking in the wrong place | |
| 14:05:18 | stephenfin | ralonsoh: Looks like a bug in devstack too. init_neutron shouldn't be finishing with result 0 if the db_sync failed, right? https://zuul.opendev.org/t/openstack/build/46c6d08486c3461dbc6f3fcb3c2b832d/log/job-output.txt#13509 | |
| 14:05:40 | ralonsoh | right, that should have been stopped the stack process | |
| 14:09:50 | opendevreview | ribaudr proposed openstack/nova-specs master: Enable memfd support for shared memory backing https://review.opendev.org/c/openstack/nova-specs/+/951689 | |
| 14:11:45 | Uggla | sean-k-mooney, gibi, dansmith if you have time I'll be happy it you can have a look at ^ | |
| 14:41:26 | zigo | sean-k-mooney: I'd prefer if it was virtio-cd *always*, yes. | |
| 14:41:50 | sean-k-mooney | so if we were to make that change it would only take effect for new vms | |
| 14:42:15 | zigo | Yeah, I get that point. | |
| 14:42:23 | sean-k-mooney | when we made teh other changes we added code to recored the currnt model in the db | |
| 14:42:32 | sean-k-mooney | we did that so we could chagne the default in the future | |
| 14:43:31 | zigo | BTW. what is the use of rabbit_qos_prefetch_count? I've set use_queue_manager and rabbit_stream_fanout, but now nova-compute complains I haven't set rabbit_qos_prefetch_count and it is to its default of zero. What is a good value for this then ? | |
| 14:43:38 | sean-k-mooney | we bassically pretent the glance image properlty is set and add it in isntance sysmtem metaadat table as if it came form teh glance image when the vm is first created | |
| 14:44:07 | sean-k-mooney | zigo: no idea i dont think we sue that in nvoa today so its proably directly coming form oslo.messaging | |
| 14:44:55 | sean-k-mooney | zigo: we directly include the oslo.messaging cofnig options in our config because all the parsieng fo those exctra are done by oslo not nova | |
| 14:45:07 | zigo | Yeah, like everyone else does! :) | |
| 14:45:26 | sean-k-mooney | well excpet swift... | |
| 14:45:52 | zigo | Which is a real pain. | |
| 14:46:14 | zigo | We tried switching to json log to send all to Elasticsearch, it worked great ... except for swift ! :/ | |
| 14:46:22 | sean-k-mooney | so i am not sure what that does but you might get a better answer for the oslo folks | |
| 14:46:43 | sean-k-mooney | zigo: i both love and hate that that exists | |
| 14:46:51 | sean-k-mooney | its really nice for suecase like that | |
| 14:47:12 | sean-k-mooney | but when our custoerm enabel that and i have to ready the raw logs in json format it sucks | |
| 14:47:44 | sean-k-mooney | i wish there was a tool to take teh json format and print them normally in human reablae forrmat in oslo | |
| 14:47:46 | zigo | It was commited together with the fanout and stream stuff by Arnaud Morin <arnaud.morin@ovhcloud.com> | |
| 14:47:54 | zigo | https://review.opendev.org/c/openstack/oslo.messaging/+/890825 | |
| 14:48:00 | sean-k-mooney | ya so nova does not supprot any of that offically | |
| 14:48:01 | zigo | Though absolutely no doc about it ... :/ | |