| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-29 | |||
| 18:17:30 | gmaan | that is admin only | |
| 18:17:41 | sean-k-mooney | is it? | |
| 18:17:48 | sean-k-mooney | ok but its valid | |
| 18:17:58 | gmaan | I do not think manager will know host/info from any API | |
| 18:18:09 | sean-k-mooney | we need too decide how mcuh awareness of this info is ok for the manager role to have | |
| 18:18:26 | gmaan | yeah | |
| 18:18:30 | sean-k-mooney | right i dont think they should eitehr | |
| 18:18:56 | gmaan | that is evacuate is little tricky because it depends on service availability checks | |
| 18:19:08 | sean-k-mooney | yep | |
| 18:19:50 | gmaan | now, I do not have any strong opinion to open it for manager at least until someone have use case and ask us to do | |
| 18:19:52 | sean-k-mooney | by the way even if diagnostics was admin only console log show or hardreboot | |
| 18:20:11 | sean-k-mooney | can all indreictlly be used to prob if the instance is still managable | |
| 18:21:00 | gmaan | reboot is member-or-admin https://github.com/openstack/nova/blob/221a3e89e8988bc664298106ee691a4e41ca71f9/nova/policies/servers.py#L376 | |
| 18:21:38 | gmaan | console log too https://github.com/openstack/nova/blob/221a3e89e8988bc664298106ee691a4e41ca71f9/nova/policies/console_output.py#L27 | |
| 18:22:02 | sean-k-mooney | i expected https://docs.openstack.org/api-ref/compute/#show-console-output-os-getconsoleoutput-action to be reader | |
| 18:23:02 | sean-k-mooney | gmaan: im less concered with the service up vs leaking hostname/fqdn | |
| 18:23:23 | sean-k-mooney | we could defer the evacuate change for now or see what gibi and other thing before updating the spec | |
| 18:23:34 | gmaan | yeah, sounds good. | |
| 18:24:48 | sean-k-mooney | the calculus im doing is i dont know waht it gets you if you knwo nova-compute on a host is not up in general. the fqdn can tell you a lot about a host if your a bad actor | |
| 18:25:20 | sean-k-mooney | public cloud still may not want to expose that but since noone will have the manager role by default | |
| 18:25:33 | gmaan | yeah | |
| 18:25:34 | sean-k-mooney | its kind of a decison they can make | |
| 20:51:25 | opendevreview | Merged openstack/nova-specs master: Add remove-os-volumes_boot-api spec https://review.opendev.org/c/openstack/nova-specs/+/947210 | |
| 21:13:09 | opendevreview | Merged openstack/nova master: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/947541 | |
| 23:52:59 | opendevreview | Merged openstack/nova master: api: Adjust validation helpers for a single-method future https://review.opendev.org/c/openstack/nova/+/936365 | |
| #openstack-nova - 2025-05-30 | |||
| 01:09:50 | opendevreview | Takashi Kajinami proposed openstack/nova master: Replace license classifier https://review.opendev.org/c/openstack/nova/+/951226 | |
| 01:12:57 | opendevreview | sean mooney proposed openstack/nova-specs master: Add cpu-teiring with cpu_shares spec https://review.opendev.org/c/openstack/nova-specs/+/951222 | |
| 01:13:31 | opendevreview | sean mooney proposed openstack/nova-specs master: Add cpu-teiring with cpu_shares spec https://review.opendev.org/c/openstack/nova-specs/+/951222 | |
| 07:03:26 | gibi | gmaan: sean-k-mooney: I replied about evacuate in https://review.opendev.org/c/openstack/nova-specs/+/937650/comment/aaf66b8d_53e159c3/ | |
| 08:58:27 | zigo | PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' | |
| 08:58:27 | zigo | Under Trixie, with RabbitMQ 4.0.5, I'm getting: | |
| 08:58:38 | zigo | Is this a miss-feature somewhere ?!? | |
| 08:58:48 | zigo | I mean, in oslo.messaging ... | |
| 10:28:39 | sean-k-mooney | zigo: not that im aware of but i belive kolla hit some issues with rabbit 4.x and i belive they also use debain | |
| 10:30:26 | sean-k-mooney | zigo: all our testing for expoy and master is still useing 3.12.1 | |
| 10:30:53 | sean-k-mooney | zigo: that what ubuntu 24.04 is shipping currently | |
| 10:34:51 | sean-k-mooney | zigo: im not sure if oslo.messaging offically supprot rabbit 4.X | |
| 10:35:35 | sean-k-mooney | you may be able to work around the issue by changign the cofnig default in service like chaging to quorm_queues | |
| 10:36:53 | sean-k-mooney | zigo: i dont know if the idea of having a debian testing image has ever been raised with infra or what the eta is for trixie. we currently use debain for spice testign in the nova-ovs-hybrid-plug job | |
| 10:37:11 | sean-k-mooney | i would be happy to see that move to trixe or sid in the future once its aviable | |
| 10:37:34 | sean-k-mooney | although the main blocker is likely to be python 3.13 support | |
| 10:39:08 | frickler | so far we would have to wait for trixie to be released to build an image with nodepool. with the zuul-launcher feature we might be able to start earlier, but I didn't have time yet to look into this | |
| 10:40:52 | frickler | also, yes, the version of oslo.messaging will likely be relevant. I think this should all be working in kolla, but maybe our CI coverage is incomplete | |
| 10:44:26 | sean-k-mooney | frickler: im wondering if the debian installer and kolla are using slightly diffent config options for the messaging transport | |
| 10:45:04 | sean-k-mooney | frickler: for example ha queue have been deprecated for a long time they may have finally killed them in 4? | |
| 10:46:44 | frickler | sean-k-mooney: yes, there was a lot of cleanup work to be done in the last cycle, but kolla does ship with rmq 4 for 2025.1 | |
| 10:47:49 | sean-k-mooney | that at least promising as it implies there is a workable config that zigo could move too | |
| 10:56:00 | zigo | sean-k-mooney: I did move to quorum queues, no choice with RabbitMQ 4.0.5, as it doesn't support HA queues anymore. Though that's not the issue. The issue is nova attempting to set not-durable, which seems to be gone from Rabbit. | |
| 10:57:19 | zigo | I wonder how can I tell oslo.messaging not to do that. | |
| 10:57:30 | sean-k-mooney | zigo: have you treied setting the durable_queses config option to true | |
| 10:57:33 | zigo | I did grep not-durable in oslo.messaging and didn't find where it is. | |
| 10:57:54 | sean-k-mooney | oslo is ment to ignore it when using quorm_queues which requrie durablity | |
| 10:57:57 | zigo | sean-k-mooney: This creates more issues that it solves, I wouldn't recommend to anyone to do that. | |
| 10:58:30 | sean-k-mooney | https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/amqp.py#L33-L37 | |
| 10:58:55 | sean-k-mooney | zigo: quourm queue are unconditonally durable in rabbit | |
| 10:59:05 | zigo | Oh ok. | |
| 10:59:15 | sean-k-mooney | https://www.rabbitmq.com/docs/quorum-queues#feature-matrix | |
| 10:59:19 | zigo | Let me try something then. | |
| 11:00:02 | sean-k-mooney | so oslo should be ignoreing that config option but it may be bugged or it may be fixed but in a newer verion | |
| 11:00:35 | zigo | I'll try setting durable_queue to true and see how it goes. | |
| 11:00:50 | zigo | Though as much as I can tell, this doesn't fix. | |
| 11:00:55 | zigo | Let me try. | |
| 11:01:49 | zigo | Same thing. | |
| 11:01:57 | zigo | Queue.declare: (406) PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' | |
| 11:01:57 | zigo | I'm getting: | |
| 11:02:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: trying to figure out what broke the unit test https://review.opendev.org/c/openstack/nova/+/951393 | |
| 11:06:36 | opendevreview | Balazs Gibizer proposed openstack/nova master: [hacking] N374 do not use time.sleep(0) to yield https://review.opendev.org/c/openstack/nova/+/950992 | |
| 11:08:14 | zigo | sean-k-mooney: Any idea what's attempting to add non-durable and how one does that in Rabbit? | |
| 11:11:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cache [pci]alias parsing https://review.opendev.org/c/openstack/nova/+/951151 | |
| 11:44:13 | sean-k-mooney | zigo: not off the top of my head sorry | |
| 11:56:06 | frickler | zigo: did you set rabbit_quorum_queue and rabbit_transient_quorum_queue? | |
| 11:56:21 | zigo | I did. | |
| 11:56:29 | zigo | Both to true. | |
| 11:57:16 | zigo | then daemons car start ... | |
| 11:57:16 | zigo | https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L447 | |
| 11:57:16 | zigo | frickler: If I comment this line: | |
| 11:57:46 | frickler | for reference here's the full template that kolla uses, but maybe some rmq config is also relevant https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/aodh/templates/aodh.conf.j2#L63-L78 | |
| 11:58:35 | zigo | frickler: And this has been tested with rabbitmq >= 4 ? | |
| 11:59:34 | frickler | yes. are you doing a fresh deploy or an upgrade? | |
| 11:59:44 | zigo | Fresh deploy. | |
| 11:59:48 | zigo | Using Trixie + Epoxy. | |
| 12:00:10 | zigo | Epoxy on Bookworm works, though I'm trying to finish my setup on Trixie, where many things just break ... :P | |
| 12:00:19 | frickler | https://opendev.org/openstack/kolla/src/branch/master/docker/rabbitmq/apt_preferences_rabbitmq.debian | |
| 12:01:10 | zigo | The pinning for erlang is wrong, Trixie has 27.3.3. | |
| 12:01:50 | frickler | well that is from erlang/rmq upstream repos, not debian pkgs | |
| 12:02:12 | zigo | Ah... ok ... | |
| 12:15:44 | zigo | Got some same kind of issues with Neutron. :/ | |
| 12:15:59 | zigo | oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on cl1-controller-2.infomaniak.ch:5671 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - invalid property 'non-durable' for queue 'q-plugin' in vhost '/' | |
| 12:22:40 | zigo | I'll redo a setup from scratch, I guess. | |
| 12:31:04 | sean-k-mooney | zigo: rabbit erqires all service on a given vhost to have the same queue config | |
| 12:31:41 | sean-k-mooney | so if nova is uing durable qurom ques and nova and neturon share a vhost then they would both have to align | |
| 12:31:56 | sean-k-mooney | with that said we do not recommend sharign vhosts between openstack services | |
| 12:32:03 | zigo | Of course ! :) | |
| 12:32:47 | zigo | I'll have a go setting durable=True everywhere, and see how that goes. | |
| 12:33:15 | sean-k-mooney | zigo: ya we tried to do that in our new installer and hit exactly the same thing it was all or nothing | |
| 12:34:02 | sean-k-mooney | apprently tripleo never used per service vhosts... and when rabbit supprot was added for our our new installer they said well if it was good enough for tripleo why change it... | |
| 12:34:34 | sean-k-mooney | it makes me sad inside as that is the most basic form of isolation that you should alwasy do ... | |
| 12:35:41 | sean-k-mooney | gibi: melwitt: just an fyi https://bugs.launchpad.net/nova/+bug/2112187 it looke like live migration after a cinder swap volume may be broken | |
| 12:36:15 | sean-k-mooney | gibi: melwitt: this was found via our watcher testing im currently poking around in the env to see if i can find anyting useful | |
| 12:38:58 | sean-k-mooney | its reproducable as well. we have not tried doign it by hand but this is the 3 time we have repoduced this in 2 difent envs | |