| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-29 | |||
| 18:12:36 | sean-k-mooney | if the host was up ya | |
| 18:12:41 | sean-k-mooney | it might be a 400 | |
| 18:12:51 | gmaan | yeah, that is something I was thikning why we did not return 409 in this case | |
| 18:12:58 | sean-k-mooney | i.e. it may be one of the api we missed when we treid to use 409 better | |
| 18:13:07 | gmaan | yeah | |
| 18:14:39 | sean-k-mooney | gmaan: is the concern that tehy will knwo the mangment of the host is still up? | |
| 18:15:01 | sean-k-mooney | like if they try to evacuate and the nova-compute is up | |
| 18:15:10 | gmaan | sean-k-mooney: that one + current code return instance' host in the error message which we can change | |
| 18:15:18 | sean-k-mooney | then they could just cold/live migratiate instead right? | |
| 18:15:53 | sean-k-mooney | gmaan: oh right i didnt look at the respocne | |
| 18:16:24 | gmaan | they can migrate other way but it is that they will have a way to know compute service is up | |
| 18:17:08 | gmaan | which they know indirectly because instance is up :) that is why I am not 100% sure if this is infra info leak or not | |
| 18:17:12 | sean-k-mooney | well they can infer but doint instnace actions on the instnace | |
| 18:17:18 | sean-k-mooney | like server diagnostics | |
| 18:17:30 | gmaan | that is admin only | |
| 18:17:41 | sean-k-mooney | is it? | |
| 18:17:48 | sean-k-mooney | ok but its valid | |
| 18:17:58 | gmaan | I do not think manager will know host/info from any API | |
| 18:18:09 | sean-k-mooney | we need too decide how mcuh awareness of this info is ok for the manager role to have | |
| 18:18:26 | gmaan | yeah | |
| 18:18:30 | sean-k-mooney | right i dont think they should eitehr | |
| 18:18:56 | gmaan | that is evacuate is little tricky because it depends on service availability checks | |
| 18:19:08 | sean-k-mooney | yep | |
| 18:19:50 | gmaan | now, I do not have any strong opinion to open it for manager at least until someone have use case and ask us to do | |
| 18:19:52 | sean-k-mooney | by the way even if diagnostics was admin only console log show or hardreboot | |
| 18:20:11 | sean-k-mooney | can all indreictlly be used to prob if the instance is still managable | |
| 18:21:00 | gmaan | reboot is member-or-admin https://github.com/openstack/nova/blob/221a3e89e8988bc664298106ee691a4e41ca71f9/nova/policies/servers.py#L376 | |
| 18:21:38 | gmaan | console log too https://github.com/openstack/nova/blob/221a3e89e8988bc664298106ee691a4e41ca71f9/nova/policies/console_output.py#L27 | |
| 18:22:02 | sean-k-mooney | i expected https://docs.openstack.org/api-ref/compute/#show-console-output-os-getconsoleoutput-action to be reader | |
| 18:23:02 | sean-k-mooney | gmaan: im less concered with the service up vs leaking hostname/fqdn | |
| 18:23:23 | sean-k-mooney | we could defer the evacuate change for now or see what gibi and other thing before updating the spec | |
| 18:23:34 | gmaan | yeah, sounds good. | |
| 18:24:48 | sean-k-mooney | the calculus im doing is i dont know waht it gets you if you knwo nova-compute on a host is not up in general. the fqdn can tell you a lot about a host if your a bad actor | |
| 18:25:20 | sean-k-mooney | public cloud still may not want to expose that but since noone will have the manager role by default | |
| 18:25:33 | gmaan | yeah | |
| 18:25:34 | sean-k-mooney | its kind of a decison they can make | |
| 20:51:25 | opendevreview | Merged openstack/nova-specs master: Add remove-os-volumes_boot-api spec https://review.opendev.org/c/openstack/nova-specs/+/947210 | |
| 21:13:09 | opendevreview | Merged openstack/nova master: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/947541 | |
| 23:52:59 | opendevreview | Merged openstack/nova master: api: Adjust validation helpers for a single-method future https://review.opendev.org/c/openstack/nova/+/936365 | |
| #openstack-nova - 2025-05-30 | |||
| 01:09:50 | opendevreview | Takashi Kajinami proposed openstack/nova master: Replace license classifier https://review.opendev.org/c/openstack/nova/+/951226 | |
| 01:12:57 | opendevreview | sean mooney proposed openstack/nova-specs master: Add cpu-teiring with cpu_shares spec https://review.opendev.org/c/openstack/nova-specs/+/951222 | |
| 01:13:31 | opendevreview | sean mooney proposed openstack/nova-specs master: Add cpu-teiring with cpu_shares spec https://review.opendev.org/c/openstack/nova-specs/+/951222 | |
| 07:03:26 | gibi | gmaan: sean-k-mooney: I replied about evacuate in https://review.opendev.org/c/openstack/nova-specs/+/937650/comment/aaf66b8d_53e159c3/ | |
| 08:58:27 | zigo | Under Trixie, with RabbitMQ 4.0.5, I'm getting: | |
| 08:58:27 | zigo | PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' | |
| 08:58:38 | zigo | Is this a miss-feature somewhere ?!? | |
| 08:58:48 | zigo | I mean, in oslo.messaging ... | |
| 10:28:39 | sean-k-mooney | zigo: not that im aware of but i belive kolla hit some issues with rabbit 4.x and i belive they also use debain | |
| 10:30:26 | sean-k-mooney | zigo: all our testing for expoy and master is still useing 3.12.1 | |
| 10:30:53 | sean-k-mooney | zigo: that what ubuntu 24.04 is shipping currently | |
| 10:34:51 | sean-k-mooney | zigo: im not sure if oslo.messaging offically supprot rabbit 4.X | |
| 10:35:35 | sean-k-mooney | you may be able to work around the issue by changign the cofnig default in service like chaging to quorm_queues | |
| 10:36:53 | sean-k-mooney | zigo: i dont know if the idea of having a debian testing image has ever been raised with infra or what the eta is for trixie. we currently use debain for spice testign in the nova-ovs-hybrid-plug job | |
| 10:37:11 | sean-k-mooney | i would be happy to see that move to trixe or sid in the future once its aviable | |
| 10:37:34 | sean-k-mooney | although the main blocker is likely to be python 3.13 support | |
| 10:39:08 | frickler | so far we would have to wait for trixie to be released to build an image with nodepool. with the zuul-launcher feature we might be able to start earlier, but I didn't have time yet to look into this | |
| 10:40:52 | frickler | also, yes, the version of oslo.messaging will likely be relevant. I think this should all be working in kolla, but maybe our CI coverage is incomplete | |
| 10:44:26 | sean-k-mooney | frickler: im wondering if the debian installer and kolla are using slightly diffent config options for the messaging transport | |
| 10:45:04 | sean-k-mooney | frickler: for example ha queue have been deprecated for a long time they may have finally killed them in 4? | |
| 10:46:44 | frickler | sean-k-mooney: yes, there was a lot of cleanup work to be done in the last cycle, but kolla does ship with rmq 4 for 2025.1 | |
| 10:47:49 | sean-k-mooney | that at least promising as it implies there is a workable config that zigo could move too | |
| 10:56:00 | zigo | sean-k-mooney: I did move to quorum queues, no choice with RabbitMQ 4.0.5, as it doesn't support HA queues anymore. Though that's not the issue. The issue is nova attempting to set not-durable, which seems to be gone from Rabbit. | |
| 10:57:19 | zigo | I wonder how can I tell oslo.messaging not to do that. | |
| 10:57:30 | sean-k-mooney | zigo: have you treied setting the durable_queses config option to true | |
| 10:57:33 | zigo | I did grep not-durable in oslo.messaging and didn't find where it is. | |
| 10:57:54 | sean-k-mooney | oslo is ment to ignore it when using quorm_queues which requrie durablity | |
| 10:57:57 | zigo | sean-k-mooney: This creates more issues that it solves, I wouldn't recommend to anyone to do that. | |
| 10:58:30 | sean-k-mooney | https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/amqp.py#L33-L37 | |
| 10:58:55 | sean-k-mooney | zigo: quourm queue are unconditonally durable in rabbit | |
| 10:59:05 | zigo | Oh ok. | |
| 10:59:15 | sean-k-mooney | https://www.rabbitmq.com/docs/quorum-queues#feature-matrix | |
| 10:59:19 | zigo | Let me try something then. | |
| 11:00:02 | sean-k-mooney | so oslo should be ignoreing that config option but it may be bugged or it may be fixed but in a newer verion | |
| 11:00:35 | zigo | I'll try setting durable_queue to true and see how it goes. | |
| 11:00:50 | zigo | Though as much as I can tell, this doesn't fix. | |
| 11:00:55 | zigo | Let me try. | |
| 11:01:49 | zigo | Same thing. | |
| 11:01:57 | zigo | I'm getting: | |
| 11:01:57 | zigo | Queue.declare: (406) PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' | |
| 11:02:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: trying to figure out what broke the unit test https://review.opendev.org/c/openstack/nova/+/951393 | |
| 11:06:36 | opendevreview | Balazs Gibizer proposed openstack/nova master: [hacking] N374 do not use time.sleep(0) to yield https://review.opendev.org/c/openstack/nova/+/950992 | |
| 11:08:14 | zigo | sean-k-mooney: Any idea what's attempting to add non-durable and how one does that in Rabbit? | |
| 11:11:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: Cache [pci]alias parsing https://review.opendev.org/c/openstack/nova/+/951151 | |
| 11:44:13 | sean-k-mooney | zigo: not off the top of my head sorry | |
| 11:56:06 | frickler | zigo: did you set rabbit_quorum_queue and rabbit_transient_quorum_queue? | |
| 11:56:21 | zigo | I did. | |
| 11:56:29 | zigo | Both to true. | |
| 11:57:16 | zigo | frickler: If I comment this line: | |
| 11:57:16 | zigo | https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L447 | |
| 11:57:16 | zigo | then daemons car start ... | |
| 11:57:46 | frickler | for reference here's the full template that kolla uses, but maybe some rmq config is also relevant https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/aodh/templates/aodh.conf.j2#L63-L78 | |
| 11:58:35 | zigo | frickler: And this has been tested with rabbitmq >= 4 ? | |
| 11:59:34 | frickler | yes. are you doing a fresh deploy or an upgrade? | |
| 11:59:44 | zigo | Fresh deploy. | |
| 11:59:48 | zigo | Using Trixie + Epoxy. | |
| 12:00:10 | zigo | Epoxy on Bookworm works, though I'm trying to finish my setup on Trixie, where many things just break ... :P | |
| 12:00:19 | frickler | https://opendev.org/openstack/kolla/src/branch/master/docker/rabbitmq/apt_preferences_rabbitmq.debian | |
| 12:01:10 | zigo | The pinning for erlang is wrong, Trixie has 27.3.3. | |
| 12:01:50 | frickler | well that is from erlang/rmq upstream repos, not debian pkgs | |
| 12:02:12 | zigo | Ah... ok ... | |