| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-30 | |||
| 13:00:01 | sean-k-mooney | zigo:anyway thanks for looking at this while on vacation but please dont spend too much time on it | |
| 13:00:41 | zigo | No worries. | |
| 13:01:35 | zigo | If I put the novnc package somewhere, can you test it? | |
| 13:01:47 | zigo | I did test 1.6.0 already... | |
| 13:02:00 | zigo | Worked, though had to fix some CSS. | |
| 13:02:21 | sean-k-mooney | frickler: ^ would you be able to test it in the test env you had set up | |
| 13:02:58 | frickler | I've now installed novnc from source already, better setup a new host I guess | |
| 13:03:00 | sean-k-mooney | zigo: i could proably install it into a devstack too i just dont have one set up for this at the moment | |
| 13:03:30 | zigo | http://shade.infomaniak.ch/novnc_1.6.0-2_all.deb | |
| 13:03:37 | frickler | (that is a held node, not a local env) | |
| 13:03:42 | zigo | It's ok, I'll just upload like that ... | |
| 13:05:34 | zigo | uploaded | |
| 13:06:05 | zigo | To get the fixed novnc in Ubuntu, you may have to ping some Canonical guys ... jamespage (in here) for example! :) | |
| 13:06:31 | sean-k-mooney | ack, it would be nice to fix in 24.04 | |
| 13:06:41 | zigo | jamespage: Could you get novnc 1:1.6.0-2 pushed to Ubuntu? | |
| 13:07:04 | zigo | sean-k-mooney: Not sure they will push a new upstream release to an already released Ubuntu though, but maybe in their PPA. | |
| 13:07:21 | sean-k-mooney | ya we have a workaround anywya | |
| 13:07:26 | sean-k-mooney | which is just install form source | |
| 13:07:30 | sean-k-mooney | and it only affects jobs | |
| 13:07:35 | sean-k-mooney | that are using unreleased oslo | |
| 13:07:42 | sean-k-mooney | which is a very small minority | |
| 13:07:48 | sean-k-mooney | so we are likey ok if they cant | |
| 13:08:27 | sean-k-mooney | we could also jsut swap to debian :) | |
| 13:11:33 | gibi | bauzas: blogpost with more data https://gibizer.github.io/posts/Eventlet-Removal-Scheduler-First-Run/ | |
| 13:13:07 | bauzas | ta a lot | |
| 13:23:03 | opendevreview | LiuWenYu proposed openstack/nova master: Implements: Fixed the issue where the code logic would throw an error when reading a variable after NUMA node 0’s CPUs were fully utilized. Related-Bug: #2109675 https://review.opendev.org/c/openstack/nova/+/948540 | |
| 13:24:48 | zigo | sean-k-mooney: By all means, please get jobs running on Debian 13 and Py 3.13 ! :) | |
| 13:25:59 | sean-k-mooney | zigo: we moved nova-hybrid-plug to debian 12 to test with spice | |
| 13:26:29 | sean-k-mooney | zigo: so we will proably move that to debain 13 once thats avaiable provided spice is not compiled out | |
| 13:33:56 | frickler | zigo: fyi jamespage is no longer with canonical | |
| 13:34:14 | zigo | Oh, I didn't know. | |
| 13:34:44 | frickler | that's why I mention it ;) happened pretty recently | |
| 13:35:45 | zigo | Who's in charge of OpenStack stuff now at Canonical then? | |
| 13:36:35 | frickler | I'm not sure, I think all the contacts I knew earlier are now gone | |
| 13:40:13 | zigo | Only me stays ... forever ! :) | |
| 14:58:55 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 14:58:55 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 16:20:15 | sean-k-mooney | bauzas: so you can just install mdevctl on ubuntu | |
| 16:20:25 | sean-k-mooney | its not there by default but it should be packaged | |
| 16:20:35 | bauzas | melwitt: fun stuff, ubuntu noble doesn't have mdevctl https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_92e/openstack/92e4f13448c94210af9f36f145e4a58c/controller/logs/screen-n-cpu.txt | |
| 16:20:38 | sean-k-mooney | https://manpages.ubuntu.com/manpages/focal/man8/mdevctl.8.html | |
| 16:20:52 | sean-k-mooney | bauzas: it does its just not instlaled by defualt | |
| 16:20:56 | bauzas | sean-k-mooney: yeah, we need to install it in nova-next | |
| 16:21:13 | sean-k-mooney | you could add it to bindep in the test profile | |
| 16:21:17 | bauzas | but honestly, I hate that silent exception | |
| 16:21:20 | sean-k-mooney | to keep it optional | |
| 16:21:54 | sean-k-mooney | bauzas: well the better fix would be to fall back to the old way of not using libvirt persitnace if mdevctl is not there | |
| 16:22:05 | bauzas | sean-k-mooney: I honestly don't know how to do it, examples would be welcomed | |
| 16:22:28 | sean-k-mooney | sure just add the package to https://github.com/openstack/nova/blob/master/bindep.txt | |
| 16:22:28 | bauzas | sean-k-mooney: yup, we should failback on a libvirt exception | |
| 16:22:48 | bauzas | I know what bindep is, I don't know how to make it optional | |
| 16:22:56 | sean-k-mooney | [platform:dpkg test] | |
| 16:23:04 | sean-k-mooney | install on debing but only for the test profile | |
| 16:23:11 | sean-k-mooney | we use test in devstack | |
| 16:23:21 | bauzas | k | |
| 16:23:30 | sean-k-mooney | we could jsut list it as required | |
| 16:23:46 | sean-k-mooney | i.e. without test | |
| 16:23:58 | sean-k-mooney | if we want to always use the new persitance api | |
| 16:24:04 | bauzas | that's what we do | |
| 16:24:21 | bauzas | anyway, I'll quickly add it to bindep before I go off | |
| 16:26:03 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 16:26:04 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 17:03:29 | haleyb | zigo: regarding Canonical contact, i would start with Billy (wolsen) who isn't usually on irc, but you could try me as a proxy, i meet with them every Monday | |
| 17:38:18 | EugenMayer440180 | after my instance stopped restart properly, i restarted the cluster. Now i see that the docker container nova-compute did not start (restarted the controller/computes entirely), with the message '3 RLock(s) were not greened, to fix this error make sure you run eventlet.monkey_patch() before importing any other modules.' | |
| 17:41:20 | EugenMayer440180 | it seems like the controller shows all the VM 'active' and running, but they are not. Seems like the state cannot be updated | |
| 17:49:35 | EugenMayer440180 | the actual issue seems to be 'My compute node 9946da6f-9a3d-45c7-a9e3-e7e8f99811fc has hypervisor_hostname compute1.cluster.kontextwork.net but virt driver reports it should be compute1. Possible rename detected, refusing to start!' | |
| 17:54:22 | EugenMayer440180 | we recently had issues with 2024.2 with hostname != hostname -f and thus we changed that the computes nor longer have a FQDN like 'compute1.cluster.kontextwork.net' but rather compute1 | |
| 17:55:01 | EugenMayer440180 | this seems to be an issue now. Is there any way to re-register the computes with the controller using the new hostname without loosing any data? | |
| 18:13:32 | dansmith | EugenMayer440180: you cannot change the hostname of the computes, ever, for any reason | |
| 18:24:09 | EugenMayer440180 | well i just did. | |
| 18:29:34 | EugenMayer440180 | changed it in the database, restarted the computes and everything is up and running again | |
| 18:30:30 | sean-k-mooney | EugenMayer440180: the problem is the hostname is also in teh cinder and neutron dbs in several tables | |
| 18:31:01 | EugenMayer440180 | not using cinder, so this one i dodged. Neutron and the network agents already think about compute1 | |
| 18:31:02 | sean-k-mooney | you would have to consitently change it in 10-20 tables | |
| 18:31:37 | EugenMayer440180 | at least when quering the network agents and so forth, they all use the hostname already (or lets say the hostname which is the new FQDN) | |
| 18:31:43 | sean-k-mooney | changing the hypervior_hostname will break all of the placement allcoations too | |
| 18:32:56 | sean-k-mooney | neutron port bindign if they are out of sync can be fixed by detachign the ports and reattachign them or live migration/shleve/resize | |
| 18:33:37 | sean-k-mooney | EugenMayer440180: the thing you hit preventign the agent form starting was likely https://specs.openstack.org/openstack/nova-specs/specs/2023.1/implemented/stable-compute-uuid.html | |
| 18:34:02 | sean-k-mooney | after years fo saying renaming compute nodes is not supproted we finally started enforcing it | |
| 18:34:18 | EugenMayer440180 | checked the allocations tables in nova/nova_api/nova_cell0, they are all empty | |
| 18:34:30 | EugenMayer440180 | sean-k-mooney i did not choose to rename those | |
| 18:34:31 | sean-k-mooney | that not where they are stored | |
| 18:34:43 | sean-k-mooney | placement has its own db | |
| 18:35:46 | sean-k-mooney | EugenMayer440180: has the value of hypervior_hostname changed in the db? if so that is the name used for the compute_node resouce provider in placement | |
| 18:37:14 | EugenMayer440180 | found them there, as you expected. Will try to fix them there too. Thanks for the hint | |
| 18:41:07 | EugenMayer440180 | sean-k-mooney i understand that i tapped into the internals. But there was no other choice for me here and i will clean up the mess that this created one by one until we migrate away i assume. | |
| 18:43:33 | EugenMayer440180 | again, so i could e.g. restore instances from glance again. What i did not notice then is that this then broke the hypervisors / nova_libvirt (which happened now) | |
| 18:43:33 | EugenMayer440180 | The reason i changed the FQDN was the migration to 2024.2 where nova stopped scheduling instances since in my case, the compute nodes always had the 'hostname' (compute1) while the hypervisors had the FQDN (hostname -f) ... it's nothing i have chosen and worked since not sure, 5 years - but then stopped here. Then i had to change the FQDN work | |
| 18:44:48 | EugenMayer440180 | i was told then 'openstack does not like when 'hostname != hostname -f' and this can lead to issues ... which it did with 2024. Kind of suprising that something of this size cannot handle what is a usual setup for me since decades. | |
| 18:45:10 | sean-k-mooney | we have not chnage how the logic in nova works | |
| 18:45:21 | sean-k-mooney | you do not have to have host and hypervior_hostname match | |
| 18:45:30 | sean-k-mooney | they can be differnt they jsut can change over time | |
| 18:45:39 | sean-k-mooney | hypervior hostname comes form libvirt | |
| 18:45:49 | sean-k-mooney | you can see what it returns with virish hostname | |
| 18:46:00 | EugenMayer440180 | well not sure who broke it, might have been neutron, since it was all about the chassis name | |
| 18:46:05 | sean-k-mooney | the host value comes form the config and defautl to socket.gethostname)( | |
| 18:46:09 | EugenMayer440180 | 'due to no OVN chassis for host: compute3' | |
| 18:46:50 | sean-k-mooney | neutron have ways to cofnigure that via data in the ovs db in the case of ovn or config options for ml2/ovs | |
| 18:47:06 | EugenMayer440180 | with 2024.2 the chassis name (which was compute1 and so forth in my case) was no longer how nova expected (since what nova expected was compute1.cluster...) - this was the change that broke it | |
| 18:48:02 | sean-k-mooney | the value we pass for nova to neutron in the port biding is the valuce form conf.host | |