| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-20 | |||
| 16:57:06 | dansmith | it uses shelve on the backend | |
| 16:57:22 | dansmith | and it has to copy between databases, so it might be missing something | |
| 16:58:46 | gibi | The config drive associated with the server, if there is one, will be re-generated on the destination host in the target cell. Therefore if the server was created with personality files they will be lost. However, this is no worse than evacuating a server that had a config drive when the source and destination compute host are not on shared storage or when shelve offloading and unshelving a server | |
| 16:58:52 | gibi | with a config drive. If necessary, the resized server can be rebuilt to regain the personality files. | |
| 16:59:07 | gibi | so maybe such config drive regen is the problem | |
| 16:59:25 | gibi | (the above is from https://docs.openstack.org/nova/latest/admin/configuration/cross-cell-resize.html#limitations ) | |
| 16:59:59 | dansmith | ah yep | |
| 17:01:04 | gibi | the bug says it is lost form metadata response as well | |
| 17:01:29 | Uggla | yep I understand it is not only config drive. | |
| 17:01:49 | gibi | and it also points to `select keypairs from dst_cell.instance_extra ` I did not know we have a copy of a keypair there | |
| 17:02:01 | gibi | but it says it is in the source cell | |
| 17:02:09 | gibi | so that is probably not copied to the dest cell DB | |
| 17:02:16 | gibi | I vote that this is valid | |
| 17:02:47 | Uggla | I can try to reproduce, but I'm unsure how to easily deploy a multi cell env. | |
| 17:03:31 | gibi | we have multicell job we just need to see if we can enable some tempest there to catch this | |
| 17:04:59 | sean-k-mooney | thats in the sepc for what its porth | |
| 17:05:29 | sean-k-mooney | at least im 99% sure we called out that any files injected in the config drive would be lost with a cross cell resize | |
| 17:06:14 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/cross-cell-resize.html#known-issues | |
| 17:06:16 | sean-k-mooney | limiation 3 | |
| 17:06:30 | sean-k-mooney | Servers created with personality files, commonly known as file injection, that are resized across cells will lose the personality files since they are not persisted in the database | |
| 17:06:42 | gibi | sean-k-mooney: keypairs are not personality files | |
| 17:06:52 | gibi | afaik | |
| 17:06:56 | dansmith | correct | |
| 17:07:06 | dansmith | but they mentioned instance_extra | |
| 17:07:20 | dansmith | so if there's something we're not copying there and then regen'ing the configdrive... | |
| 17:07:44 | sean-k-mooney | oh it the keypari | |
| 17:07:55 | sean-k-mooney | ok that shoudl be copied and regenerated | |
| 17:08:13 | sean-k-mooney | i actully otught keyparis were in teh api db | |
| 17:08:33 | sean-k-mooney | not the cell db but i guess we embded them when we first create teh instance in the cell db? | |
| 17:08:40 | dansmith | keypairs themselves are I think (maybe) but the instance's attachment to them is in the main one of course | |
| 17:08:50 | sean-k-mooney | ack | |
| 17:09:49 | sean-k-mooney | im wondering if we could see this in ci. i think this is valid in anycase | |
| 17:10:12 | sean-k-mooney | we could create a repodcuer functional tests. in thoery tempest coudl detect this also | |
| 17:10:26 | sean-k-mooney | we just need to curl the metadata after the resize | |
| 17:10:32 | sean-k-mooney | or check the config drive i guess | |
| 17:10:49 | sean-k-mooney | presumable it workign now because the keypari is already in the authorized keys | |
| 17:11:24 | Uggla | sean-k-mooney, I'd like to try to do a reproducer. | |
| 17:11:30 | gibi | yepp we should be able to catch this in tempest | |
| 17:12:16 | sean-k-mooney | the workaround for now is just pass the keypari again on rebuild i guess | |
| 17:12:23 | Uggla | So if you all agree I will set this bug to valid and I'll try to do a reproducer. But I may need your help. | |
| 17:12:38 | sean-k-mooney | ack, sure shout if you hit issues | |
| 17:13:25 | Uggla | I will also update the bug with the workaround and remind limitation. | |
| 17:14:02 | Uggla | We are overtime, so I think we can wrap for today. | |
| 17:14:20 | sean-k-mooney | im not sure if we have multi cell functional tests today so that could be "fun" to make work | |
| 17:14:55 | sean-k-mooney | reproducing it in tempest will be simpler in this specific case. | |
| 17:15:10 | sean-k-mooney | oh we do https://github.com/openstack/nova/blob/master/nova/tests/functional/test_cross_cell_migrate.py | |
| 17:15:29 | dansmith | matt riedemann did that work, so I was *sure* there was functional testing for it :D | |
| 17:16:07 | sean-k-mooney | ack. getting diffent config to work in the fucntionla tests is still a bit fo a pain so i was unsure if we did or not | |
| 17:17:15 | Uggla | anything else ? | |
| 17:17:32 | Uggla | Thanks all and thanks for the extended time for bug triage. | |
| 17:17:37 | Uggla | #endmeeting | |
| 17:17:40 | opendevmeet | Meeting ended Tue May 20 17:17:40 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 17:17:40 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-05-20-16.01.txt | |
| 17:17:40 | Uggla | #endmeeting | |
| 17:17:40 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-05-20-16.01.html | |
| 17:17:40 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-05-20-16.01.log.html | |
| 17:18:31 | elodilles | thanks o/ | |
| 17:19:51 | masahito | thank | |
| 17:21:24 | gibi | Uggla: thanks | |
| 17:21:49 | Uggla | you are welcome. | |
| 17:24:11 | sean-k-mooney | Uggla: so https://github.com/openstack/nova/blob/master/nova/tests/functional/test_cross_cell_migrate.py#L816-L826 test_cold_migrate_target_host_in_other_cell | |
| 17:24:36 | sean-k-mooney | get you to the point where the vm has be moved to a differnt cell | |
| 17:24:48 | sean-k-mooney | you shoudl be able to use that as a baseline to create a repoducer | |
| 17:25:08 | Uggla | ok but it is a fn test right ? | |
| 17:25:39 | Uggla | You have said that a tempest one should be easier. | |
| 17:25:44 | Uggla | Am i wrong ? | |
| 17:26:07 | sean-k-mooney | that was assumign we did nto have functional test coverage for this which we do | |
| 17:26:27 | sean-k-mooney | so it should be pretty simple to extend since all the infra is in place | |
| 17:27:14 | Uggla | ok I will try to reproduce with the fn ones though. | |
| 17:27:44 | sean-k-mooney | _resize_and_validate accepts a server and returns server, source_rp_uuid, target_rp_uuid, old_flavor, new_flavor | |
| 17:27:59 | sean-k-mooney | so you can create a server with a key pair | |
| 17:28:07 | sean-k-mooney | assert that its in the first cell db | |
| 17:28:13 | sean-k-mooney | then call that to move it | |
| 17:28:20 | sean-k-mooney | and then check the second db | |
| 17:28:26 | sean-k-mooney | i think that woudl work | |
| 17:29:28 | Uggla | sean-k-mooney, OK I'll try. | |
| #openstack-nova - 2025-05-21 | |||
| 01:04:54 | opendevreview | Merged openstack/placement master: improve test logging and replace psycopg2 with psycopg2-binary https://review.opendev.org/c/openstack/placement/+/945487 | |
| 06:33:27 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix missing marker with instances in build_requests https://review.opendev.org/c/openstack/nova/+/947804 | |
| 06:34:51 | opendevreview | Stefan K proposed openstack/nova-specs master: Add Cloud Hypervisor support spec https://review.opendev.org/c/openstack/nova-specs/+/945549 | |
| 09:24:57 | sean-k-mooney | Uggla: there are 2 potential specless blueprint for use to consider https://blueprints.launchpad.net/nova/+spec/xml-image-meta which has an implemeation aviabel https://review.opendev.org/q/topic:%22bp/xml-image-meta%22 is the first | |
| 09:25:56 | sean-k-mooney | Uggla: the author reached out druing feature freeze and i chatted to them about it and gave some early feedback but they never procedurlly asked for the blueprint to be specless and for it to be reviewd | |
| 09:26:31 | sean-k-mooney | i suggested that they add it to the next meeting or ask for the approave via the mailing list if they cant attend | |
| 09:27:06 | sean-k-mooney | the ohter is more of a feature request but i think it could be specless https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/NSW3OG5ME5RDPQYLHF4T4RCWPQYG57PK/ | |
| 09:27:39 | sean-k-mooney | TLDR libvirt can now auto free ram allocated to a geust when the guest releases it internally. | |
| 09:27:55 | sean-k-mooney | you would not really want to do that for realtime vms | |
| 09:28:10 | sean-k-mooney | and you cant do it for hugepage/filebacked vms | |
| 09:28:32 | sean-k-mooney | but for standard vms we proably shoudl just enabel that always | |
| 09:28:54 | bauzas | sean-k-mooney: I need to go to the physio in 30 mins but if you have time, it would be nice if you could help me on https://review.opendev.org/c/openstack/nova/+/922140 | |
| 09:29:03 | sean-k-mooney | this is more just an fyi that thsoe would both eb low hanging fruite features. | |
| 09:29:43 | sean-k-mooney | bauzas: form the logs the last time it was pretty clear that the isseu was nto with the subnode using the wrong commit | |
| 09:30:20 | sean-k-mooney | i say you said that in the team meeting yesterday but that was no tthe problem previosuly, we coudl see the commit that was beign used in the job output on the ocmptue and it was correct | |
| 09:30:26 | sean-k-mooney | witht hat said sure i can take a look | |
| 09:30:59 | Uggla | sean-k-mooney, Ok I will add them to the doc and we will be able to discuss them in next meeting if it is required. | |
| 09:31:23 | sean-k-mooney | bauzas: i assume you have tired this locally? | |
| 09:37:27 | sean-k-mooney | bauzas: https://zuul.opendev.org/t/openstack/build/d31c2c01045a409a96d9dfd5e8aabbaa/log/job-output.txt#25980 so its defintly using the expected commit on the compute node | |
| 09:40:06 | sean-k-mooney | in the comptue node devstack logs we can also see the tat compilation is enabeld 2025-05-20 12:55:42.033 | ++ /opt/stack/nova/devstack/settings:source:2 : NOVA_COMPILE_MDEV_SAMPLES=True | |
| 09:40:51 | sean-k-mooney | mdevctl is also installed | |
| 09:43:09 | sean-k-mooney | looking at the compiel itesf there are some warning but apprently they were compled and loaded properly | |
| 09:43:11 | sean-k-mooney | https://paste.opendev.org/show/bcAgW92YFhDTcFs5QLDD/ | |
| 09:45:11 | sean-k-mooney | the probelm to me look more like libvirt has not detected the mdevs yet and we only see that on the compute beasue there is less of a delay between the devstack stages | |