| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-29 | |||
| 16:42:25 | sean-k-mooney | but its my main compute node in my home openstack i jsut pop onto the host if i want to run fast | |
| 16:43:20 | sean-k-mooney | our unit test are very very paralisable | |
| 16:43:28 | sean-k-mooney | anyway i think we are a little off topic | |
| 16:43:35 | Uggla | yepp | |
| 16:43:55 | Uggla | any other topic ? | |
| 16:44:08 | sp-bmilanov | yep | |
| 16:44:45 | Uggla | oh sp-bmilanov, please go ahead. And sorry I might remove your topic thinking it was covered last time | |
| 16:44:54 | sp-bmilanov | Uggla: no probs | |
| 16:45:28 | sp-bmilanov | yeah, so from last time, gibi, dansmith, you might remember the nova-compute forced restart bug, I drafted a spec: https://review.opendev.org/c/openstack/nova-specs/+/955812 | |
| 16:45:53 | sp-bmilanov | does it make sense to discuss it with the hindsight of Sean's comments? (thanks sean-k-mooney!) | |
| 16:46:29 | sp-bmilanov | as I understand it, it would be best if we add some extra RPC functionality for the source to ask the destination if the instance is running there | |
| 16:46:42 | sean-k-mooney | just opening that now | |
| 16:46:54 | sean-k-mooney | ah the restart during live migration issue | |
| 16:47:18 | sean-k-mooney | so there are proably a few differnt way to adress that. btu there are trade off for all of them | |
| 16:47:34 | sp-bmilanov | IMO admin-locking it would be a compromise, what you suggested with the self-heal sounds best | |
| 16:48:14 | sean-k-mooney | so i think there are diffent mitigation for diffent usecase. where we can gracefully shutdown i think we likely shoudl abort the live migration ad the libvirt level | |
| 16:48:24 | sean-k-mooney | and then clean it up at the nova level on start up | |
| 16:48:45 | sean-k-mooney | buting the isntnace to error and locking them i think is a step we coudl take to improve the situraion | |
| 16:49:16 | sean-k-mooney | but the ultimate fix would be to be able to resume the management of the migration and allow it to succeded if in fact did | |
| 16:49:51 | sean-k-mooney | i feel like an incremental approch is what we shoudl try to do but i dont know what others think | |
| 16:50:11 | sean-k-mooney | sp-bmilanov: is this somethign you woudl ahve time to work on or are you rasing it as a pain point that you think we shoudl prioritize | |
| 16:50:19 | gibi | last week we discussed that sp-bmilanov interested in the non-graceful restart case | |
| 16:50:33 | gibi | so we need something in the startup code path that prevents the original problem | |
| 16:51:36 | sp-bmilanov | gibi: yes, the ungraceful restart case is what we experienced ; sean-k-mooney pain point first, I am not sure if I can help implement something that you've triaged as so complex | |
| 16:51:37 | gibi | ie. the duplication of instances reported https://bugs.launchpad.net/nova/+bug/2092391 | |
| 16:52:43 | gibi | I believe we should be able to detect during compute startup that we have a duplication based on the fact that one of the domain is on a host that is not equal to the instance.host | |
| 16:53:12 | gibi | or based on that we have an unfinished migration | |
| 16:53:16 | gibi | in the DB | |
| 16:53:33 | sp-bmilanov | just an aside, I guess that's is not a good approach since you haven't brought it up, but why not try using a flag in the database to check if the migration is at at point where the instance is running at the destination | |
| 16:53:39 | sean-k-mooney | so the duplciation i think weill need us to do an rpc to the dest compute | |
| 16:53:59 | sean-k-mooney | and determin if we shoudl revert or run post-live-migration | |
| 16:54:02 | gibi | sp-bmilanov: I hope we have a migration state that signals that already | |
| 16:55:50 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Address issues with images APIs https://review.opendev.org/c/openstack/nova/+/956102 | |
| 16:56:37 | sean-k-mooney | sp-bmilanov: so resumeing the guest is not done by nova | |
| 16:56:40 | gibi | anyhow I think it make sense to review a spec and perpare a PTG discussion about it | |
| 16:56:43 | sean-k-mooney | its doen by libvirt | |
| 16:56:55 | sean-k-mooney | and the scoue comput enode will update the dbstate if its still runing | |
| 16:57:15 | sean-k-mooney | the problem currently is if you restart it the thread we have monitoring the libvirt job will be killed | |
| 16:57:25 | sean-k-mooney | and we wont do that update. | |
| 16:57:57 | sean-k-mooney | but on startup we can see fi the job is still runing and either resume waiting or make an rpc to see if it scuceeded. or check if the vm is running lcoally | |
| 16:58:14 | sean-k-mooney | if its runnign locally and there is no migration job active in libvirt it mean the migratoin failed | |
| 16:58:25 | sean-k-mooney | if the vm is not local it means either it crashed or it moved | |
| 16:58:38 | sean-k-mooney | so we coudl be more intelegent on startup | |
| 16:58:48 | sean-k-mooney | but someone need to spend tiem documenting this bevhioer | |
| 16:58:50 | sean-k-mooney | and writing it up | |
| 16:59:10 | sean-k-mooney | so yes i think a spec/ptg discussion woudl be good | |
| 16:59:11 | gibi | yepp I agree | |
| 17:00:58 | sp-bmilanov | s/checked from the DB/set from the dst and then checked from the src from the DB/ | |
| 17:01:32 | sean-k-mooney | well | |
| 17:01:45 | sean-k-mooney | the compute agent does not have direct db access for one | |
| 17:01:55 | sean-k-mooney | so just chekc the db state is already an rpc to teh conductor | |
| 17:02:06 | sean-k-mooney | but the responisbleity for updating the host of the vm is on the souce node | |
| 17:02:14 | sean-k-mooney | as part of post live migrate | |
| 17:02:33 | sean-k-mooney | so on start up if we need to check if tis on the dest node the source node need to call the dest to do that | |
| 17:02:52 | sean-k-mooney | i.e. asumeign the vm is not still running locally | |
| 17:03:39 | sean-k-mooney | we need to know if it crashed or if its actully active on the other host. and we can do that with only local knowlsage or by checkign the db. | |
| 17:04:50 | Uggla | We are on top of the hour. So I propose to close the meeting. ^ discusion can happen right after if needed. | |
| 17:05:06 | sp-bmilanov | thanks Uggla | |
| 17:05:20 | Uggla | Thanks all. | |
| 17:05:22 | Uggla | #endmeeting | |
| 17:05:22 | opendevmeet | Meeting ended Tue Jul 29 17:05:22 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 17:05:22 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-29-16.02.html | |
| 17:05:22 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-29-16.02.txt | |
| 17:05:22 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-29-16.02.log.html | |
| 17:05:42 | sp-bmilanov | sean-k-mooney right, it makes sense that the DB will be of no use if the one responsible for the book-keeping there is the source | |
| 17:06:13 | gibi | I needs to drop. Thanks folks | |
| 17:07:29 | sean-k-mooney | Uggla: 16905 tests in 84.9128 sec. with 48 thread/24 cores even at only 2.5GHz you can run the test suite pretty fast | |
| 17:07:42 | sean-k-mooney | but looking at my slow tests | |
| 17:07:44 | sean-k-mooney | nova.tests.unit.api.openstack.compute.test_schemas.SchemaTest.test_schemas 25.415 | |
| 17:07:59 | sean-k-mooney | 25 second for the schema tests seam a bit off | |
| 17:08:22 | sp-bmilanov | got to go as well, thanks sean-k-mooney, gibi! | |
| 17:08:22 | sean-k-mooney | the other ones at the top of the list i think are ones gibi is speeding up | |
| 17:09:35 | Uggla | yes the test schema seems surprisingly long | |
| 17:12:40 | sean-k-mooney | with gibi's speedup serise applied i get this https://paste.opendev.org/show/bN1SLJLLAhPmqGFIRNID/ | |
| 17:13:01 | sean-k-mooney | so its liek 9 tiems slower the the nearest other test | |
| 17:19:59 | sean-k-mooney | https://paste.opendev.org/show/brPnnHtdLgbbaEFxJbfi/ 229.4782 sec. is not too bad for all the funcitonal tests on this host but its longer then i would perfer | |
| 17:54:09 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server usage audit log APIs https://review.opendev.org/c/openstack/nova/+/953209 | |
| 17:54:09 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for tenant network APIs https://review.opendev.org/c/openstack/nova/+/956088 | |
| 17:54:10 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for simple tenant usage APIs https://review.opendev.org/c/openstack/nova/+/956096 | |
| 18:29:23 | opendevreview | Merged openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 19:12:33 | opendevreview | Merged openstack/nova master: api: Add response body schemas for images APIs https://review.opendev.org/c/openstack/nova/+/952284 | |
| 20:26:35 | opendevreview | Merged openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754 | |
| #openstack-nova - 2025-07-30 | |||
| 01:03:17 | opendevreview | melanie witt proposed openstack/nova master: Add hw_tpm_secret_security image property https://review.opendev.org/c/openstack/nova/+/940196 | |
| 01:03:17 | opendevreview | melanie witt proposed openstack/nova master: Add hw:tpm_secret_security extra spec validation https://review.opendev.org/c/openstack/nova/+/940197 | |
| 01:03:18 | opendevreview | melanie witt proposed openstack/nova master: TPM: migrate legacy instances to new security policy https://review.opendev.org/c/openstack/nova/+/942501 | |
| 01:03:18 | opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `user` secret security https://review.opendev.org/c/openstack/nova/+/942502 | |
| 01:03:19 | opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `host` secret security https://review.opendev.org/c/openstack/nova/+/941795 | |
| 01:03:19 | opendevreview | melanie witt proposed openstack/nova master: Add vtpm_secret_(uuid|value) to LiveMigrateData https://review.opendev.org/c/openstack/nova/+/952628 | |
| 01:03:20 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `host` secret security https://review.opendev.org/c/openstack/nova/+/941483 | |
| 01:03:21 | opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `deployment` secret security https://review.opendev.org/c/openstack/nova/+/942021 | |
| 01:03:21 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771 | |
| 01:03:23 | opendevreview | melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629 | |
| 01:03:23 | opendevreview | melanie witt proposed openstack/nova master: TPM: update instance request_spec with secret security https://review.opendev.org/c/openstack/nova/+/952630 | |
| 01:03:25 | opendevreview | melanie witt proposed openstack/nova master: TPM: confirm secret security via hard reboot https://review.opendev.org/c/openstack/nova/+/955847 | |
| 01:12:59 | opendevreview | melanie witt proposed openstack/nova master: TPM: migrate legacy instances to new security policy https://review.opendev.org/c/openstack/nova/+/942501 | |
| 01:13:00 | opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `user` secret security https://review.opendev.org/c/openstack/nova/+/942502 | |
| 01:13:00 | opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `host` secret security https://review.opendev.org/c/openstack/nova/+/941795 | |
| 01:13:01 | opendevreview | melanie witt proposed openstack/nova master: Add vtpm_secret_(uuid|value) to LiveMigrateData https://review.opendev.org/c/openstack/nova/+/952628 | |
| 01:13:01 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `host` secret security https://review.opendev.org/c/openstack/nova/+/941483 | |
| 01:13:02 | opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `deployment` secret security https://review.opendev.org/c/openstack/nova/+/942021 | |