Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-07-29
16:36:53 gibi -> flag = self._event.wait(timeout)
16:37:00 gibi ahh wrong buffer
16:37:06 gibi https://github.com/openstack/nova/blob/master/tox.ini#L54
16:37:18 sean-k-mooney so context, we use stester which uses subunit internally as a db
16:37:19 gibi so you can use stestr to print it
16:37:35 sean-k-mooney and there are some tools to extract thigns like slowest tests and print them
16:37:46 sean-k-mooney that what stestr slowest is doing
16:38:00 sean-k-mooney it looking at the last test run in the subunit db and listing them
16:38:00 Uggla good to know thanks
16:38:45 Uggla sean-k-mooney I'm just wondering if the 48 cores machine is part of your homelab ? Sorry I'm curious
16:39:17 sean-k-mooney yes it part of the hardware im considerign replacing its pretty old at this point
16:39:55 sp-bmilanov sean-k-mooney sounds like a dual-xeon workstation from mid 2010s
16:40:13 sean-k-mooney yep
16:40:27 sean-k-mooney Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
16:41:03 gmaan I recently bought two 16 core dell poweredge R630 but need to setup those
16:41:43 Uggla gmaan for homelab too ?
16:42:00 gmaan yeah, my both old desktop are dead
16:42:00 sean-k-mooney honestly you can get much more power effeict systems now
16:42:25 sean-k-mooney but its my main compute node in my home openstack i jsut pop onto the host if i want to run fast
16:43:20 sean-k-mooney our unit test are very very paralisable
16:43:28 sean-k-mooney anyway i think we are a little off topic
16:43:35 Uggla yepp
16:43:55 Uggla any other topic ?
16:44:08 sp-bmilanov yep
16:44:45 Uggla oh sp-bmilanov, please go ahead. And sorry I might remove your topic thinking it was covered last time
16:44:54 sp-bmilanov Uggla: no probs
16:45:28 sp-bmilanov yeah, so from last time, gibi, dansmith, you might remember the nova-compute forced restart bug, I drafted a spec: https://review.opendev.org/c/openstack/nova-specs/+/955812
16:45:53 sp-bmilanov does it make sense to discuss it with the hindsight of Sean's comments? (thanks sean-k-mooney!)
16:46:29 sp-bmilanov as I understand it, it would be best if we add some extra RPC functionality for the source to ask the destination if the instance is running there
16:46:42 sean-k-mooney just opening that now
16:46:54 sean-k-mooney ah the restart during live migration issue
16:47:18 sean-k-mooney so there are proably a few differnt way to adress that. btu there are trade off for all of them
16:47:34 sp-bmilanov IMO admin-locking it would be a compromise, what you suggested with the self-heal sounds best
16:48:14 sean-k-mooney so i think there are diffent mitigation for diffent usecase. where we can gracefully shutdown i think we likely shoudl abort the live migration ad the libvirt level
16:48:24 sean-k-mooney and then clean it up at the nova level on start up
16:48:45 sean-k-mooney buting the isntnace to error and locking them i think is a step we coudl take to improve the situraion
16:49:16 sean-k-mooney but the ultimate fix would be to be able to resume the management of the migration and allow it to succeded if in fact did
16:49:51 sean-k-mooney i feel like an incremental approch is what we shoudl try to do but i dont know what others think
16:50:11 sean-k-mooney sp-bmilanov: is this somethign you woudl ahve time to work on or are you rasing it as a pain point that you think we shoudl prioritize
16:50:19 gibi last week we discussed that sp-bmilanov interested in the non-graceful restart case
16:50:33 gibi so we need something in the startup code path that prevents the original problem
16:51:36 sp-bmilanov gibi: yes, the ungraceful restart case is what we experienced ; sean-k-mooney pain point first, I am not sure if I can help implement something that you've triaged as so complex
16:51:37 gibi ie. the duplication of instances reported https://bugs.launchpad.net/nova/+bug/2092391
16:52:43 gibi I believe we should be able to detect during compute startup that we have a duplication based on the fact that one of the domain is on a host that is not equal to the instance.host
16:53:12 gibi or based on that we have an unfinished migration
16:53:16 gibi in the DB
16:53:33 sp-bmilanov just an aside, I guess that's is not a good approach since you haven't brought it up, but why not try using a flag in the database to check if the migration is at at point where the instance is running at the destination
16:53:39 sean-k-mooney so the duplciation i think weill need us to do an rpc to the dest compute
16:53:59 sean-k-mooney and determin if we shoudl revert or run post-live-migration
16:54:02 gibi sp-bmilanov: I hope we have a migration state that signals that already
16:55:50 opendevreview Stephen Finucane proposed openstack/nova master: api: Address issues with images APIs https://review.opendev.org/c/openstack/nova/+/956102
16:56:37 sean-k-mooney sp-bmilanov: so resumeing the guest is not done by nova
16:56:40 gibi anyhow I think it make sense to review a spec and perpare a PTG discussion about it
16:56:43 sean-k-mooney its doen by libvirt
16:56:55 sean-k-mooney and the scoue comput enode will update the dbstate if its still runing
16:57:15 sean-k-mooney the problem currently is if you restart it the thread we have monitoring the libvirt job will be killed
16:57:25 sean-k-mooney and we wont do that update.
16:57:57 sean-k-mooney but on startup we can see fi the job is still runing and either resume waiting or make an rpc to see if it scuceeded. or check if the vm is running lcoally
16:58:14 sean-k-mooney if its runnign locally and there is no migration job active in libvirt it mean the migratoin failed
16:58:25 sean-k-mooney if the vm is not local it means either it crashed or it moved
16:58:38 sean-k-mooney so we coudl be more intelegent on startup
16:58:48 sean-k-mooney but someone need to spend tiem documenting this bevhioer
16:58:50 sean-k-mooney and writing it up
16:59:10 sean-k-mooney so yes i think a spec/ptg discussion woudl be good
16:59:11 gibi yepp I agree
17:00:58 sp-bmilanov s/checked from the DB/set from the dst and then checked from the src from the DB/
17:01:32 sean-k-mooney well
17:01:45 sean-k-mooney the compute agent does not have direct db access for one
17:01:55 sean-k-mooney so just chekc the db state is already an rpc to teh conductor
17:02:06 sean-k-mooney but the responisbleity for updating the host of the vm is on the souce node
17:02:14 sean-k-mooney as part of post live migrate
17:02:33 sean-k-mooney so on start up if we need to check if tis on the dest node the source node need to call the dest to do that
17:02:52 sean-k-mooney i.e. asumeign the vm is not still running locally
17:03:39 sean-k-mooney we need to know if it crashed or if its actully active on the other host. and we can do that with only local knowlsage or by checkign the db.
17:04:50 Uggla We are on top of the hour. So I propose to close the meeting. ^ discusion can happen right after if needed.
17:05:06 sp-bmilanov thanks Uggla
17:05:20 Uggla Thanks all.
17:05:22 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-29-16.02.log.html
17:05:22 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-29-16.02.txt
17:05:22 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-07-29-16.02.html
17:05:22 opendevmeet Meeting ended Tue Jul 29 17:05:22 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
17:05:22 Uggla #endmeeting
17:05:42 sp-bmilanov sean-k-mooney right, it makes sense that the DB will be of no use if the one responsible for the book-keeping there is the source
17:06:13 gibi I needs to drop. Thanks folks
17:07:29 sean-k-mooney Uggla: 16905 tests in 84.9128 sec. with 48 thread/24 cores even at only 2.5GHz you can run the test suite pretty fast
17:07:42 sean-k-mooney but looking at my slow tests
17:07:44 sean-k-mooney nova.tests.unit.api.openstack.compute.test_schemas.SchemaTest.test_schemas 25.415
17:07:59 sean-k-mooney 25 second for the schema tests seam a bit off
17:08:22 sean-k-mooney the other ones at the top of the list i think are ones gibi is speeding up
17:08:22 sp-bmilanov got to go as well, thanks sean-k-mooney, gibi!
17:09:35 Uggla yes the test schema seems surprisingly long
17:12:40 sean-k-mooney with gibi's speedup serise applied i get this https://paste.opendev.org/show/bN1SLJLLAhPmqGFIRNID/
17:13:01 sean-k-mooney so its liek 9 tiems slower the the nearest other test
17:19:59 sean-k-mooney https://paste.opendev.org/show/brPnnHtdLgbbaEFxJbfi/ 229.4782 sec. is not too bad for all the funcitonal tests on this host but its longer then i would perfer
17:54:09 opendevreview Stephen Finucane proposed openstack/nova master: api: Add response body schemas for tenant network APIs https://review.opendev.org/c/openstack/nova/+/956088
17:54:09 opendevreview Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server usage audit log APIs https://review.opendev.org/c/openstack/nova/+/953209
17:54:10 opendevreview Stephen Finucane proposed openstack/nova master: api: Add response body schemas for simple tenant usage APIs https://review.opendev.org/c/openstack/nova/+/956096
18:29:23 opendevreview Merged openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
19:12:33 opendevreview Merged openstack/nova master: api: Add response body schemas for images APIs https://review.opendev.org/c/openstack/nova/+/952284
20:26:35 opendevreview Merged openstack/nova master: Replace eventlet.event.Event with threading.Event https://review.opendev.org/c/openstack/nova/+/949754

Earlier   Later