Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-07-01
16:30:37 fwiesel Let me think. For starters: Verify-Resize.
16:31:13 fwiesel But let's say, we have a live-migration, and that host falls down. I still would like to do a failover
16:31:57 gibi verify-resize: so while you could evacuate from that state you also need to implement a proper cleanup in the nova-compute coming up in the source node of the original resize
16:32:00 sean-k-mooney so evacuate requires you the admin
16:32:11 sean-k-mooney to assert that the vm is not running on any host
16:32:31 sean-k-mooney so allowing it in verify resize and othe rstates is questionable
16:32:52 sean-k-mooney espically since in that state your in the middle of an ongoing migration
16:32:55 cardoe Yeah there's a lot of squirrely cases there which need to be monitored for.
16:33:07 sean-k-mooney and the flavor that is in use is kind of ill defiend
16:33:47 sean-k-mooney its technialy the new one in a real reaise but it has nto beeen confirm so you woudl want ot evacuate to the old one, but hte numa toplogy and ohter things are all in a weird hybrid state
16:34:28 sean-k-mooney what you likely should be able to do is revert the resize in verify-resize
16:34:29 gibi so in general it would be nice to support these but in practic it seems like a bunch of cans of worms
16:34:37 sean-k-mooney even if the dest host is down
16:34:41 sean-k-mooney then evacuate it
16:34:43 sean-k-mooney if needed
16:35:13 cardoe We've got a whole thing out of band that's specific to the hypervisor in use for these items.
16:35:31 gibi yeah allowing revert-resize in souce host down is one way, even though a weird one. Who wants to revert a resize that will lead to the VM runing nowhere :)
16:35:33 fwiesel But technically, an evacuate is practically a rebuild. So, I can start from "scratch" (nova wise)
16:35:54 sean-k-mooney gibi: well it would revert reisze when dest down and conrim resize when source down
16:35:56 fwiesel I just have to build the VM up somewhere fitting the spec
16:36:05 gibi yeah starting is easy, cleaning up the remnants is messy
16:36:12 sean-k-mooney fwiesel: yes and no
16:36:13 cardoe yeah but you need to make sure that gets fully cleaned up.
16:36:20 fwiesel Of course
16:36:25 sean-k-mooney we have some api contract about preserving storage in some cases
16:36:40 sean-k-mooney please there are the neutorn/cidner state clean up that we need to do
16:36:48 gibi sean-k-mooney: revert-resize: ahh right
16:36:48 fwiesel Exactly
16:37:07 fwiesel Right now, I have the choice to implement all that outside of nova. Or within nova
16:37:25 sean-k-mooney to do this in nova woudl need a very detailed spec
16:37:41 fwiesel Well, I would not tackle all cases in one go.
16:37:42 sean-k-mooney and you woudl have to consier things liek cross cell resize
16:38:01 fwiesel And maybe then not resize :)
16:38:15 cardoe its a worthy goal
16:38:18 sean-k-mooney right but even if you take an incremntal approch you need to have each incremantal chagne be valid
16:38:55 sean-k-mooney i think for rezie it woubld be siiler to allwo the migration to compelte or revert based on which end is alive
16:38:57 cardoe I'll say when I did this with xen it was much easier because xapi had an object version field so I used that as a monotonic number which also then was stored in the nova DB
16:39:03 sean-k-mooney and if they are both dead well your kind of out of luck
16:39:10 fwiesel Of course. I was more fearing that it will be difficult to test that in the CI, and adds complexity. Which might be areason you might not want it at all.
16:39:26 cardoe So each time the instance got updated the monotonic number increased and that got pushed down into xapi
16:39:28 gibi fwiesel: start with a simpler state first, write up a spec how would you make sure we end up in the consistent state after the a successful evac and a recovered compute (or an failed evac and a recovered compute, of a failed evac and a retried evac and recovered compute, ...)
16:39:31 cardoe But libvirt doesn't have that
16:39:47 cardoe So when we'd go to clean up, anything that wasn't at the current version could get nuked.
16:39:59 sean-k-mooney fwiesel: so testing that in tempest would be hard, but it would be more doable in the functional suite
16:40:03 cardoe But there's all kinds of dangling references to storage and what not that really are a pain to fix
16:40:36 gibi yepp use functional test for sure
16:40:51 fwiesel Okay, I'll take that with me. Thanks.
16:40:52 gibi we have a limited evact test somewhere after tempest
16:40:57 sean-k-mooney cardoe: so we dont need montic object version in general becasue we assocate the instance with the comptue service that is currently manging it
16:40:58 gibi in a palybook
16:41:11 sean-k-mooney in the post test hook in nova-next
16:41:16 cardoe For us we implemented this more as monitoring that an unexpected object was on a host.
16:41:17 sean-k-mooney and i think the cepch job?
16:42:03 sean-k-mooney cardoe: ya so that kind of approch is not very comaptible with the nova desing although we do have some per host perodic that are intened ot clean up these types of issues
16:42:08 cardoe sean-k-mooney: when you perform a live migrate and the dest goes down at the very end during the resume
16:42:09 sean-k-mooney esplcially on compute agent start up
16:42:53 sean-k-mooney cardoe: if the vm dies before resume libvirt will abort the migraiot
16:43:01 sean-k-mooney if we got to post live migrate
16:43:21 sean-k-mooney its too late to abort but we will update the instance hsot to point to the dest
16:43:28 gibi this is the evac testing in devstack https://github.com/openstack/nova/blob/master/roles/run-evacuate-hook/tasks/main.yaml
16:43:58 Uggla guys is it ok to move to the second point ?
16:44:05 gibi OK for me
16:44:08 cardoe I'm just saying it wasn't a simple problem when I was working in the VM world because hardware dies at the worst possible times and nova can be in many different code calls when that happens.
16:44:10 sean-k-mooney yep lets move on for now.
16:44:24 Uggla #topic (jonnyb) Blueprint for new compute created weigher (https://blueprints.launchpad.net/nova/+spec/node-uptime-weigher)
16:44:47 Uggla jonnyb wants to discuss about this weigher
16:45:01 Uggla jonnyb please go ahead.
16:45:13 jonnyb so I propose a scheduler weigher based on creation date of the compute service
16:45:41 jonnyb the initial idea was to use uptime but this is too dependend on the hypervisor and its driver
16:46:11 jonnyb for us the creation is good enough and we use it since we migrate and deploy a lot of servers
16:46:13 gibi do you have the service creation time as part of the HostState the weigher running on?
16:46:29 sean-k-mooney i belive we do
16:46:36 dansmith that surprises me
16:46:53 sean-k-mooney we have it form the compute node object i belive
16:47:01 dansmith sure, but in hoststate?
16:47:38 sean-k-mooney so this is the poc https://review.opendev.org/c/openstack/nova/+/947503/9/nova/scheduler/weights/compute_created.py
16:48:27 dansmith we have an object, but I didn't think it was queried from the db and thus not complete, but maybe that's not right
16:49:08 sean-k-mooney so yes https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L176
16:49:26 gibi based on the poc we have HostState.service.created_at as created_at does not expect to change it is fair to assume it is up to date
16:49:26 sean-k-mooney you could be correct about it being a partial object
16:49:45 gibi isn't expected to change
16:50:34 sean-k-mooney so it looks like this is only populated via update its not populated iva __init__
16:51:02 sean-k-mooney so im not sure if its there if you disatble the track_isntace_state notifications stuff
16:51:17 dansmith okay we are doing a scatter to gather nodes and hosts from cells
16:51:18 sean-k-mooney we woudl have to confirm that its always present in both confiugrations
16:51:30 dansmith I was thinking we did an optimized construction from a list there
16:51:44 dansmith sean-k-mooney: yeah, good point
16:52:11 sean-k-mooney i think that config only affect the instnace object and how those are populated
16:52:16 dansmith because this could also be triggering massive lazy loads, which would be bad
16:52:19 sean-k-mooney but assuming we have that alwasy
16:52:43 sean-k-mooney i think it would be ok to use it in the weigher provide ya we are not lazy loading this
16:53:04 dansmith we need to make sure this is run in CI at least once so we can examine logs for lazy loads
16:56:32 sean-k-mooney ya we can turn this on in one fo the jobs
16:57:08 sean-k-mooney it wont really have much of an effect with only 1-2 nodes
16:57:10 dansmith we don't have to permanently, but just a DNM on top of this to get a run and make sure we don't see "lazy loading service.created_at" :)
16:58:10 sean-k-mooney we are wrappign the sercie in a readonly dict class that i have not really seen before. i would assume that shoudl also block lazyloading?
16:58:24 sean-k-mooney goign back on topic
16:58:34 sean-k-mooney are we ok with appvoing this as a specless blueprint
16:58:38 sean-k-mooney and continuing to review
16:59:16 dansmith idk

Earlier   Later