Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-05
20:23:11 mriedem as sean-k-mooney said, there are probably allocations from migration records
20:23:13 sean-k-mooney heal_allocations would add allcotion for the node the vms are now running on but does it also clean up allocation where the vms are nolonger running
20:23:17 mriedem b/c you're deleting during a resize or something
20:23:42 cfriesen sean-k-mooney: nope. there are allocations with the same consumer id but two differnet resource_provider_id
20:24:41 mriedem https://bugs.launchpad.net/nova/+bug/1793569
20:24:41 cfriesen mriedem: would there be allocations from the evacuate?
20:24:43 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
20:25:16 mriedem cfriesen: i don't think there would be migration consumer allocatoins during an evac,
20:25:26 mriedem but there could be allocations for the instances that were on that evacuated host
20:25:45 sean-k-mooney i dont think there is either but we should claim resouce on the destiation
20:25:47 mriedem when the old host is brought back, it should clean up those evacuated instance allocations on startup
20:25:47 cfriesen they would have all been live-migrated onto that host not too long before I killed the node
20:26:38 mriedem gibi recently had something merged for evacuate that sounds related https://review.opendev.org/#/c/512623/
20:28:10 sean-k-mooney oh ya i rember looking at this a few days ago but not having time to review
20:32:00 sean-k-mooney mriedem: was this the patch you were asking about performace of "x in set()" vs "x in dict()"
20:32:00 mriedem cfriesen: see my note on https://review.opendev.org/#/c/657070/2/nova/scheduler/client/report.py as well
20:32:12 mriedem sean-k-mooney: no
20:32:24 sean-k-mooney ok
20:32:36 mriedem anywya, we don't have migration consumers for evac
20:33:05 sean-k-mooney we dont but we do claim an allcoation candate for teh destination node
20:33:25 sean-k-mooney but i dont know if we free the allcoation of the source node in all cases
20:33:54 mriedem correct the scheduler will claim (or conductor if you're forcing the dest host) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L923
20:34:03 mriedem the source host allocations should be cleaned up when the service is restarted
20:34:05 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422
20:34:13 cfriesen when evacuating an instance, when are the allocations on the old node supposed to be cleaned up?
20:34:27 mriedem in here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L620
20:34:34 mriedem on restart of the service ^
20:34:54 cfriesen okay, so if we never restart the service they'll just stay stale in the allocations table
20:35:00 mriedem right,
20:35:03 sean-k-mooney cfriesen: when you start up the compute agent on the souce node after fixing it meaning you wont be able to delete the RP until after that
20:35:05 mriedem but presumably that node is dead and fenced anyway
20:35:09 mriedem so you're not scheduling to it
20:35:36 sean-k-mooney mriedem: i think cfriesen was trying to remove teh RP before repovisoing the fixed node
20:35:36 cfriesen yeah...the problem is that those allocations prevent us from deleting the resource_provider on compute node deletion
20:36:02 sean-k-mooney cfriesen: why are you delete in the compute node recored out of interest
20:36:25 cfriesen customer is deleting the node. no idea why. testing functionality?
20:36:32 sean-k-mooney if you are replacing the node with another and reusing the hostname it should not be needed
20:36:40 sean-k-mooney ah ok
20:36:50 sean-k-mooney customer always break things
20:36:52 cfriesen when we add the new node back in with a different hostname it's all fine
20:37:28 cfriesen if we reuse the same hostname we get a different UUID in the compute_node table, but we don't get a new entry in the resource_provider table, so the UUIDs don't match and we never schedule anything to the node
20:37:28 mriedem the shitty thing i didn't realize until cdent pointed it out in an open review is that https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2183 won't raise if deleting the provider fails, which means we'll continue to delete the compute service record
20:37:31 sean-k-mooney cfriesen: if you add it with the same host name it would be fine too just dont delete the compute node recored and it will fix everything up when the compute agent starts
20:37:58 cfriesen sean-k-mooney: the compute node record is deleted by running the "delete service" api
20:38:19 sean-k-mooney cfriesen: yes. which you dont need to do to replace teh server
20:38:48 mriedem cfriesen: https://bugs.launchpad.net/nova/+bug/1829479 could also be related to your issue
20:38:49 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete]
20:38:53 sean-k-mooney and if you dont delete the compute service record you will get the same uuid when the new agent conencts
20:40:17 cfriesen sean-k-mooney: I suspect you're right as a workaround. but it *should* work in an ideal world. :)
20:40:51 cfriesen mriedem: yeah, that looks like what I'm seeing
20:41:35 sean-k-mooney cfriesen: we could "fix it" by having the compute node when it fails to find a compute node record by hostname first check if there is a placemetn RP by looking it up by hostname and then reuse the uuid of the RP which was the old compute node uuid
20:41:41 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422
20:42:53 cfriesen sean-k-mooney: would work but seems kind of kludgy. can we delete allocations when deleting a resource_provider?
20:43:35 sean-k-mooney we current dont allow you to delete inventoryes if they have allocatin intentionally
20:43:40 cfriesen I think this code might even have "cascade=true"
20:44:16 sean-k-mooney cfriesen: im pretty sure we intentnioally dont use cascade=true
20:44:30 cfriesen sean-k-mooney: it's in here, actually
20:44:41 mriedem cfriesen: ack i left a comment in https://bugs.launchpad.net/nova/+bug/1829479
20:44:42 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete]
20:45:05 mriedem cfriesen: we do try to delete the allocations,
20:45:12 mriedem the problem in your case is it's doing it by instances.host,
20:45:20 mriedem which if you've evacuated those instances, the instances.host points at the new dest compute
20:45:25 mriedem but the old allocations are still there
20:45:43 mriedem this could be easily recreated in a functional test i think
20:46:17 mriedem 1. create server on host1, 2. force down and evacuate from host1, 3. delete compute service for host1, 4. assert provider and allocs still exist for host1 in placement
20:46:30 mriedem and 5. restart host1 service should blow up
20:46:43 cfriesen ah, that makes sense
20:46:43 mriedem b/c of the unique constraint on the provider name
20:48:49 mriedem so https://github.com/openstack/nova/blob/653515a45032811b6bc2f1d0fb651472005496ec/nova/scheduler/client/report.py#L2173 is really just not right for (1) in-progress / unconfirmed migrations and (2) evacuated hosts
20:49:26 cfriesen agreed
20:49:55 mriedem i could start a rolling ball with said functional test to at least recreate the issue so we can throw fixes at it
20:49:56 sean-k-mooney cfriesen: just looking at the placmenet fuctional test tehy expresly check that deleteing allocated resouce provider fails just an fyi https://github.com/openstack/placement/blob/master/placement/tests/functional/db/test_resource_provider.py#L363-L367
20:50:54 cfriesen sean-k-mooney: yeah, it's valid that it fails when there are allocations, but setting "cascade=false" is supposed to delete those allocations. except it doesn't
20:52:02 sean-k-mooney when calling delete_resource_provider
20:52:30 mriedem cfriesen: you mean cascade=true
20:52:32 sean-k-mooney initally i assumed when you said cascade=true you were refering to the db schema
20:52:42 cfriesen bah, yes. cascade=true
20:52:44 mriedem looks like that code was written specifically for an ironic case https://review.opendev.org/#/c/428375/6/nova/scheduler/client/report.py@771
20:52:52 mriedem so not really thinking about migrations and evacuates and suh
20:52:54 mriedem *such
20:53:46 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Spec to pre-filter disabled computes with placement https://review.opendev.org/657884
20:53:47 mriedem dansmith: got that spec updated ^
20:55:28 sean-k-mooney i would need to check the placmeent api to confim but the allocation table has teh RP it was allcaoed from so we should be able to simply look up all the allocation for a given rp then delete tehm assuming that is exposed at the api level
20:55:34 sean-k-mooney which i proably is not
20:56:08 icarusfactor Can I have more than one compute_driver for nova.conf or is it limited to only one?
20:56:36 sean-k-mooney ya its not
20:57:39 sean-k-mooney so we need to add a new endpoint which is /allocations?rp_uuid=XYZ to be be able to do this cleanly
20:58:15 sean-k-mooney icarusfactor: its one per nova.conf
20:58:29 sean-k-mooney icarusfactor: but you can have multiple agent on the same host in some cases
20:58:44 sean-k-mooney icarusfactor: are you trying to run ironc and libvirt on teh same host?
20:59:15 icarusfactor sean-k-mooney, Nifty , I'm running libvirt currently. Was wondering how to do that.
20:59:50 sean-k-mooney its generally not a great idea to do but if you have a second nova.conf that you pass to the second agent you can make it work
21:00:55 icarusfactor sean-k-mooney, Just trying and experimenting with different methods , want to use the Stein docker nova driver.
21:00:56 sean-k-mooney normally i would recommend running the ironic driver assuming that is the other one you want to use on one of your contoler instead of colcating it on a libvirt compute node
21:01:17 mriedem sean-k-mooney: right i asked that here https://review.opendev.org/#/c/428375/6/nova/scheduler/client/report.py@771
21:01:20 mriedem but never got a response
21:02:22 mriedem we can get the allocations for a given provider with this API https://developer.openstack.org/api-ref/placement/?expanded=list-resource-provider-allocations-detail#list-resource-provider-allocations
21:02:25 sean-k-mooney icarusfactor: i had thought that dirver nova docker driver was nolonger mainatined
21:03:03 sean-k-mooney mriedem: oh off the RP endpoint
21:03:10 sean-k-mooney cool ya that seams more sane

Earlier   Later