Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-05
20:35:00 mriedem right,
20:35:03 sean-k-mooney cfriesen: when you start up the compute agent on the souce node after fixing it meaning you wont be able to delete the RP until after that
20:35:05 mriedem but presumably that node is dead and fenced anyway
20:35:09 mriedem so you're not scheduling to it
20:35:36 cfriesen yeah...the problem is that those allocations prevent us from deleting the resource_provider on compute node deletion
20:35:36 sean-k-mooney mriedem: i think cfriesen was trying to remove teh RP before repovisoing the fixed node
20:36:02 sean-k-mooney cfriesen: why are you delete in the compute node recored out of interest
20:36:25 cfriesen customer is deleting the node. no idea why. testing functionality?
20:36:32 sean-k-mooney if you are replacing the node with another and reusing the hostname it should not be needed
20:36:40 sean-k-mooney ah ok
20:36:50 sean-k-mooney customer always break things
20:36:52 cfriesen when we add the new node back in with a different hostname it's all fine
20:37:28 mriedem the shitty thing i didn't realize until cdent pointed it out in an open review is that https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2183 won't raise if deleting the provider fails, which means we'll continue to delete the compute service record
20:37:28 cfriesen if we reuse the same hostname we get a different UUID in the compute_node table, but we don't get a new entry in the resource_provider table, so the UUIDs don't match and we never schedule anything to the node
20:37:31 sean-k-mooney cfriesen: if you add it with the same host name it would be fine too just dont delete the compute node recored and it will fix everything up when the compute agent starts
20:37:58 cfriesen sean-k-mooney: the compute node record is deleted by running the "delete service" api
20:38:19 sean-k-mooney cfriesen: yes. which you dont need to do to replace teh server
20:38:48 mriedem cfriesen: https://bugs.launchpad.net/nova/+bug/1829479 could also be related to your issue
20:38:49 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete]
20:38:53 sean-k-mooney and if you dont delete the compute service record you will get the same uuid when the new agent conencts
20:40:17 cfriesen sean-k-mooney: I suspect you're right as a workaround. but it *should* work in an ideal world. :)
20:40:51 cfriesen mriedem: yeah, that looks like what I'm seeing
20:41:35 sean-k-mooney cfriesen: we could "fix it" by having the compute node when it fails to find a compute node record by hostname first check if there is a placemetn RP by looking it up by hostname and then reuse the uuid of the RP which was the old compute node uuid
20:41:41 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422
20:42:53 cfriesen sean-k-mooney: would work but seems kind of kludgy. can we delete allocations when deleting a resource_provider?
20:43:35 sean-k-mooney we current dont allow you to delete inventoryes if they have allocatin intentionally
20:43:40 cfriesen I think this code might even have "cascade=true"
20:44:16 sean-k-mooney cfriesen: im pretty sure we intentnioally dont use cascade=true
20:44:30 cfriesen sean-k-mooney: it's in here, actually
20:44:41 mriedem cfriesen: ack i left a comment in https://bugs.launchpad.net/nova/+bug/1829479
20:44:42 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete]
20:45:05 mriedem cfriesen: we do try to delete the allocations,
20:45:12 mriedem the problem in your case is it's doing it by instances.host,
20:45:20 mriedem which if you've evacuated those instances, the instances.host points at the new dest compute
20:45:25 mriedem but the old allocations are still there
20:45:43 mriedem this could be easily recreated in a functional test i think
20:46:17 mriedem 1. create server on host1, 2. force down and evacuate from host1, 3. delete compute service for host1, 4. assert provider and allocs still exist for host1 in placement
20:46:30 mriedem and 5. restart host1 service should blow up
20:46:43 mriedem b/c of the unique constraint on the provider name
20:46:43 cfriesen ah, that makes sense
20:48:49 mriedem so https://github.com/openstack/nova/blob/653515a45032811b6bc2f1d0fb651472005496ec/nova/scheduler/client/report.py#L2173 is really just not right for (1) in-progress / unconfirmed migrations and (2) evacuated hosts
20:49:26 cfriesen agreed
20:49:55 mriedem i could start a rolling ball with said functional test to at least recreate the issue so we can throw fixes at it
20:49:56 sean-k-mooney cfriesen: just looking at the placmenet fuctional test tehy expresly check that deleteing allocated resouce provider fails just an fyi https://github.com/openstack/placement/blob/master/placement/tests/functional/db/test_resource_provider.py#L363-L367
20:50:54 cfriesen sean-k-mooney: yeah, it's valid that it fails when there are allocations, but setting "cascade=false" is supposed to delete those allocations. except it doesn't
20:52:02 sean-k-mooney when calling delete_resource_provider
20:52:30 mriedem cfriesen: you mean cascade=true
20:52:32 sean-k-mooney initally i assumed when you said cascade=true you were refering to the db schema
20:52:42 cfriesen bah, yes. cascade=true
20:52:44 mriedem looks like that code was written specifically for an ironic case https://review.opendev.org/#/c/428375/6/nova/scheduler/client/report.py@771
20:52:52 mriedem so not really thinking about migrations and evacuates and suh
20:52:54 mriedem *such
20:53:46 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Spec to pre-filter disabled computes with placement https://review.opendev.org/657884
20:53:47 mriedem dansmith: got that spec updated ^
20:55:28 sean-k-mooney i would need to check the placmeent api to confim but the allocation table has teh RP it was allcaoed from so we should be able to simply look up all the allocation for a given rp then delete tehm assuming that is exposed at the api level
20:55:34 sean-k-mooney which i proably is not
20:56:08 icarusfactor Can I have more than one compute_driver for nova.conf or is it limited to only one?
20:56:36 sean-k-mooney ya its not
20:57:39 sean-k-mooney so we need to add a new endpoint which is /allocations?rp_uuid=XYZ to be be able to do this cleanly
20:58:15 sean-k-mooney icarusfactor: its one per nova.conf
20:58:29 sean-k-mooney icarusfactor: but you can have multiple agent on the same host in some cases
20:58:44 sean-k-mooney icarusfactor: are you trying to run ironc and libvirt on teh same host?
20:59:15 icarusfactor sean-k-mooney, Nifty , I'm running libvirt currently. Was wondering how to do that.
20:59:50 sean-k-mooney its generally not a great idea to do but if you have a second nova.conf that you pass to the second agent you can make it work
21:00:55 icarusfactor sean-k-mooney, Just trying and experimenting with different methods , want to use the Stein docker nova driver.
21:00:56 sean-k-mooney normally i would recommend running the ironic driver assuming that is the other one you want to use on one of your contoler instead of colcating it on a libvirt compute node
21:01:17 mriedem sean-k-mooney: right i asked that here https://review.opendev.org/#/c/428375/6/nova/scheduler/client/report.py@771
21:01:20 mriedem but never got a response
21:02:22 mriedem we can get the allocations for a given provider with this API https://developer.openstack.org/api-ref/placement/?expanded=list-resource-provider-allocations-detail#list-resource-provider-allocations
21:02:25 sean-k-mooney icarusfactor: i had thought that dirver nova docker driver was nolonger mainatined
21:03:03 sean-k-mooney mriedem: oh off the RP endpoint
21:03:10 sean-k-mooney cool ya that seams more sane
21:03:20 icarusfactor sean-k-mooney, Humm, that is what I was wondering , the Dashboard now shows the docker option.
21:03:22 sean-k-mooney at least that way we cant get out of sync
21:03:44 sean-k-mooney icarusfactor: dashboard as in horizon?
21:03:47 icarusfactor Seeing it in Stien , I was wanting to try this option.
21:03:50 icarusfactor yes
21:04:05 sean-k-mooney icarusfactor: could that be form zun?
21:04:11 mriedem https://review.opendev.org/#/c/657016/ is my change where cdent pointed out that failure to delete the rp doesn't make us fail to delete the compute service
21:04:16 mriedem finally connecting all the dots
21:04:19 mriedem i'm slow and dumb
21:04:33 icarusfactor sean-k-mooney, Cant answer that.
21:04:59 sean-k-mooney icarusfactor: https://github.com/openstack/nova-docker
21:05:22 sean-k-mooney icarusfactor: it proably related to zun
21:05:55 sean-k-mooney mriedem: oh this is a recent change
21:06:08 icarusfactor sean-k-mooney, ok , will check zun out , that helps as I was going /dev/null fast.
21:06:14 mriedem sean-k-mooney: which?
21:06:24 sean-k-mooney https://review.opendev.org/#/c/657016/
21:06:38 sean-k-mooney its from monday where cdent commented
21:08:05 mriedem i think i wrote that at the ptg,
21:08:09 mriedem but it's a follow on fix to https://review.opendev.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80
21:08:12 mriedem which isn't so new
21:08:32 sean-k-mooney oh may 3rd not june 3rd
21:08:39 sean-k-mooney i miss read teh commit date
21:08:43 mriedem so we used to always orphan the rps when when deleting a compute service,
21:09:03 mriedem then we tried to delete the rps, but if it failed we still deleted the copute service and didn't try all nodes in the case of ironic,
21:09:14 mriedem now we realize that we can still fail to delete the rps b/c of evacs and unconfirmed migrations
21:09:20 mriedem so not the worst, but still busted...
21:09:46 mriedem meanwhile, let's pile a bunch of new code into nova yay wee!!!!
21:10:23 sean-k-mooney hehe well stephen is trying to delete a tone of it too :)

Earlier   Later