Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-05
20:38:49 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete]
20:38:53 sean-k-mooney and if you dont delete the compute service record you will get the same uuid when the new agent conencts
20:40:17 cfriesen sean-k-mooney: I suspect you're right as a workaround. but it *should* work in an ideal world. :)
20:40:51 cfriesen mriedem: yeah, that looks like what I'm seeing
20:41:35 sean-k-mooney cfriesen: we could "fix it" by having the compute node when it fails to find a compute node record by hostname first check if there is a placemetn RP by looking it up by hostname and then reuse the uuid of the RP which was the old compute node uuid
20:41:41 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422
20:42:53 cfriesen sean-k-mooney: would work but seems kind of kludgy. can we delete allocations when deleting a resource_provider?
20:43:35 sean-k-mooney we current dont allow you to delete inventoryes if they have allocatin intentionally
20:43:40 cfriesen I think this code might even have "cascade=true"
20:44:16 sean-k-mooney cfriesen: im pretty sure we intentnioally dont use cascade=true
20:44:30 cfriesen sean-k-mooney: it's in here, actually
20:44:41 mriedem cfriesen: ack i left a comment in https://bugs.launchpad.net/nova/+bug/1829479
20:44:42 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete]
20:45:05 mriedem cfriesen: we do try to delete the allocations,
20:45:12 mriedem the problem in your case is it's doing it by instances.host,
20:45:20 mriedem which if you've evacuated those instances, the instances.host points at the new dest compute
20:45:25 mriedem but the old allocations are still there
20:45:43 mriedem this could be easily recreated in a functional test i think
20:46:17 mriedem 1. create server on host1, 2. force down and evacuate from host1, 3. delete compute service for host1, 4. assert provider and allocs still exist for host1 in placement
20:46:30 mriedem and 5. restart host1 service should blow up
20:46:43 mriedem b/c of the unique constraint on the provider name
20:46:43 cfriesen ah, that makes sense
20:48:49 mriedem so https://github.com/openstack/nova/blob/653515a45032811b6bc2f1d0fb651472005496ec/nova/scheduler/client/report.py#L2173 is really just not right for (1) in-progress / unconfirmed migrations and (2) evacuated hosts
20:49:26 cfriesen agreed
20:49:55 mriedem i could start a rolling ball with said functional test to at least recreate the issue so we can throw fixes at it
20:49:56 sean-k-mooney cfriesen: just looking at the placmenet fuctional test tehy expresly check that deleteing allocated resouce provider fails just an fyi https://github.com/openstack/placement/blob/master/placement/tests/functional/db/test_resource_provider.py#L363-L367
20:50:54 cfriesen sean-k-mooney: yeah, it's valid that it fails when there are allocations, but setting "cascade=false" is supposed to delete those allocations. except it doesn't
20:52:02 sean-k-mooney when calling delete_resource_provider
20:52:30 mriedem cfriesen: you mean cascade=true
20:52:32 sean-k-mooney initally i assumed when you said cascade=true you were refering to the db schema
20:52:42 cfriesen bah, yes. cascade=true
20:52:44 mriedem looks like that code was written specifically for an ironic case https://review.opendev.org/#/c/428375/6/nova/scheduler/client/report.py@771
20:52:52 mriedem so not really thinking about migrations and evacuates and suh
20:52:54 mriedem *such
20:53:46 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Spec to pre-filter disabled computes with placement https://review.opendev.org/657884
20:53:47 mriedem dansmith: got that spec updated ^
20:55:28 sean-k-mooney i would need to check the placmeent api to confim but the allocation table has teh RP it was allcaoed from so we should be able to simply look up all the allocation for a given rp then delete tehm assuming that is exposed at the api level
20:55:34 sean-k-mooney which i proably is not
20:56:08 icarusfactor Can I have more than one compute_driver for nova.conf or is it limited to only one?
20:56:36 sean-k-mooney ya its not
20:57:39 sean-k-mooney so we need to add a new endpoint which is /allocations?rp_uuid=XYZ to be be able to do this cleanly
20:58:15 sean-k-mooney icarusfactor: its one per nova.conf
20:58:29 sean-k-mooney icarusfactor: but you can have multiple agent on the same host in some cases
20:58:44 sean-k-mooney icarusfactor: are you trying to run ironc and libvirt on teh same host?
20:59:15 icarusfactor sean-k-mooney, Nifty , I'm running libvirt currently. Was wondering how to do that.
20:59:50 sean-k-mooney its generally not a great idea to do but if you have a second nova.conf that you pass to the second agent you can make it work
21:00:55 icarusfactor sean-k-mooney, Just trying and experimenting with different methods , want to use the Stein docker nova driver.
21:00:56 sean-k-mooney normally i would recommend running the ironic driver assuming that is the other one you want to use on one of your contoler instead of colcating it on a libvirt compute node
21:01:17 mriedem sean-k-mooney: right i asked that here https://review.opendev.org/#/c/428375/6/nova/scheduler/client/report.py@771
21:01:20 mriedem but never got a response
21:02:22 mriedem we can get the allocations for a given provider with this API https://developer.openstack.org/api-ref/placement/?expanded=list-resource-provider-allocations-detail#list-resource-provider-allocations
21:02:25 sean-k-mooney icarusfactor: i had thought that dirver nova docker driver was nolonger mainatined
21:03:03 sean-k-mooney mriedem: oh off the RP endpoint
21:03:10 sean-k-mooney cool ya that seams more sane
21:03:20 icarusfactor sean-k-mooney, Humm, that is what I was wondering , the Dashboard now shows the docker option.
21:03:22 sean-k-mooney at least that way we cant get out of sync
21:03:44 sean-k-mooney icarusfactor: dashboard as in horizon?
21:03:47 icarusfactor Seeing it in Stien , I was wanting to try this option.
21:03:50 icarusfactor yes
21:04:05 sean-k-mooney icarusfactor: could that be form zun?
21:04:11 mriedem https://review.opendev.org/#/c/657016/ is my change where cdent pointed out that failure to delete the rp doesn't make us fail to delete the compute service
21:04:16 mriedem finally connecting all the dots
21:04:19 mriedem i'm slow and dumb
21:04:33 icarusfactor sean-k-mooney, Cant answer that.
21:04:59 sean-k-mooney icarusfactor: https://github.com/openstack/nova-docker
21:05:22 sean-k-mooney icarusfactor: it proably related to zun
21:05:55 sean-k-mooney mriedem: oh this is a recent change
21:06:08 icarusfactor sean-k-mooney, ok , will check zun out , that helps as I was going /dev/null fast.
21:06:14 mriedem sean-k-mooney: which?
21:06:24 sean-k-mooney https://review.opendev.org/#/c/657016/
21:06:38 sean-k-mooney its from monday where cdent commented
21:08:05 mriedem i think i wrote that at the ptg,
21:08:09 mriedem but it's a follow on fix to https://review.opendev.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80
21:08:12 mriedem which isn't so new
21:08:32 sean-k-mooney oh may 3rd not june 3rd
21:08:39 sean-k-mooney i miss read teh commit date
21:08:43 mriedem so we used to always orphan the rps when when deleting a compute service,
21:09:03 mriedem then we tried to delete the rps, but if it failed we still deleted the copute service and didn't try all nodes in the case of ironic,
21:09:14 mriedem now we realize that we can still fail to delete the rps b/c of evacs and unconfirmed migrations
21:09:20 mriedem so not the worst, but still busted...
21:09:46 mriedem meanwhile, let's pile a bunch of new code into nova yay wee!!!!
21:10:23 sean-k-mooney hehe well stephen is trying to delete a tone of it too :)
21:10:33 sean-k-mooney but im guessing we shoudl update https://github.com/openstack/nova/blob/653515a45032811b6bc2f1d0fb651472005496ec/nova/scheduler/client/report.py#L2173
21:10:52 sean-k-mooney to get teh allcotion form the RP via teh placmeent api right
21:11:11 mriedem stephen is deleting the stuff no one has been using for several years sure :)
21:11:12 sean-k-mooney and then if we combind that with your change for ironic that might do the right thing
21:11:30 mriedem i need to write the functional recreate test first before we start throwing fixes at it,
21:11:42 mriedem because there are several issues here i think and i need to start with the tests
21:13:19 sean-k-mooney sure but https://review.opendev.org/#/c/657016/2/nova/api/openstack/compute/services.py is calling delete_resource_provider
21:13:40 mriedem i know
21:13:59 mriedem ^ is no worse off than what we have today, it just handles 1:M ironic nodes,
21:14:07 sean-k-mooney so if we fix and test delete_resource_provider for a singel provider then we can layer your other fix on top.
21:14:15 sean-k-mooney yes
21:14:15 mriedem cdent's -1 is that the commit message is asserting that if we fail to delete the rp we won't delete the compute service which is false
21:14:40 sean-k-mooney i think they shoudl stay seperate patches too because its really two sperate problems
21:15:00 mriedem so before refactoring delete_resource_provider and how that works for allocations, i would need to think about whether or not those should be separate changes (i think they should for sanity)
21:15:12 mriedem right
21:15:57 mriedem but right now i have familial duties so can't work on this until tomorrow
21:16:29 sean-k-mooney mriedem: night o/ if you want me to try on work on part of this let me know
21:16:46 sean-k-mooney i need to go finish some spec reviews from yesterday

Earlier   Later