Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-05
20:13:52 efried did you check placement for allocations?
20:14:08 cfriesen mriedem: this leaves us with a stale resource_provider entry, whose UUID ends up out of sync with the nova compute_nodes table, which causes scheduling to fail when we add the node back with the same name
20:14:28 cfriesen efried: checking that now
20:14:52 sean-k-mooney cfriesen: there could be allocation related to stale migrations against that node
20:15:04 efried is this what nova manage heal allocations is for?
20:15:14 efried (however that's spelled)
20:16:25 sean-k-mooney im not sure
20:16:37 cfriesen efried: I see 14 allocations for the stale node for resource classes 0, 1, 2. (cpu/ram/disk, I think)
20:16:38 sean-k-mooney i dont think so but it may be a sideffect
20:17:04 efried cfriesen: Good to know placement wasn't lying.
20:17:10 sean-k-mooney cfriesen: do the allcoation consume ids match any instance or migrations
20:18:07 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using Q35 machine type https://review.opendev.org/663011
20:18:08 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887
20:20:19 cfriesen sean-k-mooney: they match 6 instances that are currently running (they had been running on the node that I deleted)
20:21:07 mriedem cfriesen: kvm or ironic service?
20:21:16 cfriesen mriedem: kvm
20:21:54 mriedem heal_allocations is for adding missing allocations to a provider that has instances on it
20:22:09 sean-k-mooney cfriesen: ok so placement still thinks they are running on the wrong node
20:22:42 cfriesen this is coming from a customer test, but my way of reproducing is to have instances running, then power off the node uncleanly, and then try to delete the node while the automated tools are evacuating the instances that had been running on it.
20:23:11 mriedem as sean-k-mooney said, there are probably allocations from migration records
20:23:13 sean-k-mooney heal_allocations would add allcotion for the node the vms are now running on but does it also clean up allocation where the vms are nolonger running
20:23:17 mriedem b/c you're deleting during a resize or something
20:23:42 cfriesen sean-k-mooney: nope. there are allocations with the same consumer id but two differnet resource_provider_id
20:24:41 mriedem https://bugs.launchpad.net/nova/+bug/1793569
20:24:41 cfriesen mriedem: would there be allocations from the evacuate?
20:24:43 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
20:25:16 mriedem cfriesen: i don't think there would be migration consumer allocatoins during an evac,
20:25:26 mriedem but there could be allocations for the instances that were on that evacuated host
20:25:45 sean-k-mooney i dont think there is either but we should claim resouce on the destiation
20:25:47 mriedem when the old host is brought back, it should clean up those evacuated instance allocations on startup
20:25:47 cfriesen they would have all been live-migrated onto that host not too long before I killed the node
20:26:38 mriedem gibi recently had something merged for evacuate that sounds related https://review.opendev.org/#/c/512623/
20:28:10 sean-k-mooney oh ya i rember looking at this a few days ago but not having time to review
20:32:00 sean-k-mooney mriedem: was this the patch you were asking about performace of "x in set()" vs "x in dict()"
20:32:00 mriedem cfriesen: see my note on https://review.opendev.org/#/c/657070/2/nova/scheduler/client/report.py as well
20:32:12 mriedem sean-k-mooney: no
20:32:24 sean-k-mooney ok
20:32:36 mriedem anywya, we don't have migration consumers for evac
20:33:05 sean-k-mooney we dont but we do claim an allcoation candate for teh destination node
20:33:25 sean-k-mooney but i dont know if we free the allcoation of the source node in all cases
20:33:54 mriedem correct the scheduler will claim (or conductor if you're forcing the dest host) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L923
20:34:03 mriedem the source host allocations should be cleaned up when the service is restarted
20:34:05 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422
20:34:13 cfriesen when evacuating an instance, when are the allocations on the old node supposed to be cleaned up?
20:34:27 mriedem in here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L620
20:34:34 mriedem on restart of the service ^
20:34:54 cfriesen okay, so if we never restart the service they'll just stay stale in the allocations table
20:35:00 mriedem right,
20:35:03 sean-k-mooney cfriesen: when you start up the compute agent on the souce node after fixing it meaning you wont be able to delete the RP until after that
20:35:05 mriedem but presumably that node is dead and fenced anyway
20:35:09 mriedem so you're not scheduling to it
20:35:36 sean-k-mooney mriedem: i think cfriesen was trying to remove teh RP before repovisoing the fixed node
20:35:36 cfriesen yeah...the problem is that those allocations prevent us from deleting the resource_provider on compute node deletion
20:36:02 sean-k-mooney cfriesen: why are you delete in the compute node recored out of interest
20:36:25 cfriesen customer is deleting the node. no idea why. testing functionality?
20:36:32 sean-k-mooney if you are replacing the node with another and reusing the hostname it should not be needed
20:36:40 sean-k-mooney ah ok
20:36:50 sean-k-mooney customer always break things
20:36:52 cfriesen when we add the new node back in with a different hostname it's all fine
20:37:28 cfriesen if we reuse the same hostname we get a different UUID in the compute_node table, but we don't get a new entry in the resource_provider table, so the UUIDs don't match and we never schedule anything to the node
20:37:28 mriedem the shitty thing i didn't realize until cdent pointed it out in an open review is that https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2183 won't raise if deleting the provider fails, which means we'll continue to delete the compute service record
20:37:31 sean-k-mooney cfriesen: if you add it with the same host name it would be fine too just dont delete the compute node recored and it will fix everything up when the compute agent starts
20:37:58 cfriesen sean-k-mooney: the compute node record is deleted by running the "delete service" api
20:38:19 sean-k-mooney cfriesen: yes. which you dont need to do to replace teh server
20:38:48 mriedem cfriesen: https://bugs.launchpad.net/nova/+bug/1829479 could also be related to your issue
20:38:49 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete]
20:38:53 sean-k-mooney and if you dont delete the compute service record you will get the same uuid when the new agent conencts
20:40:17 cfriesen sean-k-mooney: I suspect you're right as a workaround. but it *should* work in an ideal world. :)
20:40:51 cfriesen mriedem: yeah, that looks like what I'm seeing
20:41:35 sean-k-mooney cfriesen: we could "fix it" by having the compute node when it fails to find a compute node record by hostname first check if there is a placemetn RP by looking it up by hostname and then reuse the uuid of the RP which was the old compute node uuid
20:41:41 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add functional test for admin_password https://review.opendev.org/663422
20:42:53 cfriesen sean-k-mooney: would work but seems kind of kludgy. can we delete allocations when deleting a resource_provider?
20:43:35 sean-k-mooney we current dont allow you to delete inventoryes if they have allocatin intentionally
20:43:40 cfriesen I think this code might even have "cascade=true"
20:44:16 sean-k-mooney cfriesen: im pretty sure we intentnioally dont use cascade=true
20:44:30 cfriesen sean-k-mooney: it's in here, actually
20:44:41 mriedem cfriesen: ack i left a comment in https://bugs.launchpad.net/nova/+bug/1829479
20:44:42 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,Incomplete]
20:45:05 mriedem cfriesen: we do try to delete the allocations,
20:45:12 mriedem the problem in your case is it's doing it by instances.host,
20:45:20 mriedem which if you've evacuated those instances, the instances.host points at the new dest compute
20:45:25 mriedem but the old allocations are still there
20:45:43 mriedem this could be easily recreated in a functional test i think
20:46:17 mriedem 1. create server on host1, 2. force down and evacuate from host1, 3. delete compute service for host1, 4. assert provider and allocs still exist for host1 in placement
20:46:30 mriedem and 5. restart host1 service should blow up
20:46:43 cfriesen ah, that makes sense
20:46:43 mriedem b/c of the unique constraint on the provider name
20:48:49 mriedem so https://github.com/openstack/nova/blob/653515a45032811b6bc2f1d0fb651472005496ec/nova/scheduler/client/report.py#L2173 is really just not right for (1) in-progress / unconfirmed migrations and (2) evacuated hosts
20:49:26 cfriesen agreed
20:49:55 mriedem i could start a rolling ball with said functional test to at least recreate the issue so we can throw fixes at it
20:49:56 sean-k-mooney cfriesen: just looking at the placmenet fuctional test tehy expresly check that deleteing allocated resouce provider fails just an fyi https://github.com/openstack/placement/blob/master/placement/tests/functional/db/test_resource_provider.py#L363-L367
20:50:54 cfriesen sean-k-mooney: yeah, it's valid that it fails when there are allocations, but setting "cascade=false" is supposed to delete those allocations. except it doesn't
20:52:02 sean-k-mooney when calling delete_resource_provider
20:52:30 mriedem cfriesen: you mean cascade=true
20:52:32 sean-k-mooney initally i assumed when you said cascade=true you were refering to the db schema
20:52:42 cfriesen bah, yes. cascade=true
20:52:44 mriedem looks like that code was written specifically for an ironic case https://review.opendev.org/#/c/428375/6/nova/scheduler/client/report.py@771
20:52:52 mriedem so not really thinking about migrations and evacuates and suh
20:52:54 mriedem *such
20:53:46 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Spec to pre-filter disabled computes with placement https://review.opendev.org/657884

Earlier   Later