| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-16 | |||
| 20:01:40 | mriedem | if the actual port update fails with a 400 or 500 or something, it logs an exception trace | |
| 20:01:42 | mriedem | but keeps going | |
| 20:01:52 | mriedem | like i said, the guest is gone by this point | |
| 20:01:53 | fried_rice | jroll, TheJulia: My investigation stalled at "Yup, you tried to create a provider with the same name but a different UUID." | |
| 20:01:57 | imacdonn | oh, right, yeah | |
| 20:01:59 | mriedem | so if volume/port cleanup fails, | |
| 20:02:10 | mriedem | the guest in the hypervisor is already gone, and you have manual cleanup to do in cinder/neutron | |
| 20:02:36 | mriedem | if there are other better historical reasons for this, i'm hoping maybe leakypipes or superdan can chime in | |
| 20:03:20 | smcginnis | And if the cinder or neutron failure was due to that volume or port being deleted externally, you wouldn't want your broken instances stuck. | |
| 20:04:21 | imacdonn | I'm actually more concerned about the left-behind SCSI/multipath devices on the compute node ... we actually monitor for that, because it's caused us a lot of pain .. it may be less painful with Gorka's work on os-brick, but it's still messy to leave that stuff laying around | |
| 20:04:34 | openstackgerrit | Merged openstack/nova master: api-ref: Further clarify placement aggregates https://review.openstack.org/545356 | |
| 20:04:53 | openstackgerrit | Merged openstack/nova master: Fix and update compute schedulers config guide https://review.openstack.org/544010 | |
| 20:05:15 | openstackgerrit | Merged openstack/nova master: Fix warn api_class is deprecated, use backend https://review.openstack.org/543830 | |
| 20:05:18 | superdan | lyarwood is probably your man for understanding the multipath residue | |
| 20:05:23 | superdan | I definitely don't have much context on that | |
| 20:05:37 | imacdonn | if nova's going to take the ostrich approach to cinder failures, perhaps it should at least clean up the os-brick stuff .... I suppose I should try to confirm that it doesn't do that with the latest code .. the case I diagnosed was on Ocata | |
| 20:06:51 | mriedem | imacdonn: anything to do with os-brick, | |
| 20:06:57 | mriedem | would be when driver.destroy is called to delete the guest | |
| 20:07:03 | mriedem | which presumably didn't fail | |
| 20:07:32 | mriedem | right here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1055 | |
| 20:08:29 | imacdonn | hmmm | |
| 20:08:39 | imacdonn | something failed, because the devices were left behind | |
| 20:08:51 | mriedem | the thing you pointed out earlier, was when nova detaches the volume on the cinder side | |
| 20:08:55 | mriedem | not the compute host | |
| 20:09:20 | mriedem | well, calls terminate_connection to remove the export | |
| 20:09:30 | mriedem | detach_volume in cinder is just updating the volume status to 'available' | |
| 20:09:53 | mriedem | multipath shenanigans would be in os-brick when the libvirt driver calls disconnect_volume | |
| 20:10:20 | mriedem | btw, i don't know what an SA is | |
| 20:10:23 | mriedem | except super america | |
| 20:10:24 | mriedem | gas station | |
| 20:10:37 | imacdonn | heh .. sys admin | |
| 20:13:32 | imacdonn | seems like I'm going to have to try to reproduce this .... when I got to the case I diagnosed, the broken devices had already been cleaned up manually ... I only have non-debug logs to go on | |
| 20:15:32 | imacdonn | actually... just had a thought ... it may be that the devices DID get unplumbed, but then a subsequent instance creation caused an iSCSI rescan, and they were rediscovered .. if that's the case, Gorka's os-brick work (in Pike) should solve that | |
| 20:15:46 | mriedem | imacdonn: if os-brick failed we should have a warning from https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1064 | |
| 20:18:34 | imacdonn | ("Gorka's work" referring to https://review.openstack.org/#/c/445943/) | |
| 20:19:06 | imacdonn | or whatever that turned into .. that doesn't look like the right review | |
| 20:20:15 | imacdonn | https://review.openstack.org/#/c/433104/ | |
| 20:22:20 | imacdonn | given that, it's not so bad .... I still think it's kinda not great to leave cinder believing that the volumes are attached to instances that don't exist, but I think I can live with it | |
| 20:48:46 | TheJulia | fried_rice: jroll: In that case, it almost sounds like the list of nodes being serviced by that individual compute node is out of sync... and such a constraint almost sounds like we can't have multiple nova-compute processes running at all 8( | |
| 20:50:19 | fried_rice | TheJulia: As long as your compute node RPs are actually the same, you should be fine. Why aren't they being created with the same name & uuid? | |
| 20:53:20 | fried_rice | TheJulia: In resource tracker _update, the compute_node param has a UUID and a name. The report client is using both of them to _ensure_resource_provider. | |
| 20:53:22 | TheJulia | so, I guess then I have a data model question due to a lack of understanding on my part. is the uuid ironic's baremetal node uuid that we are talking about? Because I thought based on comments yesterday that it was getting posted as the name of the entry and the uuid was the actual nova-compute node? | |
| 20:54:33 | fried_rice | TheJulia: This is where I'm also pretty confused. But it seems to me like, whichever is the case, its uuid/name should be immutable. | |
| 20:54:46 | TheJulia | ugh | |
| 20:54:49 | fried_rice | Especially since the name is a UUID (which is even more confusing) | |
| 20:55:03 | TheJulia | if it is the compute node uuid.... then.... we have lost functionality | |
| 20:55:38 | fried_rice | We're creating the RP in placement with compute_node.uuid, compute_node.hypervisor_hostname | |
| 20:56:00 | TheJulia | so they can never fail between compute nodes.... heh | |
| 20:56:09 | fried_rice | Under what circumstances could I get a compute_node with the same hypervisor_hostname but a *different* uuid? | |
| 20:56:32 | TheJulia | if a compute process went down too long and the hash ring recalculated | |
| 20:56:51 | TheJulia | the other compute process would take over for the other nodes | |
| 20:57:14 | TheJulia | or at least, that is what we were doing as i understand it | |
| 20:57:52 | fried_rice | Cool cool. But why doesn't the second compute process perceive the nodes as having the same UUID as the first compute process did? | |
| 20:58:30 | TheJulia | and now I'm confused :) | |
| 21:00:34 | fried_rice | TheJulia: resource_tracker.py in the 900s | |
| 21:00:41 | fried_rice | sorry, 500s | |
| 21:00:50 | TheJulia | thanks | |
| 21:01:06 | fried_rice | In particular, I thought I4253cffca3dbf558c875eed7e77711a31e9e3406 was supposed to be attacking this very problem. | |
| 21:06:22 | fried_rice | TheJulia: I think I get it. Is the hypervisor_hostname is assigned based on compute_node.host? | |
| 21:06:59 | TheJulia | that is what I think, it is just I've never dug through this portion of nova, so I'm not grasping it very well | |
| 21:07:24 | TheJulia | also *squirrel* *blink* *blink* | |
| 21:07:56 | fried_rice | mriedem: Executive summary on the difference between nodename and hypervisor_hostname? | |
| 21:08:55 | jroll | in ironic-land, nodename == ironic node uuid, hypervisor_hostname == nova-compute hostname (rabbit queue) | |
| 21:09:11 | fried_rice | Aha | |
| 21:09:15 | fried_rice | I think therein lies the boggle. | |
| 21:09:27 | fried_rice | Because in non-ironic-land, I *think* they are the same. | |
| 21:09:33 | fried_rice | And we're using the wrong one. | |
| 21:09:50 | jroll | the patch you mentioned was meant to handle this, but I think only for the compute_nodes table, not resource providers | |
| 21:10:58 | jroll | (emphasis on I think) | |
| 21:11:11 | fried_rice | L869 | |
| 21:11:38 | jroll | nice. | |
| 21:12:05 | mriedem | for non-ironic, compute_nodes.host and compute_nodes.hypervisor_hostname are the same | |
| 21:13:14 | jroll | wait, 869 can't be wrong, we would have never had any resources | |
| 21:13:24 | jroll | that would have blown up all over the place | |
| 21:13:31 | jroll | did I have this backwards? | |
| 21:13:41 | TheJulia | hmmmmmmmm | |
| 21:15:46 | jroll | fried_rice: I'm sorry, I lied. hypervisor_hostname is the ironic node uuid | |
| 21:16:09 | jroll | compute_node.host is the nova-compute hostname | |
| 21:16:36 | jroll | your use of nodename threw me off, that is equivalent in (most? all?) places to hypervisor_hostname | |
| 21:17:10 | fried_rice | So then we go back to the original question: How does compute_node.uuid *change* when compute_node.hypervisor_hostname is the same? | |
| 21:18:09 | jroll | compute_node.uuid does not | |
| 21:18:14 | jroll | but resource provider uuid does | |
| 21:18:16 | jroll | afaict | |
| 21:18:40 | fried_rice | No, because the error is happening via _ensure_resource_provider(compute_node.uuid, compute_node.hypervisor_hostname) | |
| 21:18:46 | fried_rice | and we're running into a conflict on the latter. | |
| 21:19:32 | jroll | is resource provider UUID always the same as compute node uuid? | |
| 21:19:51 | fried_rice | For compute node resource providers, in Queens, yes. | |
| 21:19:56 | fried_rice | Uhm. | |
| 21:19:59 | fried_rice | Yes. | |
| 21:20:07 | fried_rice | Was gonna say get_inventory might be able to muck with it, but no. | |
| 21:20:19 | fried_rice | Is this because nova is creating the ComputeNode entries afresh, with an autogenerated UUID? | |
| 21:20:40 | jroll | seems like it would be, yes. but that's what https://review.openstack.org/#/c/508555/ should have fixed | |
| 21:20:57 | fried_rice | Exacitically | |
| 21:21:08 | fried_rice | L518-9 in fact | |
| 21:21:20 | jroll | right | |
| 21:23:02 | jroll | we demonstrate that code is running: http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/subnode-2/screen-n-cpu.txt.gz#_Feb_16_17_21_04_613786 | |
| 21:24:05 | fried_rice | I'm sorta guessing you can't change the UUID of a ComputeNode in the db. mriedem? | |
| 21:24:14 | jroll | both compute nodes appear to be attempting to create an RP for that node (9de4d7b4-51c9-4088-99b4-cd648332504e), at different times of course | |
| 21:24:31 | fried_rice | Right; the first one succeeds but the second one barfs | |
| 21:24:33 | fried_rice | right? | |