Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-16
19:58:27 mriedem similarly, if we can't unbind the port, we log an exception but don't reraise it https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L563
19:58:43 mrjk mriedem, ok so I guess I can't really solve this issue then, but I'm still annoyed because users report errors now. Increasing service_down_time has a wider impact than just the scheduler, right?
19:59:03 cfriesen anyone else find it disconcerting that _shutdown_instance() is closer to "destroy/delete" than just "shutdown"?
19:59:17 imacdonn hmm that log message for the port really should be LOG.warn(), not debug()
19:59:37 mriedem mrjk: yes service_down_time is used per service,
19:59:52 mriedem mrjk: if you have your control services running on different hosts, then you could have a separate config value for them
20:00:21 mriedem mrjk: if you are going to allow your users to have a high enough quota to create lots of instances in a single request, and scheduler/conductor is taking too longer, then you need to scale something out
20:00:24 mrjk hmm, it's a bit hackish, but I will definitely consider this
20:00:26 mriedem maybe conductor
20:00:44 imacdonn also, a port not existing is different from a "something went badly wrong" exception, IMO
20:01:14 mriedem imacdonn: it's not logging an exception on port not found
20:01:15 mrjk I'll continue to investigate on conductor, to see if I see more stuffs
20:01:21 mrjk Thx for your help
20:01:21 mriedem it doesn't care about the port not found b/c it's trying to unbind it
20:01:40 mriedem if the actual port update fails with a 400 or 500 or something, it logs an exception trace
20:01:42 mriedem but keeps going
20:01:52 mriedem like i said, the guest is gone by this point
20:01:53 fried_rice jroll, TheJulia: My investigation stalled at "Yup, you tried to create a provider with the same name but a different UUID."
20:01:57 imacdonn oh, right, yeah
20:01:59 mriedem so if volume/port cleanup fails,
20:02:10 mriedem the guest in the hypervisor is already gone, and you have manual cleanup to do in cinder/neutron
20:02:36 mriedem if there are other better historical reasons for this, i'm hoping maybe leakypipes or superdan can chime in
20:03:20 smcginnis And if the cinder or neutron failure was due to that volume or port being deleted externally, you wouldn't want your broken instances stuck.
20:04:21 imacdonn I'm actually more concerned about the left-behind SCSI/multipath devices on the compute node ... we actually monitor for that, because it's caused us a lot of pain .. it may be less painful with Gorka's work on os-brick, but it's still messy to leave that stuff laying around
20:04:34 openstackgerrit Merged openstack/nova master: api-ref: Further clarify placement aggregates https://review.openstack.org/545356
20:04:53 openstackgerrit Merged openstack/nova master: Fix and update compute schedulers config guide https://review.openstack.org/544010
20:05:15 openstackgerrit Merged openstack/nova master: Fix warn api_class is deprecated, use backend https://review.openstack.org/543830
20:05:18 superdan lyarwood is probably your man for understanding the multipath residue
20:05:23 superdan I definitely don't have much context on that
20:05:37 imacdonn if nova's going to take the ostrich approach to cinder failures, perhaps it should at least clean up the os-brick stuff .... I suppose I should try to confirm that it doesn't do that with the latest code .. the case I diagnosed was on Ocata
20:06:51 mriedem imacdonn: anything to do with os-brick,
20:06:57 mriedem would be when driver.destroy is called to delete the guest
20:07:03 mriedem which presumably didn't fail
20:07:32 mriedem right here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1055
20:08:29 imacdonn hmmm
20:08:39 imacdonn something failed, because the devices were left behind
20:08:51 mriedem the thing you pointed out earlier, was when nova detaches the volume on the cinder side
20:08:55 mriedem not the compute host
20:09:20 mriedem well, calls terminate_connection to remove the export
20:09:30 mriedem detach_volume in cinder is just updating the volume status to 'available'
20:09:53 mriedem multipath shenanigans would be in os-brick when the libvirt driver calls disconnect_volume
20:10:20 mriedem btw, i don't know what an SA is
20:10:23 mriedem except super america
20:10:24 mriedem gas station
20:10:37 imacdonn heh .. sys admin
20:13:32 imacdonn seems like I'm going to have to try to reproduce this .... when I got to the case I diagnosed, the broken devices had already been cleaned up manually ... I only have non-debug logs to go on
20:15:32 imacdonn actually... just had a thought ... it may be that the devices DID get unplumbed, but then a subsequent instance creation caused an iSCSI rescan, and they were rediscovered .. if that's the case, Gorka's os-brick work (in Pike) should solve that
20:15:46 mriedem imacdonn: if os-brick failed we should have a warning from https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1064
20:18:34 imacdonn ("Gorka's work" referring to https://review.openstack.org/#/c/445943/)
20:19:06 imacdonn or whatever that turned into .. that doesn't look like the right review
20:20:15 imacdonn https://review.openstack.org/#/c/433104/
20:22:20 imacdonn given that, it's not so bad .... I still think it's kinda not great to leave cinder believing that the volumes are attached to instances that don't exist, but I think I can live with it
20:48:46 TheJulia fried_rice: jroll: In that case, it almost sounds like the list of nodes being serviced by that individual compute node is out of sync... and such a constraint almost sounds like we can't have multiple nova-compute processes running at all 8(
20:50:19 fried_rice TheJulia: As long as your compute node RPs are actually the same, you should be fine. Why aren't they being created with the same name & uuid?
20:53:20 fried_rice TheJulia: In resource tracker _update, the compute_node param has a UUID and a name. The report client is using both of them to _ensure_resource_provider.
20:53:22 TheJulia so, I guess then I have a data model question due to a lack of understanding on my part. is the uuid ironic's baremetal node uuid that we are talking about? Because I thought based on comments yesterday that it was getting posted as the name of the entry and the uuid was the actual nova-compute node?
20:54:33 fried_rice TheJulia: This is where I'm also pretty confused. But it seems to me like, whichever is the case, its uuid/name should be immutable.
20:54:46 TheJulia ugh
20:54:49 fried_rice Especially since the name is a UUID (which is even more confusing)
20:55:03 TheJulia if it is the compute node uuid.... then.... we have lost functionality
20:55:38 fried_rice We're creating the RP in placement with compute_node.uuid, compute_node.hypervisor_hostname
20:56:00 TheJulia so they can never fail between compute nodes.... heh
20:56:09 fried_rice Under what circumstances could I get a compute_node with the same hypervisor_hostname but a *different* uuid?
20:56:32 TheJulia if a compute process went down too long and the hash ring recalculated
20:56:51 TheJulia the other compute process would take over for the other nodes
20:57:14 TheJulia or at least, that is what we were doing as i understand it
20:57:52 fried_rice Cool cool. But why doesn't the second compute process perceive the nodes as having the same UUID as the first compute process did?
20:58:30 TheJulia and now I'm confused :)
21:00:34 fried_rice TheJulia: resource_tracker.py in the 900s
21:00:41 fried_rice sorry, 500s
21:00:50 TheJulia thanks
21:01:06 fried_rice In particular, I thought I4253cffca3dbf558c875eed7e77711a31e9e3406 was supposed to be attacking this very problem.
21:06:22 fried_rice TheJulia: I think I get it. Is the hypervisor_hostname is assigned based on compute_node.host?
21:06:59 TheJulia that is what I think, it is just I've never dug through this portion of nova, so I'm not grasping it very well
21:07:24 TheJulia also *squirrel* *blink* *blink*
21:07:56 fried_rice mriedem: Executive summary on the difference between nodename and hypervisor_hostname?
21:08:55 jroll in ironic-land, nodename == ironic node uuid, hypervisor_hostname == nova-compute hostname (rabbit queue)
21:09:11 fried_rice Aha
21:09:15 fried_rice I think therein lies the boggle.
21:09:27 fried_rice Because in non-ironic-land, I *think* they are the same.
21:09:33 fried_rice And we're using the wrong one.
21:09:50 jroll the patch you mentioned was meant to handle this, but I think only for the compute_nodes table, not resource providers
21:10:58 jroll (emphasis on I think)
21:11:11 fried_rice L869
21:11:38 jroll nice.
21:12:05 mriedem for non-ironic, compute_nodes.host and compute_nodes.hypervisor_hostname are the same
21:13:14 jroll wait, 869 can't be wrong, we would have never had any resources
21:13:24 jroll that would have blown up all over the place
21:13:31 jroll did I have this backwards?
21:13:41 TheJulia hmmmmmmmm
21:15:46 jroll fried_rice: I'm sorry, I lied. hypervisor_hostname is the ironic node uuid
21:16:09 jroll compute_node.host is the nova-compute hostname
21:16:36 jroll your use of nodename threw me off, that is equivalent in (most? all?) places to hypervisor_hostname
21:17:10 fried_rice So then we go back to the original question: How does compute_node.uuid *change* when compute_node.hypervisor_hostname is the same?
21:18:09 jroll compute_node.uuid does not
21:18:14 jroll but resource provider uuid does
21:18:16 jroll afaict
21:18:40 fried_rice No, because the error is happening via _ensure_resource_provider(compute_node.uuid, compute_node.hypervisor_hostname)
21:18:46 fried_rice and we're running into a conflict on the latter.
21:19:32 jroll is resource provider UUID always the same as compute node uuid?

Earlier   Later