| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-16 | |||
| 19:37:30 | mrjk_ | That means I don't have the control on this issue ? (I was looking for a max cap allowed instance per requests) | |
| 19:46:08 | imacdonn | mriedem: I have another "pool volume handling" situation to run by you ... wondering if it'd be covered by any existing work, or if I should create a new bug for it | |
| 19:46:30 | mriedem | mrjk: there is no limit, besides the user quota, on --max-count instances in a single server create request | |
| 19:46:55 | openstackgerrit | Merged openstack/nova master: Only log during pop retry phase https://review.openstack.org/541655 | |
| 19:47:00 | mriedem | mrjk: so if you allow users to have a quota of instances to be 30, and they can create up to 30 instances in a single request, then you have to account for that in your deployment | |
| 19:47:13 | mriedem | imacdonn: pool volume handling? | |
| 19:49:12 | imacdonn | mriedem: SA reported that a volume was attached to an instance that didn't exist .... from log-trawling, I discovered that the volume was attached to an instance, and the instance was terminated while the cinder backend was down - cinder-volume threw a VolumeBackendAPIException, but nova-compute ignored it, and proceeded to delete the instance anyway - https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2404 | |
| 19:53:25 | mriedem | imacdonn: and then | |
| 19:53:42 | imacdonn | mriedem: and then we have a volume that's attached to an instance that doesn't exist | |
| 19:53:50 | mriedem | yup | |
| 19:53:54 | mriedem | you'll have to force-detach the volume | |
| 19:54:21 | imacdonn | not only that, but it left behind broken SCSI/multipath devices on the compute node | |
| 19:54:38 | imacdonn | shouldn't the instance termination fail in this case ? | |
| 19:55:35 | mriedem | this is the way it's always worked, i'm not exactly sure why, besides nova just wants to delete the instance | |
| 19:56:02 | cfriesen | mriedem: I can see an argument for not deleting the instance if it means leaving neutron/cinder in a confused state | |
| 19:56:45 | mriedem | well, | |
| 19:56:49 | mriedem | i think nova ignores it, | |
| 19:56:54 | mriedem | because at this point, nova has already destroyed the guest | |
| 19:56:55 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2354 | |
| 19:58:27 | mriedem | similarly, if we can't unbind the port, we log an exception but don't reraise it https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L563 | |
| 19:58:43 | mrjk | mriedem, ok so I guess I can't really solve this issue then, but I'm still annoyed because users report errors now. Increasing service_down_time has a wider impact than just the scheduler, right? | |
| 19:59:03 | cfriesen | anyone else find it disconcerting that _shutdown_instance() is closer to "destroy/delete" than just "shutdown"? | |
| 19:59:17 | imacdonn | hmm that log message for the port really should be LOG.warn(), not debug() | |
| 19:59:37 | mriedem | mrjk: yes service_down_time is used per service, | |
| 19:59:52 | mriedem | mrjk: if you have your control services running on different hosts, then you could have a separate config value for them | |
| 20:00:21 | mriedem | mrjk: if you are going to allow your users to have a high enough quota to create lots of instances in a single request, and scheduler/conductor is taking too longer, then you need to scale something out | |
| 20:00:24 | mrjk | hmm, it's a bit hackish, but I will definitely consider this | |
| 20:00:26 | mriedem | maybe conductor | |
| 20:00:44 | imacdonn | also, a port not existing is different from a "something went badly wrong" exception, IMO | |
| 20:01:14 | mriedem | imacdonn: it's not logging an exception on port not found | |
| 20:01:15 | mrjk | I'll continue to investigate on conductor, to see if I see more stuffs | |
| 20:01:21 | mrjk | Thx for your help | |
| 20:01:21 | mriedem | it doesn't care about the port not found b/c it's trying to unbind it | |
| 20:01:40 | mriedem | if the actual port update fails with a 400 or 500 or something, it logs an exception trace | |
| 20:01:42 | mriedem | but keeps going | |
| 20:01:52 | mriedem | like i said, the guest is gone by this point | |
| 20:01:53 | fried_rice | jroll, TheJulia: My investigation stalled at "Yup, you tried to create a provider with the same name but a different UUID." | |
| 20:01:57 | imacdonn | oh, right, yeah | |
| 20:01:59 | mriedem | so if volume/port cleanup fails, | |
| 20:02:10 | mriedem | the guest in the hypervisor is already gone, and you have manual cleanup to do in cinder/neutron | |
| 20:02:36 | mriedem | if there are other better historical reasons for this, i'm hoping maybe leakypipes or superdan can chime in | |
| 20:03:20 | smcginnis | And if the cinder or neutron failure was due to that volume or port being deleted externally, you wouldn't want your broken instances stuck. | |
| 20:04:21 | imacdonn | I'm actually more concerned about the left-behind SCSI/multipath devices on the compute node ... we actually monitor for that, because it's caused us a lot of pain .. it may be less painful with Gorka's work on os-brick, but it's still messy to leave that stuff laying around | |
| 20:04:34 | openstackgerrit | Merged openstack/nova master: api-ref: Further clarify placement aggregates https://review.openstack.org/545356 | |
| 20:04:53 | openstackgerrit | Merged openstack/nova master: Fix and update compute schedulers config guide https://review.openstack.org/544010 | |
| 20:05:15 | openstackgerrit | Merged openstack/nova master: Fix warn api_class is deprecated, use backend https://review.openstack.org/543830 | |
| 20:05:18 | superdan | lyarwood is probably your man for understanding the multipath residue | |
| 20:05:23 | superdan | I definitely don't have much context on that | |
| 20:05:37 | imacdonn | if nova's going to take the ostrich approach to cinder failures, perhaps it should at least clean up the os-brick stuff .... I suppose I should try to confirm that it doesn't do that with the latest code .. the case I diagnosed was on Ocata | |
| 20:06:51 | mriedem | imacdonn: anything to do with os-brick, | |
| 20:06:57 | mriedem | would be when driver.destroy is called to delete the guest | |
| 20:07:03 | mriedem | which presumably didn't fail | |
| 20:07:32 | mriedem | right here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1055 | |
| 20:08:29 | imacdonn | hmmm | |
| 20:08:39 | imacdonn | something failed, because the devices were left behind | |
| 20:08:51 | mriedem | the thing you pointed out earlier, was when nova detaches the volume on the cinder side | |
| 20:08:55 | mriedem | not the compute host | |
| 20:09:20 | mriedem | well, calls terminate_connection to remove the export | |
| 20:09:30 | mriedem | detach_volume in cinder is just updating the volume status to 'available' | |
| 20:09:53 | mriedem | multipath shenanigans would be in os-brick when the libvirt driver calls disconnect_volume | |
| 20:10:20 | mriedem | btw, i don't know what an SA is | |
| 20:10:23 | mriedem | except super america | |
| 20:10:24 | mriedem | gas station | |
| 20:10:37 | imacdonn | heh .. sys admin | |
| 20:13:32 | imacdonn | seems like I'm going to have to try to reproduce this .... when I got to the case I diagnosed, the broken devices had already been cleaned up manually ... I only have non-debug logs to go on | |
| 20:15:32 | imacdonn | actually... just had a thought ... it may be that the devices DID get unplumbed, but then a subsequent instance creation caused an iSCSI rescan, and they were rediscovered .. if that's the case, Gorka's os-brick work (in Pike) should solve that | |
| 20:15:46 | mriedem | imacdonn: if os-brick failed we should have a warning from https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1064 | |
| 20:18:34 | imacdonn | ("Gorka's work" referring to https://review.openstack.org/#/c/445943/) | |
| 20:19:06 | imacdonn | or whatever that turned into .. that doesn't look like the right review | |
| 20:20:15 | imacdonn | https://review.openstack.org/#/c/433104/ | |
| 20:22:20 | imacdonn | given that, it's not so bad .... I still think it's kinda not great to leave cinder believing that the volumes are attached to instances that don't exist, but I think I can live with it | |
| 20:48:46 | TheJulia | fried_rice: jroll: In that case, it almost sounds like the list of nodes being serviced by that individual compute node is out of sync... and such a constraint almost sounds like we can't have multiple nova-compute processes running at all 8( | |
| 20:50:19 | fried_rice | TheJulia: As long as your compute node RPs are actually the same, you should be fine. Why aren't they being created with the same name & uuid? | |
| 20:53:20 | fried_rice | TheJulia: In resource tracker _update, the compute_node param has a UUID and a name. The report client is using both of them to _ensure_resource_provider. | |
| 20:53:22 | TheJulia | so, I guess then I have a data model question due to a lack of understanding on my part. is the uuid ironic's baremetal node uuid that we are talking about? Because I thought based on comments yesterday that it was getting posted as the name of the entry and the uuid was the actual nova-compute node? | |
| 20:54:33 | fried_rice | TheJulia: This is where I'm also pretty confused. But it seems to me like, whichever is the case, its uuid/name should be immutable. | |
| 20:54:46 | TheJulia | ugh | |
| 20:54:49 | fried_rice | Especially since the name is a UUID (which is even more confusing) | |
| 20:55:03 | TheJulia | if it is the compute node uuid.... then.... we have lost functionality | |
| 20:55:38 | fried_rice | We're creating the RP in placement with compute_node.uuid, compute_node.hypervisor_hostname | |
| 20:56:00 | TheJulia | so they can never fail between compute nodes.... heh | |
| 20:56:09 | fried_rice | Under what circumstances could I get a compute_node with the same hypervisor_hostname but a *different* uuid? | |
| 20:56:32 | TheJulia | if a compute process went down too long and the hash ring recalculated | |
| 20:56:51 | TheJulia | the other compute process would take over for the other nodes | |
| 20:57:14 | TheJulia | or at least, that is what we were doing as i understand it | |
| 20:57:52 | fried_rice | Cool cool. But why doesn't the second compute process perceive the nodes as having the same UUID as the first compute process did? | |
| 20:58:30 | TheJulia | and now I'm confused :) | |
| 21:00:34 | fried_rice | TheJulia: resource_tracker.py in the 900s | |
| 21:00:41 | fried_rice | sorry, 500s | |
| 21:00:50 | TheJulia | thanks | |
| 21:01:06 | fried_rice | In particular, I thought I4253cffca3dbf558c875eed7e77711a31e9e3406 was supposed to be attacking this very problem. | |
| 21:06:22 | fried_rice | TheJulia: I think I get it. Is the hypervisor_hostname is assigned based on compute_node.host? | |
| 21:06:59 | TheJulia | that is what I think, it is just I've never dug through this portion of nova, so I'm not grasping it very well | |
| 21:07:24 | TheJulia | also *squirrel* *blink* *blink* | |
| 21:07:56 | fried_rice | mriedem: Executive summary on the difference between nodename and hypervisor_hostname? | |
| 21:08:55 | jroll | in ironic-land, nodename == ironic node uuid, hypervisor_hostname == nova-compute hostname (rabbit queue) | |
| 21:09:11 | fried_rice | Aha | |
| 21:09:15 | fried_rice | I think therein lies the boggle. | |
| 21:09:27 | fried_rice | Because in non-ironic-land, I *think* they are the same. | |
| 21:09:33 | fried_rice | And we're using the wrong one. | |