Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-16
17:39:51 cfriesen mriedem: I think so, but that still means it could hit the first instance that wants that image.
17:41:01 mriedem on the dest host
17:41:09 mriedem the image would be cached on the source host but that doesn't help you
17:41:30 mriedem you guys don't use ceph?
17:41:37 cfriesen mriedem: up to the end user
17:41:48 mriedem ok; figured with all of the live migration you seem to do, you'd push ceph
17:41:58 cfriesen mriedem: we support compute nodes with ceph, with local qcow2, and with local thin LVM
17:42:45 cfriesen mriedem: some installs are really small, like 2 all-in-one nodes
17:48:55 cfriesen mriedem: looking at my notes, that patch cut the neutron load by a factor of 3 and reduced lock contention in nova. We also had an oslo.lockutils change to introduce fair locks--they merged it but then it seemed to cause mysterious issues in the CI for a couple of projects so they reverted it.
17:49:39 mriedem how many ports on that instance?
17:49:41 mriedem in that test?
17:50:16 cfriesen actually, wait, that factor of 3 reduction was for removing redundant calls for the same instance
17:51:46 cfriesen 16 ports on the instance (our max)
17:57:09 cfriesen mriedem: found some numbers. as of april last year each call to get_instance_nw_info() cost roughly 200ms plus about 125ms per port. And without that patch there are O(N) network-changed events in a live-migration (where N is the number of interfaces), so the overall cost is O(N^2)
17:57:17 cfriesen with that patch, it drops to O(N)
17:58:37 mriedem cool
17:58:48 mriedem_lunch i'll feast on those results at lunch
18:30:02 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: Trivial: Update help of enabled_filters https://review.openstack.org/545431
18:31:23 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: Trivial: Update help of enabled_filters https://review.openstack.org/545431
19:06:06 mrjk_ mriedem, ok, thank you for your hints
19:08:17 mnaser if nova creates neutron ports when it boots an instance, and the port is manually detached with nova instance-detach, the port is deleted (as per preserve_on_delete=False).. anyone else feel this behaviour is bleh?
19:08:29 mnaser if i detach a port manually, it means i want to use it
19:15:11 imacdonn hmm, interesting .... it could be argued that if you want to make your ports available for use with other instances, you should pre-create them separately
19:18:22 imacdonn assuming that, perhaps you shouldn't be allowed to detach a port that was created along with the instance (thinking out loud)
19:29:18 TheJulia jroll: do you, or anyone else, remember if there was any outcome from the duplicate placement issues we encountered yesterday with ironic's multinode grenade job on queens->master ? It passed without issues once, revised the patch slightly and hit the same issue again
19:29:53 jroll TheJulia: I didn't see anything, fried_rice said he might look into it
19:30:03 jroll I probably won't get bandwidth to investigate today
19:30:23 jroll all I got was that yes, we likely triggered a rebalance of ironic n-cpu things
19:36:39 mrjk_ cfriesen, I'm reading back your comments, "I'd suggest keeping the number of servers in a single boot request small enough" => Google does not answer to this question :)
19:37:30 mrjk_ That means I don't have the control on this issue ? (I was looking for a max cap allowed instance per requests)
19:46:08 imacdonn mriedem: I have another "pool volume handling" situation to run by you ... wondering if it'd be covered by any existing work, or if I should create a new bug for it
19:46:30 mriedem mrjk: there is no limit, besides the user quota, on --max-count instances in a single server create request
19:46:55 openstackgerrit Merged openstack/nova master: Only log during pop retry phase https://review.openstack.org/541655
19:47:00 mriedem mrjk: so if you allow users to have a quota of instances to be 30, and they can create up to 30 instances in a single request, then you have to account for that in your deployment
19:47:13 mriedem imacdonn: pool volume handling?
19:49:12 imacdonn mriedem: SA reported that a volume was attached to an instance that didn't exist .... from log-trawling, I discovered that the volume was attached to an instance, and the instance was terminated while the cinder backend was down - cinder-volume threw a VolumeBackendAPIException, but nova-compute ignored it, and proceeded to delete the instance anyway - https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2404
19:53:25 mriedem imacdonn: and then
19:53:42 imacdonn mriedem: and then we have a volume that's attached to an instance that doesn't exist
19:53:50 mriedem yup
19:53:54 mriedem you'll have to force-detach the volume
19:54:21 imacdonn not only that, but it left behind broken SCSI/multipath devices on the compute node
19:54:38 imacdonn shouldn't the instance termination fail in this case ?
19:55:35 mriedem this is the way it's always worked, i'm not exactly sure why, besides nova just wants to delete the instance
19:56:02 cfriesen mriedem: I can see an argument for not deleting the instance if it means leaving neutron/cinder in a confused state
19:56:45 mriedem well,
19:56:49 mriedem i think nova ignores it,
19:56:54 mriedem because at this point, nova has already destroyed the guest
19:56:55 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2354
19:58:27 mriedem similarly, if we can't unbind the port, we log an exception but don't reraise it https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L563
19:58:43 mrjk mriedem, ok so I guess I can't really solve this issue then, but I'm still annoyed because users report errors now. Increasing service_down_time has a wider impact than just the scheduler, right?
19:59:03 cfriesen anyone else find it disconcerting that _shutdown_instance() is closer to "destroy/delete" than just "shutdown"?
19:59:17 imacdonn hmm that log message for the port really should be LOG.warn(), not debug()
19:59:37 mriedem mrjk: yes service_down_time is used per service,
19:59:52 mriedem mrjk: if you have your control services running on different hosts, then you could have a separate config value for them
20:00:21 mriedem mrjk: if you are going to allow your users to have a high enough quota to create lots of instances in a single request, and scheduler/conductor is taking too longer, then you need to scale something out
20:00:24 mrjk hmm, it's a bit hackish, but I will definitely consider this
20:00:26 mriedem maybe conductor
20:00:44 imacdonn also, a port not existing is different from a "something went badly wrong" exception, IMO
20:01:14 mriedem imacdonn: it's not logging an exception on port not found
20:01:15 mrjk I'll continue to investigate on conductor, to see if I see more stuffs
20:01:21 mrjk Thx for your help
20:01:21 mriedem it doesn't care about the port not found b/c it's trying to unbind it
20:01:40 mriedem if the actual port update fails with a 400 or 500 or something, it logs an exception trace
20:01:42 mriedem but keeps going
20:01:52 mriedem like i said, the guest is gone by this point
20:01:53 fried_rice jroll, TheJulia: My investigation stalled at "Yup, you tried to create a provider with the same name but a different UUID."
20:01:57 imacdonn oh, right, yeah
20:01:59 mriedem so if volume/port cleanup fails,
20:02:10 mriedem the guest in the hypervisor is already gone, and you have manual cleanup to do in cinder/neutron
20:02:36 mriedem if there are other better historical reasons for this, i'm hoping maybe leakypipes or superdan can chime in
20:03:20 smcginnis And if the cinder or neutron failure was due to that volume or port being deleted externally, you wouldn't want your broken instances stuck.
20:04:21 imacdonn I'm actually more concerned about the left-behind SCSI/multipath devices on the compute node ... we actually monitor for that, because it's caused us a lot of pain .. it may be less painful with Gorka's work on os-brick, but it's still messy to leave that stuff laying around
20:04:34 openstackgerrit Merged openstack/nova master: api-ref: Further clarify placement aggregates https://review.openstack.org/545356
20:04:53 openstackgerrit Merged openstack/nova master: Fix and update compute schedulers config guide https://review.openstack.org/544010
20:05:15 openstackgerrit Merged openstack/nova master: Fix warn api_class is deprecated, use backend https://review.openstack.org/543830
20:05:18 superdan lyarwood is probably your man for understanding the multipath residue
20:05:23 superdan I definitely don't have much context on that
20:05:37 imacdonn if nova's going to take the ostrich approach to cinder failures, perhaps it should at least clean up the os-brick stuff .... I suppose I should try to confirm that it doesn't do that with the latest code .. the case I diagnosed was on Ocata
20:06:51 mriedem imacdonn: anything to do with os-brick,
20:06:57 mriedem would be when driver.destroy is called to delete the guest
20:07:03 mriedem which presumably didn't fail
20:07:32 mriedem right here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1055
20:08:29 imacdonn hmmm
20:08:39 imacdonn something failed, because the devices were left behind
20:08:51 mriedem the thing you pointed out earlier, was when nova detaches the volume on the cinder side
20:08:55 mriedem not the compute host
20:09:20 mriedem well, calls terminate_connection to remove the export
20:09:30 mriedem detach_volume in cinder is just updating the volume status to 'available'
20:09:53 mriedem multipath shenanigans would be in os-brick when the libvirt driver calls disconnect_volume
20:10:20 mriedem btw, i don't know what an SA is
20:10:23 mriedem except super america
20:10:24 mriedem gas station
20:10:37 imacdonn heh .. sys admin
20:13:32 imacdonn seems like I'm going to have to try to reproduce this .... when I got to the case I diagnosed, the broken devices had already been cleaned up manually ... I only have non-debug logs to go on
20:15:32 imacdonn actually... just had a thought ... it may be that the devices DID get unplumbed, but then a subsequent instance creation caused an iSCSI rescan, and they were rediscovered .. if that's the case, Gorka's os-brick work (in Pike) should solve that
20:15:46 mriedem imacdonn: if os-brick failed we should have a warning from https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1064
20:18:34 imacdonn ("Gorka's work" referring to https://review.openstack.org/#/c/445943/)
20:19:06 imacdonn or whatever that turned into .. that doesn't look like the right review
20:20:15 imacdonn https://review.openstack.org/#/c/433104/
20:22:20 imacdonn given that, it's not so bad .... I still think it's kinda not great to leave cinder believing that the volumes are attached to instances that don't exist, but I think I can live with it

Earlier   Later