Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-04
17:25:20 mnaser also if any stable cores for nova are around, this is pretty useful - https://review.openstack.org/#/c/529385/
17:25:49 mriedem 2. If we're rescheduling (we should know this via filter_properties 'retry' entry), then we need to modify how we swap allocations to only change the allocation for the instance record, and leave the migration allocation on the source node untouched
17:26:31 mriedem edleafe: so i'm thinking 2 separate patches
17:27:20 mriedem edleafe: i could probably wip up a simple regression test for the caching scheduler one and fix in the same patch, which could go below yours
17:27:30 mriedem since resize is just busted with caching scheduler regardless of reschedule
17:27:43 edleafe mriedem: on #2, not following. The source allocs will be the migration, and the dest allocs won't be done because we haven't picked a target host yet
17:28:13 mriedem edleafe: on #2 the problem is when we reschedule right?
17:28:17 edleafe yeah
17:28:33 edleafe it's checking that the instance is allocated against the source
17:28:35 mriedem at the point of the first reschedule, the source node allocs are on the migration record and the failed first chosen host allocs are on the instance
17:28:41 mriedem right, i'm saying,
17:28:47 edleafe on a retry, the allocs on the source will be the migration
17:28:49 mriedem we need to modify the logic that assumes the source node allocs are on the instance,
17:28:55 mriedem if we know we're doing a reschedule,
17:29:01 mriedem and we can determine that based on the filter_properties 'retry' entry
17:29:19 mriedem so on a retry, we don't do anything with the source,
17:29:22 mriedem we just update the dest
17:29:36 mriedem update the instance allocation from failed dest 1 to next dest 2
17:29:39 edleafe mriedem: what did you mean then by "only change the allocation for the instance record"?
17:30:06 mriedem on a reschedule, we move the instance allocation from the failed dest host to the next alternative host
17:30:12 mriedem and don't touch the migration allocation on the source host
17:30:18 edleafe the instance won't be allocated after a fail. Those should be rolled back, no?
17:30:49 edleafe no, the next host isn't selected until after this part of the code is run
17:31:05 edleafe This is happening in _preallocate_migration
17:32:05 mriedem where do we rollback allocations for the instance on a fail?
17:32:08 edleafe When we eventually select a host (either through select_destinations (now) or alternate host (when that last patch merges), that's when we allocate the instance to the target
17:32:40 edleafe mriedem: in compute. Let me look
17:33:36 mriedem delete_allocation_for_failed_resize in prep_resize?
17:33:52 mriedem oh i see, the call to _revert_allocation
17:34:14 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4095
17:34:46 edleafe damn, you're faster than I am :)
17:35:08 mriedem so if the compute reverts the allocations before casting back to conductor to reschedule,
17:35:14 mriedem why is conductor failing on the reschedule
17:35:15 mriedem ?
17:35:34 edleafe Because it is confirming that the instance has allocations on the source
17:35:55 edleafe But at that point on a retry, it's the migration that has allocations on the source
17:35:55 mriedem yeah, which should be the case if we got here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4000
17:36:00 mriedem not if we did ^
17:36:38 mriedem is that failing?
17:37:37 edleafe dunno - all I know is that the instance didn't have allocations on the source. Let me check that out.
17:43:58 mriedem i've created https://bugs.launchpad.net/nova/+bug/1741307 to deal with the caching scheduler + resize stuff
17:43:59 openstack Launchpad bug 1741307 in OpenStack Compute (nova) "Resize always fails when using the CachingScheduler" [High,Triaged] - Assigned to Matt Riedemann (mriedem)
17:45:12 mriedem kashyap: fyi that i updated the devstack patch to test multiattach to set CONF.libvirt.images_type=raw and CONF.use_cow_images=False to see if that makes a difference
17:48:57 openstackgerrit Merged openstack/nova master: Revert "Modify _poll_shelved_instances periodic task call _shelve_offload_instance()" https://review.openstack.org/530284
18:19:19 mriedem wow, if i'm reading his results correctly, Kevin_Zheng got a 90% improvement in listing instances with details using an ip filter after we proxy to neutron first
18:19:25 mriedem sdague: ^
18:19:40 mriedem 2000 instances in nova and 2K ports in neutron
18:21:04 mriedem told him to send the details to the dev list
18:22:27 openstackgerrit Marcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64 https://review.openstack.org/530965
18:27:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for resizing failing when using CachingScheduler https://review.openstack.org/531211
18:27:18 mriedem edleafe: ^ functional regression test for resize + caching scheduler failure
18:29:05 edleafe mriedem: thx
18:29:53 cdent efried: sorry for delay, better ✔
18:30:22 efried Thanks cdent. jaypipes, you want to look before I send it? http://paste.openstack.org/raw/638137/
18:36:48 efried going, going, gone.
18:47:09 cdent shut your filthy mouth efried
18:47:17 cdent (re notifications)
18:47:28 efried Hehehehe, thought you'd like that.
19:02:38 openstackgerrit Merged openstack/nova master: Add support for getting volume details with a specified microversion https://review.openstack.org/529656
19:20:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle no allocations during migrate https://review.openstack.org/531220
19:36:08 openstackgerrit Merged openstack/nova master: DriverBlockDevice: make subclasses inherit _proxy_as_attr https://review.openstack.org/524167
19:36:14 openstackgerrit Merged openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037
19:40:36 mriedem mdbooth: i was looking at where _get_volume_config is used and it's used in quite a few places in the libvirt driver, like live migration, swap volume, and generally anywhere we build a guest xml, so also for resize/cold migrate,
19:40:52 mriedem and in a lot of those places, we don't have a volume to know if it's multiattach or not to set the conf.shareable flag,
19:41:19 mriedem so despite it not being great, stashing the multiattach value in the connection_info which gets saved in the bdm is about the best we can do right now
19:41:39 mriedem because all of these places in the driver that call _get_volume_config are using the connection_info off the bdm
19:53:41 sdague mriedem: nice
19:53:58 sdague mriedem: I'd totally believe it, we do extra dumb stuff in there
19:54:11 mriedem substring match is a bit slower, as expected, but still much better than doing it in nova
19:54:17 sdague mriedem: yep
19:54:28 mriedem 4000ms w/o the patch, 400ms with the patch and no substring, about 900ms with the patch and substring match
20:03:37 mriedem cdent: edleafe: it's your favorite game show, "IS IT 400 OR 409?!"
20:04:01 mriedem so, say i'm trying to attach a multiattach volume to an instance and the compute is too old to support multiattach,
20:04:13 mriedem is that a 400, or a 409? knowing that at some point the compute might be upgraded to support multiattach.
20:06:14 cdent the state of the compute is out of sync with the desired state, but you made a valid request (had it been in the right state), so I'd go with 409
20:06:28 mriedem artom: remember how we punted for tagged attached on a shelved offloaded instance because we couldn't tell if the host supported it?
20:06:39 mriedem cdent: ok that's what i was thinking too
20:06:54 cdent \o/
20:07:09 mriedem artom: but we don't make that same distinction in the API for tagged bdms with boot from volume
20:07:17 mriedem can you remember why?
20:08:52 mriedem artom: my guess was "you could reschedule and attempt to hit a host that does support device tagging" but we raise BuildAbortException so we don't even reschedule in that case
20:13:40 openstackgerrit melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614
20:13:40 openstackgerrit melanie witt proposed openstack/nova master: Optionalize instance_uuid in console_auth_token_get_valid() https://review.openstack.org/481700
20:13:41 openstackgerrit melanie witt proposed openstack/nova master: Add ConsoleAuthToken object https://review.openstack.org/320063
20:13:41 openstackgerrit melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381
20:13:42 openstackgerrit melanie witt proposed openstack/nova master: Use ConsoleAuthToken object to generate authorizations https://review.openstack.org/325414
20:13:42 openstackgerrit melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990
20:15:29 edleafe yeah, the request isn't malformed; the system can't support it
20:16:12 artom mriedem, yeah, I think your guess is correct
20:16:14 cdent support it _right now_
20:16:44 mriedem artom: ok, so that's not really any different from punting when trying to attach a volume to a shelved offloaded instance with tags
20:16:58 mriedem artom: i'm only asking b/c we're going to have a similar situation with multiattach volumes
20:17:31 mriedem trying to decide if we should support attaching a multiattach volume to a shelved offloaded instance - same with bfv, either way it might fail once we get to the compute and you're dead
20:18:08 mriedem at some point in the future, which no one will probably ever work on, we could do a scheduler filter for this stuff to make it pick a host which supports the thing you need
20:18:17 artom With the full 100% knowledge of every bit of context about multiattach, I'd vote to be consistent and refuse it for shelved offloaded
20:18:19 mgagne if running mitaka, is legacy v2 API still used or is it dead code?
20:18:39 artom Because we have no idea what host it'll end up on
20:18:43 artom When it's unshelved
20:18:51 mriedem artom: the same is true for boot from volume
20:19:09 artom mriedem, eh, how so?

Earlier   Later