Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-04
20:17:31 mriedem trying to decide if we should support attaching a multiattach volume to a shelved offloaded instance - same with bfv, either way it might fail once we get to the compute and you're dead
20:18:08 mriedem at some point in the future, which no one will probably ever work on, we could do a scheduler filter for this stuff to make it pick a host which supports the thing you need
20:18:17 artom With the full 100% knowledge of every bit of context about multiattach, I'd vote to be consistent and refuse it for shelved offloaded
20:18:19 mgagne if running mitaka, is legacy v2 API still used or is it dead code?
20:18:39 artom Because we have no idea what host it'll end up on
20:18:43 artom When it's unshelved
20:18:51 mriedem artom: the same is true for boot from volume
20:19:09 artom mriedem, eh, how so?
20:19:13 mriedem but we allow tagged bdms with bfv
20:19:33 mriedem when you create a server, or unshelve a server, they both go through the scheduler to pick a host,
20:19:46 mriedem you have an equal chance in either scenario of picking a host that doesn't support the capability
20:20:05 mriedem so the fact we allow tagged bdms with bfv but not shelved offloaded instances is inconsistent
20:20:12 artom mriedem, how is bfv different from normal volume tagging? IIRC the api just checked service level, to see if the cloud was fully upgraded to support it
20:20:17 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Attach and detach encryptors during swap_volume https://review.openstack.org/531233
20:21:31 mriedem artom: ok yeah that's the one difference then, is that we check in the api when creating a server if the computes are new enough to handle tagged bdms
20:21:43 mriedem we could easily have done the same for shelved offloaded attach
20:22:03 artom mriedem, so there's this 2 x 2 matrix of stuff
20:22:10 artom we have tagged boot / tagged attach
20:22:11 mriedem that's basically what i was saying we'd do in those two cases for multiattach too - check the min compute service version and fail if computes aren't upgraded yet
20:22:20 artom And compute manager supports it / virt driver supports it
20:22:57 artom For tagged boot, we used the service level check to see if compute manager supports it
20:23:12 artom For tagged boot/virt driver... I forget how we handled it
20:23:21 mriedem we just fail in the compute
20:23:24 mriedem and raise BuildAbortException
20:23:50 artom Ah, right, hopefully rescheduling to a virt driver that supports it
20:23:57 mriedem no, we don't reschedule
20:24:09 mriedem BuildAbortException means abort, don't reschedule
20:24:14 artom Ah, ok
20:24:28 artom Yeah, that's not awesome, but no other way of doing it I guess
20:24:31 mriedem BuildRescheduledException means kick edleafe in the head a few times
20:24:43 artom And then for tagged attach...
20:25:05 artom I think they were all RPC calls down to the compute
20:25:13 artom So virt driver support was straightforward
20:25:25 artom Except for shelved offloaded, which was a cast
20:25:28 mriedem yeah that one is easy
20:25:28 artom (Obviously)
20:25:35 mriedem unless the server is shelved offloaded
20:25:42 artom So that one we decided to just fail in the API
20:25:52 mriedem in the case of shelved offloaded, we don't do anything with the compute b/c there is no compute
20:25:58 artom Instead of having the unshelve fail like, 3 years later
20:26:13 mriedem yeah, i'm saying, i think at a minimum we could do the same for shelved offloaded as we do for tagged boot which is check the service version
20:26:39 mriedem we don't reschedule on an unshelve failure, but we don't reschedule on a tagged boot failure either
20:26:43 mriedem so it's basically the same
20:26:57 artom One is definitely more immediate than the other
20:27:22 edleafe mriedem: so it seems that the bug I found was the result of a race
20:27:22 artom Though I guess we if we relay the exception properly, it's not a massive deal
20:27:32 edleafe The call to retry on https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4087 runs the retry before the 'finally:' clause reight below it gets hit.
20:27:44 edleafe If I add the call the self._revert_allocation() before L4087, the test I wrote passes.
20:27:53 artom Like, if nova show unshelved-instance has 'you unshelved on a virt that doesn't support tagging' somewhere in there (but more cloudy) it can work
20:28:14 mriedem edleafe: ah just like the thing we hit in the other patch
20:28:23 mriedem need to cleanup the allocation before casting to conductor
20:28:56 mriedem artom: we'd record a fault
20:28:58 mriedem that's about it
20:29:00 edleafe mriedem: do you have a fix for the caching scheduler? Or should I do that?
20:29:16 artom mriedem, is that obvious to the user?
20:29:18 mriedem edleafe: https://review.openstack.org/#/c/531220/
20:29:31 mdbooth mriedem: My patch series removes it from swap_volume, btw, but not surprised by the others.
20:29:32 mriedem artom: not really, the fault's traceback is only available to admins
20:29:55 artom mriedem, so give this lack of feedback to the user, it's probably better to fail fast in the API, no?
20:29:55 mriedem mdbooth: swap_volume is the one case where i don't think this matters because we don't rely on the conf.shareable attribute
20:30:23 mriedem artom: maybe
20:30:29 mriedem i can see the argument for being consistent
20:30:34 mdbooth mriedem: Ok, so how can we flag this as a second connection_info wart?
20:30:47 mriedem mdbooth: 2nd as in device_path is #1?
20:30:52 artom mriedem, what's the issue we see with multiattach though?
20:31:03 mdbooth I mean there's this and also device_path
20:31:13 edleafe mriedem: ah, I just saw the test patch
20:31:19 mdbooth mriedem: Right, yeah. What you said :)
20:31:21 artom I haven't followed closely, so it may or may not be worth it to dump all the context on me for whatever my opinion is worth ;)
20:31:35 mriedem artom: nothing - i'm just weighing options since we don't have the api plumbed in for all of the multiattach stuff
20:31:41 mriedem so i'm thinking through how to handle this,
20:31:50 mriedem because the backend capabilty checking is the same as with tagged attach
20:32:19 mriedem and i need to get out of my head sometimes
20:32:37 mriedem mdbooth: i'm not sure how to flag this
20:32:41 artom Are we talking about attaching to a shelved offloaded instance? And we don't know whether the eventual compute would support it?
20:32:51 mriedem artom: correct
20:33:11 mdbooth mriedem: I don't think there's a good central focal point for connection_info cruft. Which is another problem, tbh.
20:33:39 jackie-truong sdague: The nova-queens-blueprint-status etherpad mentioned that you needed Johns Hopkins to sync up with you on the certificate validation feature
20:34:20 artom mriedem, so exactly the same problem? 1. check that compute manager is new enough 2. check that virt driver supports it
20:34:28 artom Or is multi-attach virt-agnostic?
20:35:18 jackie-truong sdague: We created an etherpad (https://etherpad.openstack.org/p/queens-nova-certificate-validation) to walk through usage and testing. Let me know if you need more information or have any questions.
20:35:45 mriedem artom: it's the same problem
20:36:04 mriedem in queens, assuming we ship this code, only the libvirt driver will support multiattach
20:37:03 artom mriedem, one thing we talked about was scheduling with compute driver capabilities taken into account, probably through placement/resource providers
20:37:20 artom I don't think that work is ready yet, though
20:37:56 artom Then again, by the time multiattach lands, there might be talks of replacing placement with a new quantum-powered scheduler
20:37:58 artom ;)
20:39:10 mriedem sure, the ambiguity goes away if we had the CapabilitiesFilter aware of this and handling it
20:39:22 mriedem "request says it wants multiattach, find a host that supports multiattach"
20:39:23 mriedem done
20:40:02 mriedem anywho, i'll work on what i know we need to support for now, and get testing going, and then bikeshed on the rest
20:42:58 openstackgerrit Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753
20:42:59 openstackgerrit Jay Pipes proposed openstack/nova master: WIP Support aggregate affinity filters https://review.openstack.org/529201
20:42:59 openstackgerrit Jay Pipes proposed openstack/nova master: Remove server group sched filter support caching https://review.openstack.org/529200
20:43:00 openstackgerrit Jay Pipes proposed openstack/nova master: get instance group's aggregate associations https://review.openstack.org/531243
20:47:27 mgagne answering myself, legacy v2 code is still used in mitaka: https://docs.openstack.org/nova/latest/reference/stable-api.html
20:52:11 openstackgerrit Matt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments https://review.openstack.org/267587
20:52:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Use volume shared_targets to lock during attach/detach https://review.openstack.org/529695
20:52:12 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Pass multiattach flag to reserve_block_device_name https://review.openstack.org/531244
20:52:12 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: [api] Allow multi-attach in compute api https://review.openstack.org/271047

Earlier   Later