Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-14
19:30:18 efried Or before the allocation, as part of testing for viable candidates?
19:30:20 dansmith efried: well, "the allocation" is the wrong thing to call the pre-create I think, because cyborg can't do a whole lot at that point, because it has no idea what host you're going to land on
19:32:16 efried dansmith: This is what I'm asking. By "allocation" I'm specifically talking about the placement allocation. Does the pre-create happen while we're still considering allocation candidates (which I guess would involve n-cond => n-cpu => os-acc => plugin and waiting for a response) and allow us to discard some of them, or does it happen after the allocation and if it fails we have to move on to our next of three tries within
19:33:05 efried Reading between the lines of your answers above I'm guessing it's the latter.
19:33:22 dansmith efried: when you said pre-create in cyborg, I assumed you meant before you do boot
19:33:48 dansmith in which case, it can't create an allocation in placement for you, because it doesn't know what RP to use (and more importantly, the compute node parent of the RP)
19:33:50 efried dansmith: I guess so, because boot includes plug, which has to come later.
19:34:56 Sundar How about this model? Accelerators are RCs, pre-published to Nova. Scheduling works with known RPs, traits and RCs, but attach handles are not part of it. Cyborg alone deals with attach handles. When a request comes to n-cpu, one or more attach handles get picked (nova virt either calls a plugin, or calls Cyborg agent, which int turn calls a Cyborg driver/plugin). The pre-create would be done by the plugin
19:35:47 dansmith um
19:35:51 dansmith I'm not sure what to do with that
19:36:07 dansmith Sundar: IMHO, cyborg or os-acc should not be mucking with the instance's allocation in placement
19:36:25 Sundar Yes, it does not
19:36:40 dansmith the discussion above is not about attach handles
19:36:53 Sundar Cyborg publishes RPs etc to placement. Nova/plcaement do all the allocation of candidates
19:36:58 openstackgerrit Brianna Poulos proposed openstack/python-novaclient master: Microversion 2.63 - Add trusted_image_certificates https://review.openstack.org/500396
19:37:53 Sundar My difficulty is in relating attach handles to accelerators, and making sure all possibilities and use cases are covered :)
19:38:11 dansmith Sundar: I'm confused.. the things you're saying are true and not really negotiable, but it sounds like you're suggesting something else
19:38:34 Sundar Sorry, let me try again
19:38:44 Sundar Consider this flow:
19:39:52 Sundar A request comes to n-api. The n-sch works works with placement to decide allocation candidates and choose a compute node. The request then comes to n-cpu in the compute node. No Cyborg so far (except for populating Placement at the start). is this good so far?
19:40:26 efried cyborg was involved in creating the artifact to begin with. I think we're calling this an "attachment".
19:40:46 efried Sundar: How about this, let me draw up the flow as I think I understand it and then we'll have a starting point for discussion.
19:41:08 efried dansmith: What's this pre-create thing called in the neutron flow?
19:41:17 Sundar efried: Cyborg publishes accelerators as RCs. You are saying attach handles (like PCI VFs or Power location codes) should also be published?
19:41:17 dansmith create port?
19:42:04 efried Sundar: I'm saying the user does a "openstack create accelerator-attachment <details>". Cyborg creates what you're calling an "attachment" and returns its UUID.
19:42:23 efried Sundar: Then the user does "openstack server create <details> --accelerator <that UUID>"
19:43:04 Sundar efired: why is that needed? For a edevice, the #attachments can be discovered automatically., unlike Neutron ports whose # is not fixed or discoverable
19:43:14 efried This gets to n-sch, which asks cyborg for the Acc* object corresponding to the attachment UUID. Cyborg pulls the original profile info out of its database and feeds it back.
19:43:24 dansmith efried: and nova uses that id to ask cyborg what resources it should ask for based on what accelerator the user asked for
19:43:26 Sundar E.g. for a SR-IOV device,
19:43:30 dansmith efried: and include that in the allocation
19:43:46 Sundar the #VFs is known at device discovery time
19:43:52 dansmith it gets that claim, and then can immediately tell cyborg "okay, I just committed $foo to $user on host $bar, cool?"
19:44:08 dansmith and if cyborg says yes, then cyborg reswizzles its inventory if needed, and compute gets started
19:44:09 efried Right, n-sch converts the Acc* object into placement query material to be included in the GET /a-c query.
19:44:26 dansmith when ready, it asks cyborg for "okay how are we going to attach this mofo?", which might be os-acc, and then we're off
19:46:04 efried Okay, I think dansmith and I are on the same page. So dansmith in your view, the plugin needs to have methods for plug(), unplug(), and this prepare_for_plug() thing?
19:46:24 dansmith probablyish
19:46:42 dansmith it could also have the "tell cyborg we chose a host for attachment $foo", which n-cond could use in the early part
19:47:13 dansmith which neutron calls a host binding
19:47:15 tlb any devs around?+
19:48:38 efried dansmith: check_host_for_attach(), then bind(), then plug() ?
19:50:08 dansmith efried: no, I think just bind(), plug()
19:50:13 efried okay.
19:50:19 dansmith efried: I mean, right?
19:50:28 dansmith we know what host we want to use because we just got granted an allocation for one
19:50:32 efried dansmith: Well, I didn't know this bind step was a thing at all.
19:50:37 dansmith we try to bind(), if that works, then we move on
19:50:49 Sundar_ IRC froze on me. Reconnecting
19:51:06 dansmith bind() tells cyborg that we're consuming a thing, it can adjust its inventory before it says success or failure
19:51:31 dansmith once bind returns true, we know that cyborg is ready to commit that thing to us and we'll next talk to cyborg during the plug() part on the compute
19:52:08 tlb Q: _get_computes_for_cells returning subset of available compute nodes across AZs (wrong subset, as it happens); suggestions in the dbeug log that it's cached somewhere (get_or_set_cached_cell_and_set_connections); any idea how/where? not obvious from the code, or at least, not to me.
19:52:14 efried dansmith: Okay. So the PluginBase should have a bind() method with a default `pass` impl which will probably not be overridden in many cases. The bind() step in cyborg will do stuff that diddles with cyborg's internal db and whatnot, and then call into the plugin's bind() method just in case.
19:52:18 tlb *debug
19:52:20 Sundar_ The difference between Neutron ports and an attach handle is this. A port is a virtual object that needs to be created by CLI. An attach handle (at least in some common cases like PCI VFs) are device properties, which can be discovered. We don't need 'openstack create' for that
19:52:58 efried Sundar_: Discovered...
19:53:01 dansmith efried: anything that has a potentially-varying inventory would need to implement it
19:53:15 dansmith Sundar_: no
19:53:18 dansmith that's not it
19:53:33 dansmith Sundar_: we're discussing an attachment handle as purely a handle to some state on the cyborg side
19:54:10 efried Sundar_: I'm going to tell you what frame rate and display heads and resolution I need from my GPU, and you're going to put that profile into your database and return a UUID to me.
19:54:18 dansmith right
19:54:24 dansmith which may mean you're just going to give me a device,
19:54:34 dansmith or it may mean you'll reconfigure some device to give me a slice of it which will support that
19:54:37 efried This is so that, later on, when nova says "hey, give me the details on this UUID" you can give it back the AccGPU object with those details filled in.
19:54:37 dansmith I don't care
19:54:46 dansmith just give me a uuid
19:54:51 dansmith right
19:55:02 efried It's so we don't have to carry (big random dict) across the nova boot command line and down through the conductor to the scheduler and compute etc.
19:55:21 dansmith which is how it used to work with cinder, and that's why it's been painful every single day since
19:55:44 efried I'm going to work up a seqdiag...
19:55:47 Sundar_ "it may mean you'll reconfigure some device" without knowing which compute node is involved, how do you pick a device to reconfigure?
19:56:02 efried Sundar_: That won't happen until the bind() and/or plug() steps.
19:56:06 dansmith Sundar_: you won't until we bind
19:56:16 dansmith at that point, you get to do that, decide if there's room, etc
19:56:21 efried That's the point - we want to save off configuration details before we know where we're going to land.
19:57:05 Sundar_ "bind() tells cyborg that we're consuming a thing" Is this invoked by n-cpu on os-acc/Cyborg?
19:57:28 dansmith it's likely done by conductor, talking to cyborg, through os-acc
19:58:28 dansmith efried: I have to wander off for a bit, but I think we're on the same page
19:58:52 efried dansmith: ack. I'll make sure you see this diagram :)
19:59:02 dansmith I literally can not friggin wait
19:59:09 dansmith (that's a lie, I can wait)
19:59:38 efried dansmith: If you need something to keep your horses held until then, I still need your comeback on the reshape spec.
19:59:46 Sundar_ efried: You offered to build a flow/sequence diagram. Thanks. Please LMK when it is ready. May be it will make things more concrete. Not sure if we have the same mental model, but let's discuss on an etherpad
20:00:00 efried Sundar_: Roger wilco.
20:04:29 melwitt I started the rocky blueprint status etherpad here, still have some more to go through https://etherpad.openstack.org/p/nova-rocky-blueprint-status
20:04:49 mriedem melwitt: thanks for doing that
20:04:52 mriedem don't you feel better now?!
20:05:15 melwitt sorta
20:13:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158
20:13:10 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add trusted certs to feature support matrix docs https://review.openstack.org/574890
20:13:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove max_size parameter from fake_libvirt_utils.fetch_*image methods https://review.openstack.org/574911
20:13:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix nits from trusted certs notification change https://review.openstack.org/575521
20:22:09 mriedem huh, for some reason the ironic job is running on our docs-only changes https://review.openstack.org/#/c/574890/
20:22:12 mriedem ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa SUCCESS in 45m 30s (non-voting)
20:23:04 mriedem http://git.openstack.org/cgit/openstack/ironic/tree/zuul.d/legacy-ironic-jobs.yaml#n4
20:23:24 mriedem ^doc/.*$ should match and skip
20:23:27 melwitt heh. that's odd
20:23:38 melwitt oh, hold the phone

Earlier   Later