Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-27
09:45:13 efried gibi: That would be true, BUT I'm not certain this workflow involves assigning the accelerator at spawn time. I think we were only talking about post-deploy attach.
09:46:34 alex_xu efried: if cyborg create allocation, that will be a lot of race?
09:46:55 efried alex_xu: Not if cyborg is the Source Of Truth for managing the accelerator resource.
09:47:18 gibi alex_xu, efried: I if the scenario is post-deploy attach then I can accept that cyborg creates the allocation as in this case no need for asking for allocation candidates
09:48:03 efried gibi: Maybe. That would be up to cyborg and how they want to manage the RPs.
09:48:19 efried gibi: But Nova wouldn't, that's true.
09:49:04 gibi efried: do you mean a compute host might have more than one accelearator and cyborg want to select from that pool during post-deploy attach?
09:49:45 efried gibi: For sure.
09:49:46 gibi efried: yes, if there is cyborg calls allocation candidate API then cyborg can create the allocation too
09:50:24 efried gibi: Especially if we're talking about "regions" or whatever the VF equivalent is for FPGA.
09:51:16 efried gibi: And maybe cyborg wants to precreate those. Or maybe not.
09:51:42 efried gibi: But surely there could be more than one FPGA on the system in any case.
09:51:45 alex_xu gibi: sorry, what is post-deploy attach?
09:51:57 openstackgerrit Takashi NATSUME proposed openstack/nova master: Transform live_migration.post.dest notifications https://review.openstack.org/469784
09:51:58 efried gibi: And IIUC, cyborg wants to manage not just FPGAs but other accelerators too.
09:52:11 openstackgerrit Takashi NATSUME proposed openstack/nova master: Transform live_migration.post.dest notifications https://review.openstack.org/469784
09:52:31 efried alex_xu: Meaning nova does spawn with CPU/Mem/disk, and then after that's finished and the instance is already created, cyborg comes along and attaches the FPGA
09:53:23 gibi alex_xu: I think about it like adding volumes to an existing VMs
09:53:43 efried ^
09:53:52 efried or attach_interface()
09:54:03 alex_xu efried: gibi, got it
09:54:05 efried 'cept not done by nova :) So gibi's example is much better :)
09:54:32 alex_xu so the cyborg calling the allocation candidates API to get a accelerator from the pool of the host?
09:54:39 gibi efried: thanks, I think I got it. post-deploy sounds like an easy shortcut for not getting tangled into the nova boot process
09:54:46 alex_xu since the instance already boot up on specific host
09:54:48 efried gibi: ++
09:55:26 efried alex_xu: Yes. TBH, I'm not completely sure how one would call GET /allocation_candidates in such a way that it restricts to a certain host.
09:55:57 alex_xu efried: the we have parameter like in_tree?
09:55:59 gibi efried: it might be part of the 'anchor a request' discussion
09:56:04 efried alex_xu: Just so.
09:56:36 openstackgerrit Stephen Finucane proposed openstack/nova master: Stop using mox in virt/xenapi/image/test_vdi_through_dev.py https://review.openstack.org/342682
09:56:37 efried alex_xu: It depends to some extent on how the accelerators are modeled. If they're included in the compute node's provider tree, then yes, ?in_tree=<compute_uuid> on GET /allocation_candidates would be very appropriate.
09:57:18 alex_xu efried: yea
09:57:33 efried alex_xu: And/or if the accelerator is made part of an aggregate (not sure if that makes sense ever, but it extends to the more general model for e.g. bandwidth providers) then member_of
09:57:47 efried The latter is a Rocky bp. The former is not (yet).
09:58:42 Kevin_Zheng Any idea how Nova can consume resource reported by cyborg?
09:59:20 alex_xu efried: I didn't get the accelerator is made part of an aggregate. The aggregate in Nova?
09:59:51 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Move resource provider objects into placement hierarchy https://review.openstack.org/540049
09:59:52 openstackgerrit Chris Dent proposed openstack/nova master: Isolate placement database config https://review.openstack.org/541435
09:59:52 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
10:00:05 alex_xu Kevin_Zheng: depends on the direction we go I guess, with bauzas way, it can be done by cyborg API to return some resource class and traits. With the post-deploy attach, that is cyborg own thing
10:00:10 bauzas folks, I'm in the Scientifig WG room so I can't follow the convo
10:00:21 efried alex_xu: It probably doesn't make sense for accelerators to be modeled as true sharing providers.
10:00:23 alex_xu bauzas: no worries
10:00:43 efried alex_xu: Though be aware that sharing providers aren't the only possible use case for aggregates.
10:01:20 efried alex_xu: In the case of accelerators, which are pieces of hardware physically attached to the host, it most likely makes sense for them to be somewhere in the compute node's provider tree.
10:02:25 efried alex_xu: But for something like a bandwidth provider, where the component doing the work is maybe an external switch that manages traffic from multiple hosts, a sharing provider might be totally appropriate. (I need to reread the network bandwidth provider spec.)
10:02:26 alex_xu efried: yea, I can't image how accelerators shared between hosts
10:02:35 gibi Kevin_Zheng: as a wild alternative there is flavor extra_spec https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/custom-resource-classes-in-flavors.html
10:03:08 alex_xu efried: ah, that sounds make sense for bindwidth
10:03:10 gibi efried: correct, a TOR switch would be a shared provider
10:03:11 efried gibi: Here's a not-so-secret secret - that's how PowerVM is planning to do accelerator attach.
10:03:24 efried at least tentatively
10:03:35 bauzas efried: how the FPGA accelerators are modeled looks to me a Cyborg-only discusion
10:03:51 efried gibi: Though we may or may not need custom RCs.
10:03:54 gibi efried: We had our internal plans to do bandwidth with flavor extra_spec but it got voted down internally
10:04:06 bauzas I mean, I like the idea that the folks model their resources like they want and just allocate the resources like they want too
10:04:19 efried bauzas: ++
10:04:23 Kevin_Zheng gibi, alex_xu so cyborg is in charge of reporting accelerator resources, but how should it be connected to Nova Hosts?
10:04:43 efried Kevin_Zheng: That's a great question.
10:04:50 bauzas the only Nova piece would be the fact to tie the instance to the compute that has the accelerated resource and do the binding
10:05:03 gibi bauzas: the allocation part getting tricky if we want to avoid races during boot. With post-attach it seems a lot easyer
10:05:11 melwitt fyi we have some time scheduled to chat with the cyborg team at 2pm this afternoon at the cyborg room
10:05:11 efried bauzas, Kevin_Zheng: Like attach_interface, but for device?
10:05:19 openstackgerrit Stephen Finucane proposed openstack/nova master: Stop using mox in api/openstack/fakes.py https://review.openstack.org/325759
10:05:38 gibi melwitt: yeah, I'm warming up for that now :)
10:05:44 bauzas melwitt: gtk, I wasn't knowing that
10:05:56 bauzas melwitt: I just spent 2 hours of my time already in that room :p
10:07:07 gibi Kevin_Zheng: nova creates the compute RP and names it after the host name of the compute so if cyborg knows the host name where the accelerator is then cyborg can look up the compute RP created by nova in placement by the host name
10:07:33 alex_xu gibi: ++
10:07:36 melwitt oh, okay. on the dev ML last week or the week before we had a thread where we agreed on some time to discuss nova interaction things with the cyborg team
10:08:18 melwitt apologies if I didn't communicate that well enough, I put it on our ptg etherpad agenda times last week when I added times
10:08:29 Kevin_Zheng gibi hmm
10:08:36 bauzas melwitt: my bad, I probably messed up reading that ;)
10:08:53 bauzas melwitt: 2pm in their room ?
10:09:17 melwitt yes
10:10:05 gibi Kevin_Zheng: at least the host name will be the key information when neturon will create the networking RPs under the compute RP
10:10:20 gibi Kevin_Zheng: so I think the same can be done in cyborg
10:10:42 mriedem nova passes neutron the host name in the port binding so neutron can lookup the compute node RP yeah
10:10:43 Kevin_Zheng gibi yeah, I can see the picture now
10:10:46 mriedem for ironic, not so much...
10:12:13 melwitt FYI on thursday we're gonna start nova/cinder at 9:15 instead of 9:00
10:12:33 alex_xu gibi: I guess in the future, the accelerator device also has the NUMA affinity thing, so the cyborg also need to look up the NUMA rp
10:12:53 alex_xu at that time, NUMA rp should have some naming rule also, i guess
10:14:02 mriedem need another core on this https://review.openstack.org/#/c/526772/ - leaking passwords isn't good
10:15:34 gibi mriedem: the network RP during the init of the neturon agent running on the given compute host, so that agent knows the host name without any nova involvement
10:16:01 gibi mriedem: the network RP is created during...
10:16:01 mriedem gibi: true
10:16:17 openstackgerrit Chris Dent proposed openstack/nova master: [placement] use simple FaultWrapper https://review.openstack.org/533752
10:16:18 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Refactor WSGI apps and utils to limit imports https://review.openstack.org/533797
10:16:47 mriedem lyarwood: tonyb: we have some stable/ocata changes that need another +2 https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/ocata+label:Code-Review=2
10:17:42 gibi mriedem: but yeah, if we talk about letting neutron create the bandwidth allocation in placement then the host name in the binding helps. However I'm more favor of duing the allocation from nova according to the selected allocation candidate
10:18:34 gibi alex_xu: yeah, NUMA is a good question for me. We have to make NUMA RP named after some host level id that can be known by other agents
10:21:48 gibi 10:58 < frickler> infra-root: seeing bumpy response times from gerrit, increasing GC times, too. maybe
10:21:51 gibi restart during PTG lunch time?
10:22:07 gibi alex_xu: so gerrit is slow for everybody
10:22:46 jhesketh gibi: it's being restarted now
10:24:03 Kevin_Zheng So I guess after calling cyborg/neutron, nova will have filter hosts by names to get allocation candidates?
10:24:25 Kevin_Zheng if we have specific network/accelerator requests?
10:24:35 tonyb mriedem: okay I'll look over them

Earlier   Later