| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2020-06-02 | |||
| 06:08:46 | Yumeng | hello | |
| 06:08:57 | brinzhang | Sundar means add two nova APIs to support bind and unbinding ARQs? | |
| 06:09:09 | songwenping_ | Hello all | |
| 06:09:11 | chenke | Hi I move here. | |
| 06:09:17 | brinzhang | hello all | |
| 06:10:05 | Sundar | brinzhang -- That is not clear to me. If we can do HTTP PATCH on the server object, we can apply or remove devic eprofiles. | |
| 06:10:14 | Sundar | SO I think it is one API only | |
| 06:10:56 | brinzhang | Yeah, one or two all ok for me | |
| 06:11:22 | Sundar | Second problem is with libvirt/qemu -- I don;t think we can add PCI interfaces to a running VM. We have to stop it first, regenerate the doain XML and then restart the server | |
| 06:11:38 | Sundar | *domain XML | |
| 06:12:50 | Sundar | What is the openstack CLI command you are thinking of? | |
| 06:13:24 | Sundar | One option is to avoid all this, and just do a resize with a different flavor, which has a different device profile | |
| 06:13:31 | Sundar | Then no need to change Nova API at all | |
| 06:13:39 | songwenping_ | Sundar, I have test the case of add and remove PCI interfaces to a running VM like this https://review.opendev.org/#/c/729945/5/nova/virt/libvirt/driver.py | |
| 06:14:51 | Sundar | songwenping_ : How does this work? How will libvirt change the domain XML on the fly? | |
| 06:14:52 | brinzhang | Sundar, I mean I want add two nova api to support binding and unbinding arqs from nova. that we can use "nova bind-accelerators and/or unbind-accelerators" | |
| 06:15:44 | brinzhang | this need a spec to nova team | |
| 06:16:12 | Sundar | brinzhang: The Nova CLI is old. In the new openstack CLI, we say: $ openstack server create --flavor myflavor ..., where myflavor has the device profile. What is the openstack CLI you plan to use for hot attach/detach? | |
| 06:17:02 | songwenping_ | `guest.detach_device(hostdevPCI, persistent=True, live=True)' this function with "live=True" param can work for running VM. | |
| 06:17:33 | songwenping_ | we can dumpxml the VM and see the hostdev PCI attach. | |
| 06:17:37 | brinzhang | Sundar, they are same, we should do completed the cli in novaclinet and then to openstackclient to support, otherwise we cannt completed this feature in Nova | |
| 06:18:26 | brinzhang | Sundar: binding/unbinding will be an independent operation in nova | |
| 06:19:09 | brinzhang | Now we dont consider to support resize operation, that will be out of my control | |
| 06:19:19 | brinzhang | I think | |
| 06:19:53 | Sundar | brinzhang: will you pass a device profile to this CLI? if so, will one VM have 2 or more associated device profiles? | |
| 06:21:58 | brinzhang | Sundar: maybe you are right, but I am sure I can answer your question, I dont know does it need pass device_profile to the CLI | |
| 06:22:14 | Sundar | songwenping_: Great. I found this: https://libvirt.org/pci-hotplug.html Thanks for the pointer. | |
| 06:23:15 | brinzhang | If we want to bind an ARQ to an instance that its flavor has not accel:device_profile, maybe we should pass it. | |
| 06:24:34 | brinzhang | with unbind, I think we should judge the instance does have the accel:device_profile, access unbind or reject. | |
| 06:25:07 | songwenping_ | Yeah, right Sundar. Libvirt has supported hotplug for PCI devices. | |
| 06:25:41 | Sundar | IMHO, it is better to support resize: it covers hot-add and is more generally useful also. It allows us to keep the model that one VM has one device profile. | |
| 06:26:24 | Sundar | It is a standrd Nova operation: so less resistance, and may be we can even get help from Nova developers | |
| 06:27:01 | shaohe_feng | Libvirt has supported hotplug for PCI devices for a long time-_- | |
| 06:27:10 | Sundar | brinzhang: on a resize, does Nova scheduler get involved? If so, can it pick a different host? | |
| 06:27:37 | brinzhang | Sundar, resize may need to change host,that not easy to control, I know it should support, but it need time | |
| 06:27:43 | xinranwang | I think it will do re-schedule | |
| 06:27:45 | Sundar | Libvirt does not support it for the default config: looks like you have to specify a different cpu model q35. | |
| 06:29:20 | Sundar | brinzhang: we already have the scheduler flow for device profiles and ARQs. Why do you think it is difficult? | |
| 06:30:59 | Yumeng | brinzhang: I think unshelve will also do re-schedule, right? | |
| 06:31:48 | brinzhang | unshelve jsut need to re-create ARQs, it's easy to get | |
| 06:32:38 | brinzhang | I am not search into resize, so I can not answer clearly. | |
| 06:32:48 | Sundar | We can bring this up in the Nova discussion. | |
| 06:33:35 | Sundar | Anyway, we need to tlak to them about other VM ops for accelerators. | |
| 06:33:39 | Sundar | *talk | |
| 06:33:42 | shaohe_feng | Sundar: a question about schedule. You have bring up a discussion about this: | |
| 06:33:52 | brinzhang | This release we have rebuild/evcaute, suspend/resume, shelve/unshelve, I think it's enough to do in V | |
| 06:34:03 | shaohe_feng | add a new filter for dynamic program | |
| 06:34:23 | shaohe_feng | let cyborg do the filter, right? | |
| 06:34:41 | Sundar | shaohe_feng: not sue what you are referring to | |
| 06:34:50 | brinzhang | In https://review.opendev.org/#/c/729945/5/api-guide/source/accelerator-support.rst we record the operations supported in cyborg | |
| 06:36:07 | Yumeng | brinzhang: I think we can also start to support resize, chenke can investigate in. anyway, we can start discuss with nova during the session. | |
| 06:36:28 | brinzhang | agree | |
| 06:37:53 | brinzhang | after chenke talked with nova team, I think we can get more info about resize. | |
| 06:38:19 | shaohe_feng | such as, currently the function of the FPGA is ovs with 4 VF, but no VM uses it. and now on more FPGA available , we can select this one ovs PFGA as IPsec with 2 NICs, and change its function. | |
| 06:38:52 | shaohe_feng | ^ Sundar so this need a scheduler improvement, right? | |
| 06:39:05 | shaohe_feng | and you bring up to add a new filter. | |
| 06:39:13 | shaohe_feng | I remember. | |
| 06:40:49 | Sundar | shaohe_feng: That doesn;t involve the Nova scheduler. We can define a device profile with a trait that selects that type of FPGA, and an accel:bitstream that refers to ipsec. Then Placement may return some FPGA with ovs image if it is free. But Cyborg will notice it does not have ipsec and do the programming. | |
| 06:41:36 | Sundar | What we don't have is this: | |
| 06:42:05 | shaohe_feng | yes, that's the current flow in cyborg at present | |
| 06:42:13 | Sundar | Say, we have two free FPGAs, one with ovs and with ipsec. In this case, we want the scheduler to pick the one with ipsec preferentially. That is not there today. | |
| 06:43:05 | Sundar | The scheduler has a weigher but it selects among hosts, not among allocation candidates from Placement | |
| 06:43:13 | shaohe_feng | you can not fill ovs or ipsec in the trait | |
| 06:43:44 | Yumeng | brinzhang: I was curious, given "unshelve jsut need to re-create ARQs, it's easy to get", does that mean shelve operation does not actually release the accelerator in hypervisor, the accelerator is still a claimed allocation, that's why there is no re-scheduler in unshelve. | |
| 06:44:00 | shaohe_feng | but the weigher also not support for FPGA, right? | |
| 06:44:00 | Yumeng | please correct me if I was wrong | |
| 06:44:08 | Sundar | That is right. At least, if you do, it makes it too specific -- if you ask for ipsec and no FPGA with ipsec is free, the request will fail. No reprogramming. | |
| 06:44:17 | Sundar | What we really want is preferred traits: | |
| 06:44:36 | Sundar | We should be able to say I prefer the resource provider have this trait but, if not, give me something | |
| 06:44:42 | Sundar | Alex Xu knows about this | |
| 06:45:00 | shaohe_feng | everyone also know about it | |
| 06:45:07 | shaohe_feng | -_- | |
| 06:45:19 | shaohe_feng | but we want to improve it | |
| 06:45:46 | Sundar | shaohe_feng: There are many weighers, and we cna write custom weighers too. None of them cover FPGA, because they deal with hosts, not device RPs. | |
| 06:45:57 | shaohe_feng | and everyone know placement is so weak :') | |
| 06:46:14 | shaohe_feng | we can consider how to improve it. | |
| 06:46:37 | Sundar | What I meant is, Alex Xu knows Cyborg wants preferred traits. I think he was looking into implementing it at some point, but he got busy with other things | |
| 06:47:00 | shaohe_feng | yes, we can consider write a custom weigher for FPGA | |
| 06:47:25 | Sundar | Nope, that won;t help, as I said | |
| 06:47:39 | shaohe_feng | he did not give us a good suggestion | |
| 06:47:54 | shaohe_feng | what he know, others also know \ | |
| 06:48:01 | shaohe_feng | we want a good solution | |
| 06:48:13 | shaohe_feng | not the common knowledge | |
| 06:48:33 | brinzhang | Yumeng: when shelve instance we delete the instance's ARQs from nova and cyborg, when unshelve we should create it, it need to re-scheduler in unshelve | |
| 06:49:16 | shaohe_feng | yes, we can improve placement, every one can improve it. but we can let upstream accept it. | |
| 06:49:31 | brinzhang | Yumeng: the poc code is here, you can review https://review.opendev.org/#/c/729563/2 | |
| 06:50:42 | shaohe_feng | such as SDL can used for placement, but it is a big change for placement, right? vendor can improve it in this way. but it is difficult in upstream, right? | |
| 06:51:12 | Sundar | If we can get preferred traits, that would help. Shall we add that to Nova discussion? | |
| 06:51:19 | brinzhang | we will introduce the service version to support these operations, controled in https://review.opendev.org/#/c/715326/13/nova/compute/api.py@288 | |
| 06:52:23 | shaohe_feng | preferred is just a case of SDL. SDL can cover it. | |
| 06:54:18 | shaohe_feng | and preferred can not resolve the complex scenarios :') | |
| 06:54:24 | Sundar | Yumeng: ^ | |
| 06:54:38 | Sundar | brinzhang: ^ | |
| 06:55:15 | Yumeng | Sundar,yes sure. I will add operations to Nova etherpad | |
| 06:55:32 | shaohe_feng | Yumeng no, not this release. | |
| 06:55:32 | Sundar | No I mean discuss preferred traits | |
| 06:55:46 | shaohe_feng | the next release. | |
| 06:56:00 | shaohe_feng | I just want to know some history from sunder | |
| 06:56:04 | Yumeng | I don't think we have enough time to discuss preferred traits. | |
| 06:56:09 | shaohe_feng | but not preferred history | |