| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-08-08 | |||
| 10:17:51 | openstackgerrit | Merged openstack/cyborg master: Docs: Autogenerate config documentation https://review.openstack.org/588480 | |
| 14:01:54 | zhipeng | #startmeeting openstack-cyborg | |
| 14:01:55 | openstack | Meeting started Wed Aug 8 14:01:54 2018 UTC and is due to finish in 60 minutes. The chair is zhipeng. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 14:01:56 | openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 14:01:58 | openstack | The meeting name has been set to 'openstack_cyborg' | |
| 14:02:25 | zhipeng | #topic Roll Call | |
| 14:02:51 | zhipeng | cmd is definitely not working today | |
| 14:04:21 | zhipeng | #info Howard | |
| 14:11:58 | shaohe_feng_ | #info shaohe_feng | |
| 14:12:05 | xinran | #info xinran | |
| 14:13:03 | zhipeng | shaohe_feng_ xinran let's catch up in wechat | |
| 14:14:08 | shaohe_feng_ | OK | |
| 14:14:53 | xinran | ok | |
| 14:14:57 | zhipeng | #endmeeting | |
| 14:14:59 | openstack | Meeting ended Wed Aug 8 14:14:57 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 14:15:00 | openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_cyborg/2018/openstack_cyborg.2018-08-08-14.01.html | |
| 14:15:01 | openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_cyborg/2018/openstack_cyborg.2018-08-08-14.01.txt | |
| 14:15:02 | openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_cyborg/2018/openstack_cyborg.2018-08-08-14.01.log.html | |
| 14:38:49 | Coco_gao | Hi | |
| 15:35:35 | Li_Liu | finally, I am in... | |
| #openstack-cyborg - 2018-08-09 | |||
| 02:50:00 | openstackgerrit | wangxu proposed openstack/cyborg master: Fix deployable get all sort issue and unit test failure. https://review.openstack.org/590075 | |
| 08:39:55 | openstackgerrit | ShaoHe Feng proposed openstack/cyborg master: support sub provider https://review.openstack.org/585146 | |
| 11:04:40 | openstackgerrit | ShaoHe Feng proposed openstack/cyborg master: support sub provider https://review.openstack.org/585146 | |
| 11:07:58 | openstackgerrit | ShaoHe Feng proposed openstack/cyborg master: support sub provider https://review.openstack.org/585146 | |
| 12:09:10 | openstackgerrit | ShaoHe Feng proposed openstack/cyborg master: support sub provider https://review.openstack.org/585146 | |
| 20:40:52 | Sundar_ | Hi Eric and Sean | |
| 20:41:18 | efried | Sundar_: hold on, Sean's not here yet... | |
| 20:45:14 | efried | Sundar_, sean-k-mooney: Absolutely agree libvirt-XML-specific code should live in the os-acc driver and/or plugin, NOT in nova. | |
| 20:46:02 | Sundar_ | Sean, I am just saying that the os-acc provides device-specific info to Nova virt drivers depending on the hypervisor. For e.g., for libvirt + PCI, it may provide a XML snippet for a PCI device, but does not modify the VM's domain XML. That is left to the libvirt driver. | |
| 20:46:22 | Sundar_ | efried: Agreed | |
| 20:46:27 | sean-k-mooney | efried: only if the os-acc driver is part of nova and not cyborg | |
| 20:46:32 | Sundar_ | efried: Agreed | |
| 20:47:06 | efried | "part of nova and not cyborg" - the driver stands alone. It talks to both nova and cyborg, but it's not really "part of" either. | |
| 20:47:13 | sean-k-mooney | what is the argument for allowing arbitary xml to be instered form an external lib that is loaded dynamically at runtime | |
| 20:48:13 | sean-k-mooney | efried: this exact usecase was forbin for both os-brick and os-vif | |
| 20:49:22 | Sundar_ | sean-k-mooney: It is not necessarily arbitrary: the return value, including the schema, can be specified and checked by the virt driver. But it allows for more extensibility -- environment/device-specific variations can be handled better | |
| 20:50:29 | sean-k-mooney | Sundar_: how do you know how to asign virtual pci address for acclorator withing the guest | |
| 20:50:39 | sean-k-mooney | jsut as an example | |
| 20:51:11 | Sundar_ | sean-k-mooney: The analogy with os-vif cannot be taken too far. The os-vif handles a small number of plugins (Linux bridge, OVS, etc.) and it is intertwined with virt driver logic in practice. With os-acc, we will have far more variation in devices. Also, we do not want to repeat the coupling and legacy pracices of os-vif | |
| 20:52:08 | sean-k-mooney | Sundar_: you are going to end up coupleing the drivers to hyperviros in a way that is highly fragile | |
| 20:52:18 | Sundar_ | sean-k-mooney: For the virtual PCI, the os-acc plugin/driver will get it in a device-specific way and, depending on the hypervisor (e.g. libvirt+qemu), can construct the needed snippets | |
| 20:53:12 | sean-k-mooney | Sundar_: which would mean i would have to pass the iamge and flavor to you in addtion to the list of all interfaces | |
| 20:53:12 | Sundar_ | sean-k-mooney: "Coupling the drivers to hypervisors' -- can you make that more concrete with an example or specific scenario? | |
| 20:53:45 | sean-k-mooney | well for a start the fact that os-acc needs to know you are using libvirt at all is a failing in my view | |
| 20:54:56 | sean-k-mooney | whey canre we not define a datamodle using oslo versioned object that will be returned by os-acc and have each of the virt driver interperate them internally | |
| 20:56:41 | Sundar_ | For the specific case of libvirt and PCI, I don't see the issue with this. But I am not sure if that works for all devices and hypervisors. Power seems quite different, for example. may be efried can chime in. | |
| 20:58:01 | efried | Yeah, sean-k-mooney, platform-gnostic code is like half of the raison d'être for os-acc. | |
| 20:58:18 | Sundar_ | As long as the XML snippet is well-defined and constrained, what is the issue? | |
| 20:58:34 | efried | sean-k-mooney: Because e.g. the 'plug' operation for libvirt entails modifying xml, whereas on power it involves issuing a REST command to a proprietary API. | |
| 20:59:02 | sean-k-mooney | efried: plug does not modify xml | |
| 20:59:24 | sean-k-mooney | os-vif cant by design. | |
| 20:59:34 | efried | what does, then? | |
| 20:59:40 | efried | the nova virt driver? | |
| 20:59:43 | sean-k-mooney | the libvirt driver | |
| 20:59:55 | efried | okay, so be it. | |
| 21:00:06 | efried | what about discovery? | |
| 21:00:12 | efried | who's responsible for that? | |
| 21:00:13 | sean-k-mooney | os-vif is invoked with a ovo construted by the driver to wire up the backend | |
| 21:00:25 | efried | Again, libvirt => lspci (I think); Power => REST call. | |
| 21:00:26 | sean-k-mooney | efried: current the libvirt driver alos | |
| 21:00:50 | sean-k-mooney | i had proposed haveing a call in os-vif for discovery | |
| 21:01:24 | sean-k-mooney | it was pushed back on however so we can discover what plugins are install but thats about it | |
| 21:02:08 | sean-k-mooney | efried: in the os-vif the spcific example of modiging the hypervior code came up for smartnic. | |
| 21:02:56 | sean-k-mooney | sorry i shoudl retype that but what i ment to say originaly having os-vif generate xml so that new nic could be added was proposed | |
| 21:03:05 | sean-k-mooney | it was also rejected | |
| 21:03:20 | Sundar_ | sean-k-mooney: Please look at #link https://libvirt.org/formatdomain.html#elementsHostDevSubsys . How would the virt driver determine XML elements like |
|
| 21:03:40 | Sundar_ | Those are device-specific and configuration-specific | |
| 21:04:05 | sean-k-mooney | Sundar_: today we already do that. at least part of it. | |
| 21:04:21 | sean-k-mooney | we would pass that infomation from neutron | |
| 21:04:27 | sean-k-mooney | in the vif_binding details | |
| 21:05:09 | efried | how does neutron know? | |
| 21:05:27 | efried | based on the ml2 thingy, or the plugin? | |
| 21:05:47 | sean-k-mooney | ml2 in general. | |
| 21:06:09 | Sundar_ | sean-k-mooney: That results in a double translation: first you have to fill in a VIF, and then translate that to another syntax. Since the final syntax is also well-defined, we may as well go to that directly | |
| 21:06:14 | sean-k-mooney | so for example ml2/ovs detect if ovs is kernel or dpdk and select vhost-user or a tap device | |
| 21:07:27 | Sundar_ | As devices and requirements both evolve, it would be faster to change os-acc alone rather than both os-acc and all virt drivers in Nova | |
| 21:07:29 | sean-k-mooney | Sundar_: yes its call the bridge pattern it allow both impmentation to vary indepently | |
| 21:08:28 | Sundar_ | We are not preventing independent evolution of the virt driver by providing XML snippets or equivalent | |
| 21:08:40 | Sundar_ | Restating: As devices and requirements both evolve, it would be faster to change os-acc alone rather than both os-acc and all virt drivers in Nova | |
| 21:09:15 | sean-k-mooney | Sundar_: will os-acc take into account the numa affinity of the ram and cpus selected by the driver as part of it xml generation | |
| 21:09:59 | sean-k-mooney | Sundar_: will it also account for the fact that if qe are using libvirt/qemu instead of libvirt/kvm we dont provide numa affintiy | |
| 21:10:37 | Sundar_ | That goes to the heart of it: NUMA affinity etc. would be stated in a device-specific way, and presumably will be in Cyborg or os-acc. Nova does not want to deal with such device-specific config | |
| 21:10:42 | sean-k-mooney | they are both libvirt but there is more copeling then i think you are hoping form to the internal of the libvirt driver | |
| 21:11:49 | Sundar_ | In your scheme, where would NUMA affinity for, say, a Quick Assist device be specified? | |
| 21:11:57 | sean-k-mooney | nova | |
| 21:12:30 | Sundar_ | But I thought Nova developers stated loud and clear that they don;t want to handle devices and their configurations | |
| 21:13:38 | sean-k-mooney | unless we are talking about moveing all passthough, pci,vgpu,pmem, sriov out of nova and out of the resouce tracker into a singel lib that is hypervior agnistic i dont see how it reduces compleity | |
| 21:14:24 | sean-k-mooney | there is a difference between phyical device management and virtualistation of the same | |
| 21:14:39 | sean-k-mooney | efried: what is your perspectiv | |
| 21:15:30 | Sundar_ | Back at the Ireland PTG, it was stated in one of the Nova sessions that handling PCI devices in general could be out of Nova. There is no clear roadmap for migrating device handling out of Nova, but I believe that's the general direction that people want to move | |
| 21:17:00 | sean-k-mooney | device handeling yes hypoervisor xml generation no | |
| 21:17:28 | efried | I'd like nova to be able to handle *scheduling* of devices generically like any other resource. But doing real stuff like attaching them to VMs, programming them, carving physical into virtual, configuring QoS, etc. - all that is going to need to be platform-specific (and sometimes device-specific code). There's no way around that. | |
| 21:17:49 | sean-k-mooney | and also i had suggested doing that back in denver by refactoring the pci code in the resouce tracker into a seperate lib with an ovo interrface and no rest or user facing api. | |
| 21:17:55 | efried | So it's going to need to be done in the virt driver, or some plugin. | |
| 21:19:22 | sean-k-mooney | efried: atching them to vms however is something the hypervior will do at the end of the day no? | |
| 21:20:17 | sean-k-mooney | the discovery is going to be device specific. in somecases the discover will only be posible via the hypervor | |
| 21:21:13 | sean-k-mooney | but in the later case that is because we are using something like hyperv or vspher where we cannont run code on the hypervior | |
| 21:21:31 | Sundar_ | There are lots of touch points between devices and CPU/memory, and they are going to grow and evolve over time. Having a restrictive API could constrain or slow such evolution. For example, shared virtual memory along the lines of CAPI | |
| 21:22:45 | sean-k-mooney | Sundar_: shared virtual memory beteen device and guest require you to generat the xml why | |
| 21:24:30 | Sundar_ | By shared virtual memory, I don;t mean DMA, but coherent memory where the device is participating in the processor's cache coherence protocol. AFAIK, that is not standard yet in libvirt. It is going to require further evolution. That will presumably result in some XML tags | |