| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-08-09 | |||
| 21:03:40 | Sundar_ | Those are device-specific and configuration-specific | |
| 21:04:05 | sean-k-mooney | Sundar_: today we already do that. at least part of it. | |
| 21:04:21 | sean-k-mooney | we would pass that infomation from neutron | |
| 21:04:27 | sean-k-mooney | in the vif_binding details | |
| 21:05:09 | efried | how does neutron know? | |
| 21:05:27 | efried | based on the ml2 thingy, or the plugin? | |
| 21:05:47 | sean-k-mooney | ml2 in general. | |
| 21:06:09 | Sundar_ | sean-k-mooney: That results in a double translation: first you have to fill in a VIF, and then translate that to another syntax. Since the final syntax is also well-defined, we may as well go to that directly | |
| 21:06:14 | sean-k-mooney | so for example ml2/ovs detect if ovs is kernel or dpdk and select vhost-user or a tap device | |
| 21:07:27 | Sundar_ | As devices and requirements both evolve, it would be faster to change os-acc alone rather than both os-acc and all virt drivers in Nova | |
| 21:07:29 | sean-k-mooney | Sundar_: yes its call the bridge pattern it allow both impmentation to vary indepently | |
| 21:08:28 | Sundar_ | We are not preventing independent evolution of the virt driver by providing XML snippets or equivalent | |
| 21:08:40 | Sundar_ | Restating: As devices and requirements both evolve, it would be faster to change os-acc alone rather than both os-acc and all virt drivers in Nova | |
| 21:09:15 | sean-k-mooney | Sundar_: will os-acc take into account the numa affinity of the ram and cpus selected by the driver as part of it xml generation | |
| 21:09:59 | sean-k-mooney | Sundar_: will it also account for the fact that if qe are using libvirt/qemu instead of libvirt/kvm we dont provide numa affintiy | |
| 21:10:37 | Sundar_ | That goes to the heart of it: NUMA affinity etc. would be stated in a device-specific way, and presumably will be in Cyborg or os-acc. Nova does not want to deal with such device-specific config | |
| 21:10:42 | sean-k-mooney | they are both libvirt but there is more copeling then i think you are hoping form to the internal of the libvirt driver | |
| 21:11:49 | Sundar_ | In your scheme, where would NUMA affinity for, say, a Quick Assist device be specified? | |
| 21:11:57 | sean-k-mooney | nova | |
| 21:12:30 | Sundar_ | But I thought Nova developers stated loud and clear that they don;t want to handle devices and their configurations | |
| 21:13:38 | sean-k-mooney | unless we are talking about moveing all passthough, pci,vgpu,pmem, sriov out of nova and out of the resouce tracker into a singel lib that is hypervior agnistic i dont see how it reduces compleity | |
| 21:14:24 | sean-k-mooney | there is a difference between phyical device management and virtualistation of the same | |
| 21:14:39 | sean-k-mooney | efried: what is your perspectiv | |
| 21:15:30 | Sundar_ | Back at the Ireland PTG, it was stated in one of the Nova sessions that handling PCI devices in general could be out of Nova. There is no clear roadmap for migrating device handling out of Nova, but I believe that's the general direction that people want to move | |
| 21:17:00 | sean-k-mooney | device handeling yes hypoervisor xml generation no | |
| 21:17:28 | efried | I'd like nova to be able to handle *scheduling* of devices generically like any other resource. But doing real stuff like attaching them to VMs, programming them, carving physical into virtual, configuring QoS, etc. - all that is going to need to be platform-specific (and sometimes device-specific code). There's no way around that. | |
| 21:17:49 | sean-k-mooney | and also i had suggested doing that back in denver by refactoring the pci code in the resouce tracker into a seperate lib with an ovo interrface and no rest or user facing api. | |
| 21:17:55 | efried | So it's going to need to be done in the virt driver, or some plugin. | |
| 21:19:22 | sean-k-mooney | efried: atching them to vms however is something the hypervior will do at the end of the day no? | |
| 21:20:17 | sean-k-mooney | the discovery is going to be device specific. in somecases the discover will only be posible via the hypervor | |
| 21:21:13 | sean-k-mooney | but in the later case that is because we are using something like hyperv or vspher where we cannont run code on the hypervior | |
| 21:21:31 | Sundar_ | There are lots of touch points between devices and CPU/memory, and they are going to grow and evolve over time. Having a restrictive API could constrain or slow such evolution. For example, shared virtual memory along the lines of CAPI | |
| 21:22:45 | sean-k-mooney | Sundar_: shared virtual memory beteen device and guest require you to generat the xml why | |
| 21:24:30 | Sundar_ | By shared virtual memory, I don;t mean DMA, but coherent memory where the device is participating in the processor's cache coherence protocol. AFAIK, that is not standard yet in libvirt. It is going to require further evolution. That will presumably result in some XML tags | |
| 21:24:36 | sean-k-mooney | Sundar_: looking at https://www.ibm.com/developerworks/aix/library/au-aix-achieve-performance-capi/index.html it looks like ivhsmem | |
| 21:24:57 | efried | sean-k-mooney: "run code on the hypervisor" is not something nova should ever, ever do. and by nova I mean "outside of the virt driver". The virt driver can do it; or some platform-specific plugin can do it. | |
| 21:25:43 | sean-k-mooney | efried: yes i am fine with that. | |
| 21:27:11 | sean-k-mooney | efried: when i was refering to running code on the hypervieros i was refing to the nova-compute agent and how it typically deployed on the hypervior with libvirt | |
| 21:27:47 | sean-k-mooney | im aware for ironic and orther hyperviros that the compute agent is not always colocated with the hypervior | |
| 21:28:42 | sean-k-mooney | although i also thend to reseve the them hypervior for the softare layer and not the phyical server that the vm/compute context executs on | |
| 21:29:33 | efried | By definition, the compute service runs on a system that knows how to talk to the hypervisor. Whether that's by exec'ing commands or communicating over a REST API. And either way, it's the responsibility of platform-specific code to do whatever that is. | |
| 21:30:49 | Sundar_ | efried: What is your take w.r.t Power/CAPI/IBM technologies on passing just the PCI address (or mediated device UUID, or some such) from os-acc to the virt driver? | |
| 21:31:15 | sean-k-mooney | efried: yes and by said definition the nova compute agent should only interfact with the phyical system through the api exposed by the hypervior that driver is talking to though there are some expetion to that in the libvirt case | |
| 21:32:37 | sean-k-mooney | Sundar_: in the CAPI case if this was libvirt kvm. the CAPI protocal would be impemnted between qemu and the device not libvirt and would require passing the memory regoins to qemu. | |
| 21:33:18 | Sundar_ | How would such device-specific data like memory regions be communicated to qemu? | |
| 21:33:26 | sean-k-mooney | Sundar_: we map a dma region in a simlar fashion for vhost user. in that case qemu passes the virt to phyical page traslation table to dpdk via the unix socket | |
| 21:33:42 | efried | Sundar_: Passing just the <identifier> to the virt driver for what operation? Plug? That works as long as there's some other place at which any necessary configuration has been done. Which again would need to be platform-specific code. | |
| 21:34:39 | sean-k-mooney | efried: vhost-user is useing a shared memory interface via dma to hugepages without ever passing that memory mapping to nova | |
| 21:35:11 | sean-k-mooney | the mappings are negociated between the qemu frontend and the dpdk backend | |
| 21:35:39 | Sundar_ | My model is that Cyborg discovers devices and their properties. There needs to be some way to communicate their properties and requirements to the hypervisor. We could extend the VAN with specific fields for each such property/requirement, or ... | |
| 21:35:58 | Sundar_ | we could use whatever the corresponding hypervisor already expects, as long as it is standard | |
| 21:38:56 | sean-k-mooney | how many different protocls do you expect to have to supprot | |
| 21:40:14 | sean-k-mooney | we will have mmio, ivshmem and capi but seam to take a simlar approch to do mmio via pci bars | |
| 21:41:15 | sean-k-mooney | we will have vfio-mediated devices | |
| 21:41:48 | sean-k-mooney | we will have pci passthough of different kinds. | |
| 21:41:53 | Sundar_ | That is because PCI MMIO is standardized. Whereas newer things like OpenCAPI and device-coherent SVM are all relatively new and likely to grow. The short answer is, I don't know how many such protocols there will be, nor do I think that anybody can tell you for sure today | |
| 21:43:23 | sean-k-mooney | Sundar_: so you are porposing creating a libary that needs to support not just multiple protocols, but multiple hyperviors, and multiple versions of those hyperviors instad of create an abstration layer container the common datataype | |
| 21:43:30 | Sundar_ | Going back to the Nova's philosophy, I think Nova wants to be agnostic of devices. Even things like NUMA affinity of devices may need to be modularized to minimize Nova involvement | |
| 21:46:06 | Sundar_ | 'support multiple hypervisors' -- yes, by using plugins/drivers. The existing Nova virt drivers are not (or should not be) device-aware. The os-acc plugins are expected to fill that need | |
| 21:47:44 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/os-vif-library.html#proposed-change if your spec does not already contain a detailed breakdown of the split of responiblities as was done for os-vif can i ask that you add one | |
| 21:48:26 | sean-k-mooney | Sundar_: i think the only interface between opentack and the hypervior should be via the nova virt driver | |
| 21:49:05 | sean-k-mooney | at least wehn it comre to the virtualisation fo the guest | |
| 21:52:34 | Sundar_ | Yes, ultimately the Nova virt driver handles all hypervisor-related actions for launching an instance. The os-acc (or its plugins) are helpers -- specific areas are delegated by Nova virt to os-acc. | |
| 21:52:54 | Sundar_ | The os-acc should not directly modift the instance XML, for instanc | |
| 21:53:33 | Sundar_ | The split of ownership can be added to the spec. Sure. | |
| 21:53:43 | Sundar_ | Does that address your concern? | |
| 21:54:11 | sean-k-mooney | Sundar_: if the nova-virt driver calls oss acc and is returned an ovo with then the virt driver uses that to generate the xml or the equvalent for another driver then yes. | |
| 21:54:57 | sean-k-mooney | but only there is a clear seperation between preparing the device. and generating the hyperviors specific configration input | |
| 22:02:18 | Sundar_ | I am wondering how the separation would be so clear-cut for device/cpu/memory interaction, such as conveying device properties for NUMA etc. while keeping Nova device-agnostic. Anyways, I need to join a call now. We can touch base later. | |
| 22:03:27 | Sundar_ | Thanks, efried and sean-k-mooney | |
| #openstack-cyborg - 2018-08-10 | |||
| 04:12:47 | openstackgerrit | Merged openstack/cyborg master: Fix deployable get all sort issue and unit test failure. https://review.openstack.org/590075 | |
| 05:17:47 | openstackgerrit | YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590647 | |
| 05:26:52 | openstackgerrit | YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590647 | |
| 08:02:17 | openstackgerrit | ShaoHe Feng proposed openstack/cyborg master: support sub provider https://review.openstack.org/585146 | |
| 08:54:10 | openstackgerrit | YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706 | |
| 08:55:13 | openstackgerrit | YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706 | |
| 12:07:53 | openstackgerrit | zhurong proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706 | |
| 14:43:11 | openstackgerrit | Merged openstack/cyborg master: support sub provider https://review.openstack.org/585146 | |
| 16:06:10 | annabelleB | zhipengh[m]: I’m reviewing your cycle-highlights now; does the “quota support” you mention on line 30 correspond to this review? https://review.openstack.org/#/c/564968/ | |
| #openstack-cyborg - 2018-08-11 | |||
| 04:28:15 | openstackgerrit | YumengBao proposed openstack/cyborg master: Update html_theme so cyborg doc page displays in standard page theme https://review.openstack.org/591120 | |
| 04:56:03 | openstackgerrit | YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706 | |
| #openstack-cyborg - 2018-08-13 | |||
| 08:27:55 | openstackgerrit | Merged openstack/cyborg master: Update html_theme so cyborg doc page displays in standard page theme https://review.openstack.org/591120 | |
| 10:26:14 | openstackgerrit | Merged openstack/os-acc master: Update reno for stable/rocky https://review.openstack.org/584362 | |
| 13:37:10 | openstackgerrit | Merged openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706 | |
| 14:01:10 | shaohe_feng | #startmeeting openstack-cyborg-driver | |
| 14:01:11 | openstack | Meeting started Mon Aug 13 14:01:10 2018 UTC and is due to finish in 60 minutes. The chair is shaohe_feng. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 14:01:13 | shaohe_feng | #topic Roll Call | |
| 14:01:13 | openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 14:01:15 | openstack | The meeting name has been set to 'openstack_cyborg_driver' | |
| 14:01:40 | shaohe_feng | #info shaohe_feng | |
| 14:02:16 | edleafe | #info ed leafe | |
| 14:05:13 | Sundar | #info Sundar | |
| 14:05:54 | shaohe_feng | Sundar, edleafe morning | |
| 14:06:10 | edleafe | good UGT morning to you! | |
| 14:06:36 | shaohe_feng | :) | |
| 14:06:43 | Sundar | Good day, Shaohe | |
| 14:07:50 | shaohe_feng | edleafe, Sundar now only we three on line. | |
| 14:08:28 | Li_Liu | #info Li_Liu | |
| 14:08:40 | Li_Liu | do we still have the meeting today? | |
| 14:08:41 | Sundar | shaohe: I had sent an email yesterday to openstack-dev. Shall we talk about that? | |