Earlier  
Later
New Search
Posted Nick Remark
#openstack-cyborg - 2018-08-09
21:10:37 Sundar_ That goes to the heart of it: NUMA affinity etc. would be stated in a device-specific way, and presumably will be in Cyborg or os-acc. Nova does not want to deal with such device-specific config
21:10:42 sean-k-mooney they are both libvirt but there is more copeling then i think you are hoping form to the internal of the libvirt driver
21:11:49 Sundar_ In your scheme, where would NUMA affinity for, say, a Quick Assist device be specified?
21:11:57 sean-k-mooney nova
21:12:30 Sundar_ But I thought Nova developers stated loud and clear that they don;t want to handle devices and their configurations
21:13:38 sean-k-mooney unless we are talking about moveing all passthough, pci,vgpu,pmem, sriov out of nova and out of the resouce tracker into a singel lib that is hypervior agnistic i dont see how it reduces compleity
21:14:24 sean-k-mooney there is a difference between phyical device management and virtualistation of the same
21:14:39 sean-k-mooney efried: what is your perspectiv
21:15:30 Sundar_ Back at the Ireland PTG, it was stated in one of the Nova sessions that handling PCI devices in general could be out of Nova. There is no clear roadmap for migrating device handling out of Nova, but I believe that's the general direction that people want to move
21:17:00 sean-k-mooney device handeling yes hypoervisor xml generation no
21:17:28 efried I'd like nova to be able to handle *scheduling* of devices generically like any other resource. But doing real stuff like attaching them to VMs, programming them, carving physical into virtual, configuring QoS, etc. - all that is going to need to be platform-specific (and sometimes device-specific code). There's no way around that.
21:17:49 sean-k-mooney and also i had suggested doing that back in denver by refactoring the pci code in the resouce tracker into a seperate lib with an ovo interrface and no rest or user facing api.
21:17:55 efried So it's going to need to be done in the virt driver, or some plugin.
21:19:22 sean-k-mooney efried: atching them to vms however is something the hypervior will do at the end of the day no?
21:20:17 sean-k-mooney the discovery is going to be device specific. in somecases the discover will only be posible via the hypervor
21:21:13 sean-k-mooney but in the later case that is because we are using something like hyperv or vspher where we cannont run code on the hypervior
21:21:31 Sundar_ There are lots of touch points between devices and CPU/memory, and they are going to grow and evolve over time. Having a restrictive API could constrain or slow such evolution. For example, shared virtual memory along the lines of CAPI
21:22:45 sean-k-mooney Sundar_: shared virtual memory beteen device and guest require you to generat the xml why
21:24:30 Sundar_ By shared virtual memory, I don;t mean DMA, but coherent memory where the device is participating in the processor's cache coherence protocol. AFAIK, that is not standard yet in libvirt. It is going to require further evolution. That will presumably result in some XML tags
21:24:36 sean-k-mooney Sundar_: looking at https://www.ibm.com/developerworks/aix/library/au-aix-achieve-performance-capi/index.html it looks like ivhsmem
21:24:57 efried sean-k-mooney: "run code on the hypervisor" is not something nova should ever, ever do. and by nova I mean "outside of the virt driver". The virt driver can do it; or some platform-specific plugin can do it.
21:25:43 sean-k-mooney efried: yes i am fine with that.
21:27:11 sean-k-mooney efried: when i was refering to running code on the hypervieros i was refing to the nova-compute agent and how it typically deployed on the hypervior with libvirt
21:27:47 sean-k-mooney im aware for ironic and orther hyperviros that the compute agent is not always colocated with the hypervior
21:28:42 sean-k-mooney although i also thend to reseve the them hypervior for the softare layer and not the phyical server that the vm/compute context executs on
21:29:33 efried By definition, the compute service runs on a system that knows how to talk to the hypervisor. Whether that's by exec'ing commands or communicating over a REST API. And either way, it's the responsibility of platform-specific code to do whatever that is.
21:30:49 Sundar_ efried: What is your take w.r.t Power/CAPI/IBM technologies on passing just the PCI address (or mediated device UUID, or some such) from os-acc to the virt driver?
21:31:15 sean-k-mooney efried: yes and by said definition the nova compute agent should only interfact with the phyical system through the api exposed by the hypervior that driver is talking to though there are some expetion to that in the libvirt case
21:32:37 sean-k-mooney Sundar_: in the CAPI case if this was libvirt kvm. the CAPI protocal would be impemnted between qemu and the device not libvirt and would require passing the memory regoins to qemu.
21:33:18 Sundar_ How would such device-specific data like memory regions be communicated to qemu?
21:33:26 sean-k-mooney Sundar_: we map a dma region in a simlar fashion for vhost user. in that case qemu passes the virt to phyical page traslation table to dpdk via the unix socket
21:33:42 efried Sundar_: Passing just the <identifier> to the virt driver for what operation? Plug? That works as long as there's some other place at which any necessary configuration has been done. Which again would need to be platform-specific code.
21:34:39 sean-k-mooney efried: vhost-user is useing a shared memory interface via dma to hugepages without ever passing that memory mapping to nova
21:35:11 sean-k-mooney the mappings are negociated between the qemu frontend and the dpdk backend
21:35:39 Sundar_ My model is that Cyborg discovers devices and their properties. There needs to be some way to communicate their properties and requirements to the hypervisor. We could extend the VAN with specific fields for each such property/requirement, or ...
21:35:58 Sundar_ we could use whatever the corresponding hypervisor already expects, as long as it is standard
21:38:56 sean-k-mooney how many different protocls do you expect to have to supprot
21:40:14 sean-k-mooney we will have mmio, ivshmem and capi but seam to take a simlar approch to do mmio via pci bars
21:41:15 sean-k-mooney we will have vfio-mediated devices
21:41:48 sean-k-mooney we will have pci passthough of different kinds.
21:41:53 Sundar_ That is because PCI MMIO is standardized. Whereas newer things like OpenCAPI and device-coherent SVM are all relatively new and likely to grow. The short answer is, I don't know how many such protocols there will be, nor do I think that anybody can tell you for sure today
21:43:23 sean-k-mooney Sundar_: so you are porposing creating a libary that needs to support not just multiple protocols, but multiple hyperviors, and multiple versions of those hyperviors instad of create an abstration layer container the common datataype
21:43:30 Sundar_ Going back to the Nova's philosophy, I think Nova wants to be agnostic of devices. Even things like NUMA affinity of devices may need to be modularized to minimize Nova involvement
21:46:06 Sundar_ 'support multiple hypervisors' -- yes, by using plugins/drivers. The existing Nova virt drivers are not (or should not be) device-aware. The os-acc plugins are expected to fill that need
21:47:44 sean-k-mooney https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/os-vif-library.html#proposed-change if your spec does not already contain a detailed breakdown of the split of responiblities as was done for os-vif can i ask that you add one
21:48:26 sean-k-mooney Sundar_: i think the only interface between opentack and the hypervior should be via the nova virt driver
21:49:05 sean-k-mooney at least wehn it comre to the virtualisation fo the guest
21:52:34 Sundar_ Yes, ultimately the Nova virt driver handles all hypervisor-related actions for launching an instance. The os-acc (or its plugins) are helpers -- specific areas are delegated by Nova virt to os-acc.
21:52:54 Sundar_ The os-acc should not directly modift the instance XML, for instanc
21:53:33 Sundar_ The split of ownership can be added to the spec. Sure.
21:53:43 Sundar_ Does that address your concern?
21:54:11 sean-k-mooney Sundar_: if the nova-virt driver calls oss acc and is returned an ovo with then the virt driver uses that to generate the xml or the equvalent for another driver then yes.
21:54:57 sean-k-mooney but only there is a clear seperation between preparing the device. and generating the hyperviors specific configration input
22:02:18 Sundar_ I am wondering how the separation would be so clear-cut for device/cpu/memory interaction, such as conveying device properties for NUMA etc. while keeping Nova device-agnostic. Anyways, I need to join a call now. We can touch base later.
22:03:27 Sundar_ Thanks, efried and sean-k-mooney
#openstack-cyborg - 2018-08-10
04:12:47 openstackgerrit Merged openstack/cyborg master: Fix deployable get all sort issue and unit test failure. https://review.openstack.org/590075
05:17:47 openstackgerrit YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590647
05:26:52 openstackgerrit YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590647
08:02:17 openstackgerrit ShaoHe Feng proposed openstack/cyborg master: support sub provider https://review.openstack.org/585146
08:54:10 openstackgerrit YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706
08:55:13 openstackgerrit YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706
12:07:53 openstackgerrit zhurong proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706
14:43:11 openstackgerrit Merged openstack/cyborg master: support sub provider https://review.openstack.org/585146
16:06:10 annabelleB zhipengh[m]: I’m reviewing your cycle-highlights now; does the “quota support” you mention on line 30 correspond to this review? https://review.openstack.org/#/c/564968/
#openstack-cyborg - 2018-08-11
04:28:15 openstackgerrit YumengBao proposed openstack/cyborg master: Update html_theme so cyborg doc page displays in standard page theme https://review.openstack.org/591120
04:56:03 openstackgerrit YumengBao proposed openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706
#openstack-cyborg - 2018-08-13
08:27:55 openstackgerrit Merged openstack/cyborg master: Update html_theme so cyborg doc page displays in standard page theme https://review.openstack.org/591120
10:26:14 openstackgerrit Merged openstack/os-acc master: Update reno for stable/rocky https://review.openstack.org/584362
13:37:10 openstackgerrit Merged openstack/cyborg master: Follow the new PTI for document build https://review.openstack.org/590706
14:01:10 shaohe_feng #startmeeting openstack-cyborg-driver
14:01:11 openstack Meeting started Mon Aug 13 14:01:10 2018 UTC and is due to finish in 60 minutes. The chair is shaohe_feng. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:13 shaohe_feng #topic Roll Call
14:01:13 openstack Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:15 openstack The meeting name has been set to 'openstack_cyborg_driver'
14:01:40 shaohe_feng #info shaohe_feng
14:02:16 edleafe #info ed leafe
14:05:13 Sundar #info Sundar
14:05:54 shaohe_feng Sundar, edleafe morning
14:06:10 edleafe good UGT morning to you!
14:06:36 shaohe_feng :)
14:06:43 Sundar Good day, Shaohe
14:07:50 shaohe_feng edleafe, Sundar now only we three on line.
14:08:28 Li_Liu #info Li_Liu
14:08:40 Li_Liu do we still have the meeting today?
14:08:41 Sundar shaohe: I had sent an email yesterday to openstack-dev. Shall we talk about that?
14:09:01 shaohe_feng Sundar: OK, go ahead.
14:09:16 shaohe_feng Li_Liu: morning.
14:09:46 shaohe_feng good evening, xinran
14:09:48 Li_Liu shaohe_feng: good evening :P
14:09:58 xinran Hi all
14:10:33 shaohe_feng Sundar: You can talk about your email
14:10:40 Sundar It was apparently decided in some meeting that, to record the discovered devices, Cyborg agent will call Cyborg REST API. Also, to allocate and deallocate accelerators. If so, that will make the public and ha many disadvantages.
14:11:45 Sundar Among other things, it means it is not internal to Cyborg any more. Any user can call it. So, we should authenticate. Even if we open it only to operators, it is still error-prone. We can just keep it Cyborg-internal, right?
14:12:43 xinran When Cyborg agent will call restful API?
14:13:38 shaohe_feng Cyborg agent should not call Cyborg REST API.
14:13:53 Li_Liu Agent should stay with rpc
14:13:59 Sundar xinran: Rest API is meant for things that can be accessed by external users, operators or other services.
14:14:44 xinran Sundar: can you give us an example when agent call restful api
14:15:31 Sundar shaohe, Li_Liu: Agreed. I amreferring to #link https://etherpad.openstack.org/p/cyborg-rocky-development (Line 44)
14:15:38 wangzhh Hi all. Test connection.

Earlier   Later