Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-14
14:21:31 sdague because it really really seems far on the edge
14:21:52 sdague and it gets a disproportunate amount of debugging time relative to use
14:23:08 mriedem i was hoping we could just pull from INFO logs in n-api for when a POST request to shelve comes in, but because it's an 'action' API and the bit we care about is in the body, we don't have that info...
14:24:09 mriedem i'm fine with noting limitations in the api, we do that for some other things as well, or should: https://review.openstack.org/#/c/452970/
14:24:19 mriedem https://review.openstack.org/#/c/452970/2/api-ref/source/servers-admin-action.inc
14:24:31 sdague mriedem: yep, that's one of those issues. I do think it's elsewhere though. Anyway, need to afk for a bit.
14:26:00 feeblepipes mriedem: happen to know Jan Gutter's IRC nick?
14:26:07 mriedem jangutter
14:26:14 feeblepipes mriedem: k, thx.
14:26:30 mriedem sdague: on "an idea I was thinking about the other day was to actually build something into nova that builds API stats in a format that's easy to annonymously dump back out" - thinking about that, i think we already have that, it's instance actions
14:27:15 mriedem so compute_shelve_instance is an action event
14:27:15 feeblepipes mriedem: pls see my -1 and comments from that netronome spec. I'm cool with it, but have concerns about the PCI device allocation code proposed for neutronv2/api.py module.
14:31:03 mriedem feeblepipes: they just mean this i think https://review.openstack.org/#/c/456481/1/nova/network/neutronv2/api.py
14:31:14 mriedem in addition to handling direct-physical, handle this other vnic type
14:31:30 mriedem that would need to be cleaned up a bit and non-vendorized,
14:31:34 mriedem but i think the spec notes that
14:35:44 feeblepipes mriedem: yeah, not really a fan of that approach as it embeds way too much vendor/product specific stuff into the model. It shouldn't matter what vendor/product_id is used underneath the covers. all that should matter is what *functionality* the end user can derive from the particular vNIC. so the things that should be passed from the neutronv2 api should really only be feature requests/capabilities, not vendor/product information.
14:35:49 feeblepipes mriedem: make sense?
14:36:59 cdent feeblepipes++
14:37:38 mriedem feeblepipes: so you're also saying you're not a fan of the existing direct-physical thing we already have in that code?
14:37:42 mriedem which i'm pretty sure doesn't work...
14:37:53 mriedem because a week after it landed, nikola created 2 bugs about it not working
14:38:10 feeblepipes mriedem: correct, I wasn't a fan of it when it entered the code and I'm still not a fan of it. you can check the review on that if you'd like :)
14:38:11 mriedem either way i'll totally defer here
14:38:49 mriedem i don't really know what the alternative implementation is, they are going to need some help there
14:38:58 feeblepipes mriedem: again, I'm not opposed to the netronome stuff at all. just have reservations (pun intended) about this particular part of th eproposal.
14:39:18 feeblepipes mriedem: yeah, I'll wait for Jan to answer and think about alternatives to recommedn.
14:40:23 feeblepipes mriedem: also note that the direct-physical thing is *only* passing through a network tag (the physnet tag) from Neutron's port profile. It isn't passing a vendor-specific piece of information.
14:41:30 feeblepipes mriedem: or perhaps you were referring to a different part of the codebase w.r.t. the direct-physical thing?
14:42:46 mriedem no i was just pointing at that thing
14:43:06 mriedem this part dev_type = pci_request.DEVICE_TYPE_FOR_VNIC_TYPE.get(vnic_type)
14:43:11 feeblepipes right
14:43:19 mriedem pci_request.DEVICE_TYPE_FOR_VNIC_TYPE is only mapped to direct-physical
14:43:25 feeblepipes yes, understood.
14:44:06 jangutter Oh hey, thanks for taking the time to review!
14:44:14 feeblepipes jangutter: hi :)
14:44:26 feeblepipes jangutter: sorry for the delay. been quite ill last few days :(
14:45:16 jangutter feeblepipes: not a problem, it took waaay too long on our side to jump the necessary hurdles. No-one to blame but me.
14:46:33 jangutter feeblepipes: yeah, regarding that allocation, the idea is not to put in a new place for it, but to allow port profiles that are "non-SR-IOV" to also request pci devices.
14:49:18 jangutter feeblepipes: and you're absolutely right, it is the crux of the spec. I had looked at other places, but most of them would happen after scheduling had already occurred.
14:49:30 openstackgerrit Ildiko Vancsa proposed openstack/nova master: Add Cinder v3 detach call to _terminate_volume_connections https://review.openstack.org/456896
14:49:33 feeblepipes jangutter: see what I wrote to mriedem above about being pretty firm about keeping the information that we pass about the vNIC as vendor-agnostic as possible (and focusing instead on the *features* or functionality that the underlying technology supports)
14:49:38 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Idempotent PUT /resource_classes/{name} https://review.openstack.org/448791
14:49:39 openstackgerrit Chris Dent proposed openstack/nova master: Update resource tracker to PUT custom resource classes https://review.openstack.org/456915
14:50:10 openstackgerrit Jan Gutter proposed openstack/nova-specs master: Spec to enable Netronome SmartNIC support https://review.openstack.org/456386
14:50:33 feeblepipes jangutter: what I'd like to see is a description of those features of the virtual switch technology that we could use instead of specific PCI vendor_id/product_id information.
14:52:00 jangutter Yep, so after a discussion yesterday with sean-k-mooney, it looks like it should be possible to push that check out to the OS-VIF side.
14:52:34 feeblepipes jangutter: another way to think about things... what is the difference between network_model.VNIC_TYPE_AGILIO_PASSTHROUGH and network_model.VNIC_TYPE_DIRECT or, frankly, network_model.VNIC_TYPE_DIRECT_PHYSICAL?
14:54:22 jangutter feeblepipes: Those two AGILIO VNIC types were added to in an effort not to re-use internal OpenStack code: they should go completely away.
14:56:00 jangutter feeblepipes: The idea is to have all the necessary whitelist info sent down via the os-vif port profile, so that the API will do matching.
14:57:24 feeblepipes jangutter: ok. that's cool with me. How exactly, however, do you plan to keep vendor_id/product_id information out of the os-vif modeling? In other words, what I'm looking for is an indication that you've thought of how to communicate the *functionality* that Agilio SmartNICs provide to the end user instead of passing a PCI vendor_id/product_id :)
14:58:59 jangutter feeblepipes: That's done in a couple of ways: a deployer runs a script that generates a nova.conf fragment, this adds the VF's to the compute node with in a full pci-whitelist format.
14:59:37 feeblepipes jangutter: sure, understood. that merely lets nova-compute "use" the Agilio NICs for tenants.
15:00:08 jangutter feeblepipes: Secondly, the OS-VIF object for the external plugin gets a set of fields. Likely this would be vendor/productids, we'll definitely have to coordinate with Mellanox on that.
15:00:47 jangutter feeblepipes: so when the API code at that place hits, the port profile is available, and the pci devices has been registered.
15:01:06 feeblepipes jangutter: right, so that's the thing I'd be -1 on :) we should definitely work with Moshe and others from Mellanox to ensure vendor_id/product_id information isn't needed to be passed via os-vif models.
15:03:45 feeblepipes jangutter: what I'm looking for is an object model that says "hey, I need a VIF plug that supports feature X, Y, and Z", not an object model that says "hey, I need a VIF plug that is specific to Mellanox Indigo NPS"
15:03:52 feeblepipes jangutter: that make sense?
15:03:53 jangutter feeblepipes: hmmm... that's interesting... you just made me think quite a lot...
15:04:34 jangutter feeblepipes: that would basically mean a "capability set" sort-of?
15:04:40 feeblepipes jangutter: BINGO :)
15:05:36 feeblepipes jangutter: ala the traits/capabilities that have begun to emerge here: https://review.openstack.org/#/c/448283/4
15:06:17 jangutter feeblepipes: I can understand the appeal, but WOW, the parameter space is _big_.
15:06:29 feeblepipes jangutter: so, as an example, the Neutron port profile and os-vif object model would, say, have a list of these acceleration traits: https://review.openstack.org/#/c/448283/4/os_traits/hw/nic/accel.py
15:06:42 feeblepipes that would be required to match to an available PCI device on the target host.
15:07:52 feeblepipes jangutter: and Netronome and Mellanox would add a set of traits/features that their NICs supported to https://review.openstack.org/#/c/448283/4/os_traits/hw/nic/sriov.py
15:07:58 jangutter feeblepipes: I'm just worried that the acceleration traits would not dovetail much. For example, our NIC's got an insane amount of flexibility. The same HW can support many different features.
15:09:49 jangutter feeblepipes: our internal lingo for the firmware running on the chip is "application" - the image I get when l think of negotiating capabilities is some insane duel.
15:10:11 feeblepipes jangutter: an end user, though, only wants to launch a VM on a host that has a NIC with certain features. an end user doesn't say "hey, I want a VM with a Netronome Agilio SmartNIC" (because, frankly, the end user doesn't *know* that the underlying hardware has Agilio NICs). Instead, the end user says "hey, give me a VM that has access to an SR-IOV virtual function that has a particular set of features"
15:11:05 jangutter feeblepipes: Ah, I've been confusing operators with end-users.
15:11:47 feeblepipes jangutter: no worries. it's a common NFV problem :) in NFV land, there is no end user. The only user of the system is a privileged administrator user embodied in the MANO/VNFM system.
15:12:50 feeblepipes jangutter: but Nova is for end users -- the people/systems that launch VMs. So, we're constantly trying to shape the new NFV-specific features into a world that "normal users" can understand and utilize via the Nova APIs.
15:12:57 jangutter feeblepipes: OK, let's take the current flat/vlan SR-IOV mechanism as an example. An end user creates a port on a particular network and specs "it should be direct". (I think "direct" tends to go with VF's and "direct-physical" tend to go with PFs).
15:13:17 feeblepipes ok, continue
15:13:47 jangutter feeblepipes: if there are a mix of NICs then some instances would pop up with one type of driver required, and others would pop up with another, correct?
15:15:08 jangutter feeblepipes: It should be possible to do some flavor/image filtering or something, to match the flavors with the drivers they support?
15:16:06 feeblepipes jangutter: if there are a mix of NICs, they should be differentiated by their feature set, not by their vendor_id/product_id.
15:17:18 feeblepipes jangutter: so, for example, if a user launches a VM with nova boot --net port_id=<PORT_UUID> and that port has binding information that constructs a PCI request, the stuff in that PCI request should be a feature set, not a vendor_id/product_id. That's what I'm getting at :)
15:17:57 jangutter feeblepipes: with iommu passthrough, the instances get a raw PCI device, with the vendor/product exposed. I've been looking at that tuple as "a feature" :-p
15:18:00 feeblepipes jangutter: and that feature set is what should be used by the scheduler and the nova-compute pci device manager to match a particular NIC among many on the host.
15:18:30 feeblepipes jangutter: what about having "iommu_passthrough" as the feature, then?
15:19:44 jangutter feeblepipes: well, iommu_passthrough won't tell us anything of the device type that's going to appear inside the VM.
15:19:46 feeblepipes jangutter: would an end-user say "I want a vNIC that is associated to a VF that has IOMMU passthrough features" or would the end user say "I want a vNIC that is associated with a VF that is a Netronome Agilio SmartNIC PF"?
15:23:10 jangutter feeblepipes: my view of one usecase is this: an end-user says "I want a vNIC, associated with a VF, passed through via IOMMU, with <insert-hw-driver-here> PCI driver, additionally pin everything for me optimally, pretty please!"
15:23:29 feeblepipes :)
15:23:42 feeblepipes jangutter: ok, cool. we're getting somewhere here :)
15:23:56 jangutter feeblepipes: with DPDK applications being specifically compiled for specific PCI drivers, it's sometimes a _must_.
15:24:11 feeblepipes jangutter: the <insert-hw-driver-here> goes in the *image metadata*, though.
15:25:04 feeblepipes jangutter: because the image metadata is what describes the workload to be launched. that image metadata can describe the HW driver's installed on the image, and that information can/should be used by os-vif in setting up the VIF plug on the host.
15:25:07 jangutter feeblepipes: yeah, but that leads to the other bit of fun. Give me one vNIC on control, standard TAP, and vNIC 2 for my DPDK NFV application.
15:25:14 feeblepipes jangutter: so I think we're *almost* there.
15:25:46 jangutter feeblepipes: is it possible to spec stuff that fine-grained?
15:26:02 feeblepipes jangutter: ah, indeed. so perhaps that information (hw driver) does need to be in the port binding profile.
15:26:18 feeblepipes jangutter: yes, it is possible. via the port profile information.
15:26:34 feeblepipes jangutter: I still don't think we need to have vendor_id/product_id in there, though. :)
15:26:54 feeblepipes jangutter: in any case... I think this is also something we can hash out in the code rather than the spec.
15:27:33 jangutter feeblepipes: I am definitely not averse to having a "driver list". That's actually a pretty good idea from a user's perspective.
15:27:34 feeblepipes jangutter: I'm willing to +2 the spec on condition that you just place a note in the spec that we've discussed on IRC how to make the information passed between Neutron and Nova via os-vif as generic as possible.
15:27:50 feeblepipes jangutter: if you can do that quick, I'll +2 it and mriedem can as well.
15:30:53 mriedem sfinucan: how do i reference another link within the same docs using rst again?

Earlier   Later