Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-14
14:58:59 jangutter feeblepipes: That's done in a couple of ways: a deployer runs a script that generates a nova.conf fragment, this adds the VF's to the compute node with in a full pci-whitelist format.
14:59:37 feeblepipes jangutter: sure, understood. that merely lets nova-compute "use" the Agilio NICs for tenants.
15:00:08 jangutter feeblepipes: Secondly, the OS-VIF object for the external plugin gets a set of fields. Likely this would be vendor/productids, we'll definitely have to coordinate with Mellanox on that.
15:00:47 jangutter feeblepipes: so when the API code at that place hits, the port profile is available, and the pci devices has been registered.
15:01:06 feeblepipes jangutter: right, so that's the thing I'd be -1 on :) we should definitely work with Moshe and others from Mellanox to ensure vendor_id/product_id information isn't needed to be passed via os-vif models.
15:03:45 feeblepipes jangutter: what I'm looking for is an object model that says "hey, I need a VIF plug that supports feature X, Y, and Z", not an object model that says "hey, I need a VIF plug that is specific to Mellanox Indigo NPS"
15:03:52 feeblepipes jangutter: that make sense?
15:03:53 jangutter feeblepipes: hmmm... that's interesting... you just made me think quite a lot...
15:04:34 jangutter feeblepipes: that would basically mean a "capability set" sort-of?
15:04:40 feeblepipes jangutter: BINGO :)
15:05:36 feeblepipes jangutter: ala the traits/capabilities that have begun to emerge here: https://review.openstack.org/#/c/448283/4
15:06:17 jangutter feeblepipes: I can understand the appeal, but WOW, the parameter space is _big_.
15:06:29 feeblepipes jangutter: so, as an example, the Neutron port profile and os-vif object model would, say, have a list of these acceleration traits: https://review.openstack.org/#/c/448283/4/os_traits/hw/nic/accel.py
15:06:42 feeblepipes that would be required to match to an available PCI device on the target host.
15:07:52 feeblepipes jangutter: and Netronome and Mellanox would add a set of traits/features that their NICs supported to https://review.openstack.org/#/c/448283/4/os_traits/hw/nic/sriov.py
15:07:58 jangutter feeblepipes: I'm just worried that the acceleration traits would not dovetail much. For example, our NIC's got an insane amount of flexibility. The same HW can support many different features.
15:09:49 jangutter feeblepipes: our internal lingo for the firmware running on the chip is "application" - the image I get when l think of negotiating capabilities is some insane duel.
15:10:11 feeblepipes jangutter: an end user, though, only wants to launch a VM on a host that has a NIC with certain features. an end user doesn't say "hey, I want a VM with a Netronome Agilio SmartNIC" (because, frankly, the end user doesn't *know* that the underlying hardware has Agilio NICs). Instead, the end user says "hey, give me a VM that has access to an SR-IOV virtual function that has a particular set of features"
15:11:05 jangutter feeblepipes: Ah, I've been confusing operators with end-users.
15:11:47 feeblepipes jangutter: no worries. it's a common NFV problem :) in NFV land, there is no end user. The only user of the system is a privileged administrator user embodied in the MANO/VNFM system.
15:12:50 feeblepipes jangutter: but Nova is for end users -- the people/systems that launch VMs. So, we're constantly trying to shape the new NFV-specific features into a world that "normal users" can understand and utilize via the Nova APIs.
15:12:57 jangutter feeblepipes: OK, let's take the current flat/vlan SR-IOV mechanism as an example. An end user creates a port on a particular network and specs "it should be direct". (I think "direct" tends to go with VF's and "direct-physical" tend to go with PFs).
15:13:17 feeblepipes ok, continue
15:13:47 jangutter feeblepipes: if there are a mix of NICs then some instances would pop up with one type of driver required, and others would pop up with another, correct?
15:15:08 jangutter feeblepipes: It should be possible to do some flavor/image filtering or something, to match the flavors with the drivers they support?
15:16:06 feeblepipes jangutter: if there are a mix of NICs, they should be differentiated by their feature set, not by their vendor_id/product_id.
15:17:18 feeblepipes jangutter: so, for example, if a user launches a VM with nova boot --net port_id=<PORT_UUID> and that port has binding information that constructs a PCI request, the stuff in that PCI request should be a feature set, not a vendor_id/product_id. That's what I'm getting at :)
15:17:57 jangutter feeblepipes: with iommu passthrough, the instances get a raw PCI device, with the vendor/product exposed. I've been looking at that tuple as "a feature" :-p
15:18:00 feeblepipes jangutter: and that feature set is what should be used by the scheduler and the nova-compute pci device manager to match a particular NIC among many on the host.
15:18:30 feeblepipes jangutter: what about having "iommu_passthrough" as the feature, then?
15:19:44 jangutter feeblepipes: well, iommu_passthrough won't tell us anything of the device type that's going to appear inside the VM.
15:19:46 feeblepipes jangutter: would an end-user say "I want a vNIC that is associated to a VF that has IOMMU passthrough features" or would the end user say "I want a vNIC that is associated with a VF that is a Netronome Agilio SmartNIC PF"?
15:23:10 jangutter feeblepipes: my view of one usecase is this: an end-user says "I want a vNIC, associated with a VF, passed through via IOMMU, with <insert-hw-driver-here> PCI driver, additionally pin everything for me optimally, pretty please!"
15:23:29 feeblepipes :)
15:23:42 feeblepipes jangutter: ok, cool. we're getting somewhere here :)
15:23:56 jangutter feeblepipes: with DPDK applications being specifically compiled for specific PCI drivers, it's sometimes a _must_.
15:24:11 feeblepipes jangutter: the <insert-hw-driver-here> goes in the *image metadata*, though.
15:25:04 feeblepipes jangutter: because the image metadata is what describes the workload to be launched. that image metadata can describe the HW driver's installed on the image, and that information can/should be used by os-vif in setting up the VIF plug on the host.
15:25:07 jangutter feeblepipes: yeah, but that leads to the other bit of fun. Give me one vNIC on control, standard TAP, and vNIC 2 for my DPDK NFV application.
15:25:14 feeblepipes jangutter: so I think we're *almost* there.
15:25:46 jangutter feeblepipes: is it possible to spec stuff that fine-grained?
15:26:02 feeblepipes jangutter: ah, indeed. so perhaps that information (hw driver) does need to be in the port binding profile.
15:26:18 feeblepipes jangutter: yes, it is possible. via the port profile information.
15:26:34 feeblepipes jangutter: I still don't think we need to have vendor_id/product_id in there, though. :)
15:26:54 feeblepipes jangutter: in any case... I think this is also something we can hash out in the code rather than the spec.
15:27:33 jangutter feeblepipes: I am definitely not averse to having a "driver list". That's actually a pretty good idea from a user's perspective.
15:27:34 feeblepipes jangutter: I'm willing to +2 the spec on condition that you just place a note in the spec that we've discussed on IRC how to make the information passed between Neutron and Nova via os-vif as generic as possible.
15:27:50 feeblepipes jangutter: if you can do that quick, I'll +2 it and mriedem can as well.
15:30:53 mriedem sfinucan: how do i reference another link within the same docs using rst again?
15:31:08 mriedem like i want to link to a section in the nova-manage man page from the nova-cells man page
15:31:43 jangutter I shall definitely do so: to summarize: It is undesirable to send the VF selection parameters to Nova as PCI vendor/product ID, further work should be done to develop a capability or driver list that would abstract this away from users.
15:32:01 mriedem oh maybe :ref:
15:32:11 efried mriedem Looks like that kolla thing got sorted. Can you throw me a link to whatever dashboard/view I should be using, so I can maybe pick off some lhf?
15:32:42 mriedem efried: not sorted yet, docs change coming soon
15:32:55 efried I meant triaged
15:33:05 efried That's the goal of the current sweep, right?
15:33:23 efried Not to resolve, but to get 'em out of 'New'?
15:33:44 mriedem efried: yeah
15:33:54 mriedem like 'unexpected api error' is usually always invalid
15:33:58 mriedem and a misconfiguration
15:34:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Add release note and update cell install guide for multi-cell limitations https://review.openstack.org/456923
15:34:12 mriedem melwitt: ^
15:35:40 mriedem EmilienM: owalsh: ^
15:38:26 efried This thing about having to run a new discovery command to make stuff work... Could it not just be made to work out of the box if you only have one cell?
15:47:49 efried [irony: when --help doesn't automatically page, and you have to say --help|less]
15:50:14 mriedem efried: we have a config option if you want things discovered automatically,
15:50:20 mriedem but it's a bit heavy for large deployments
15:50:27 mriedem dan has a series up to improve that
15:50:32 jangutter Is there a way to refer to today's IRC chat in a spec?
15:50:34 efried mriedem Okay, neat.
15:50:40 mriedem jangutter: yeah, esc
15:50:41 mriedem *Sec
15:50:43 efried http://eavesdrop.openstack.org/irclogs/%23openstack-nova/
15:50:47 efried jangutter ^^
15:50:48 mriedem efried: https://blueprints.launchpad.net/nova/+spec/discover-hosts-faster
15:51:03 mriedem jangutter: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-04-14.log.html#t2017-04-14T14:44:06
15:52:24 jangutter Whatever you write is now memorialized forever. Or until cats reclaim their rightful place on top of the food chain.
15:54:10 mriedem our cat can easily be defeated with a bag of chips
15:54:47 openstackgerrit Jan Gutter proposed openstack/nova-specs master: Spec to enable Netronome SmartNIC support https://review.openstack.org/456386
15:57:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Tell people that the nova-cells man page is for cells v1 https://review.openstack.org/456925
15:57:29 jangutter Another newbie question. I've replied to a bunch of comments on an older review. They're in "draft" in gerrit now, am I too late?
15:57:41 mriedem jangutter: no, you can still publish those
15:57:43 efried jangutter No - go to that patch set and they'll show up.
15:58:15 efried Still not entirely understanding why a compute node can't automatically 'discover' itself into a cell if there's only one cell...
15:58:31 mriedem efried: it would require an upcall from the compute to the api layer, which we don't want
15:58:35 mriedem upcalls are the devil
15:58:43 mriedem s/the/a/
15:58:53 openstackgerrit Ildiko Vancsa proposed openstack/nova master: Add Cinder v3 detach call to _terminate_volume_connections https://review.openstack.org/456896
15:59:12 mriedem efried: also, that would require either an upcall or putting the nova api db credentials in the compute config
15:59:16 mriedem another thing we don't want
16:00:20 efried mriedem And all of that is different from how a compute node registered itself B.C. (before cells)?
16:00:52 mriedem yes
16:00:58 mriedem in the before times,
16:01:01 mriedem there was no api db
16:01:08 mriedem there were no host mappings in the api db
16:01:36 mriedem when you start nova-compute, it creates it's own compute_nodes table record for that host (happens in ResourceTracker.update_available_resource eventually)
16:01:57 mriedem when you start the service it also creates a record in the services table for the nova-compute binary
16:02:05 mriedem those all get mangled up in the api somehow
16:02:08 mriedem feeblepipes loves it
16:02:32 mriedem but with no cells, i.e. regular old stock nova, there was just a single db and everything was in there

Earlier   Later