Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-01
16:13:20 stephenfin giblet: I'm not sure yet. I'm trying to decide if it's a good idea or not :)
16:13:40 sean-k-mooney but that didnt work with openstack until verry verry recently and has been supperseeded
16:13:52 mdbooth lyarwood: phase 1 complete
16:13:52 sean-k-mooney maybe it was connectx-2
16:14:41 giblet stephenfin: if the check only depends on the actula configured value and does not need to call out to the virt driver or other 3rd parties then I can be convinced to add a check
16:14:41 sean-k-mooney giblet: there may also be confusion between teh existing "devname" field and teh new "pf_interface_name"
16:14:52 lyarwood mdbooth: ack thanks
16:15:11 cfriesen can anyone figure out why this is failing in zuul? http://logs.openstack.org/63/631363/14/check/openstack-tox-py27/f6aa9a3/testr_results.html.gz It works fine for me locally.
16:15:11 giblet sean-k-mooney: jeah the existing devname is a filtering field while the new pf_interface_name is a tag
16:15:41 sean-k-mooney giblet: ya didnt we say we were not going to add more tag to the whitelist like 3 or 4 cycles ago
16:15:46 giblet sean-k-mooney: if there is devname provided in the filter nova could copy that as a pf_inteface_name tag automatically
16:16:11 giblet sean-k-mooney: I did not know about that rule. what should I use instead to drive the pci claim?
16:16:15 stephenfin giblet: The devname would only match a single PF or VF though
16:16:16 sean-k-mooney giblet: well we can also lookup the pf name via sysfs
16:16:45 giblet sean-k-mooney: if there is more devname for a PF then we don't know which one to use
16:16:47 stephenfin sean-k-mooney: Yeah, but as giblet pointed out, there can be a mismatch for those ConnectX-3 NICs
16:17:19 sean-k-mooney yes that is true but do we want to cater for that edgecase
16:17:21 stephenfin bauzas: Before you knock off for the weekend, fancy hitting https://review.openstack.org/#/c/626932/ up?
16:17:32 stephenfin sean-k-mooney: Aye, that was going to be my question
16:17:40 bauzas stephenfin: looking
16:17:43 sean-k-mooney we could but it feels like we are workign around a silicon/driver bug
16:17:49 bauzas ah this one
16:17:51 bauzas +d
16:17:51 stephenfin yuuup.
16:18:36 sean-k-mooney giblet: stephenfin by the way only the PF had a duplicate pci adress
16:18:50 giblet sean-k-mooney: if nova looks up the name form sysfs and then selects one then the neutron admin needs to use the same one in the config, but we cannot really tell the admin which one nova selected
16:19:01 sean-k-mooney the vf could still be assocated with the pf netdev via sysfs
16:19:37 sean-k-mooney giblet: the admin already has to set up the config correctly on both the nova and neturon side
16:20:13 stephenfin Let's double check this
16:20:32 giblet sean-k-mooney: no not really as so far if the neutron admin selected any of the devname it worked as the ip tool accepted any of the devname. But now the name became a key between the nova and neutron communication to decide which PF the VF is allocated from
16:20:34 sean-k-mooney if we add the pf_interface_name all pci whitelist will have to be updated to make the bandwith based schduling work
16:21:13 sean-k-mooney giblet: but its not a key we send today and modifying the whitelist iis an upgrade impact
16:21:25 giblet sean-k-mooney: yes, as well as every neutron config needs to be updated to enable bandwidth
16:21:33 sean-k-mooney what we use as a key today is the pci address
16:22:08 stephenfin sean-k-mooney: I doubt there's much that can be done about that at this point though
16:22:18 sean-k-mooney what you are proposing would work but it feels like a hack that we should be able to avoid without modifing the config
16:22:24 giblet sean-k-mooney: if the admin does not add the bandwith inventory to the neutron agent config then she dont need to modify the whitelist either
16:22:30 openstackgerrit Chris Friesen proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.openstack.org/639934
16:22:30 openstackgerrit Chris Friesen proposed openstack/nova master: Add emulated TPM support to Nova https://review.openstack.org/631363
16:22:52 sean-k-mooney giblet: yes but this can be done without modifying the whitelist at all
16:23:09 stephenfin sean-k-mooney: Let's just make sure it can though
16:23:24 stephenfin We can't access sysfs outside of a driver, right?
16:23:28 giblet sean-k-mooney: yes, if neutron changes the naming to the RPs to use pci address for example
16:24:14 sean-k-mooney stephenfin: the driver could add the pf_interface_tag itself if we really wanted it too automatically
16:24:16 giblet sean-k-mooney: and we change pci claim to allow selecting PF for a PCIrequest by PF pci address
16:24:57 sean-k-mooney so neutron should not force nova to select a pf
16:25:14 sean-k-mooney it does not know the numa affintity so it cant make the right choice
16:25:16 giblet sean-k-mooney: it is nova and placement that forces the pf selection
16:25:50 giblet sean-k-mooney: port has resource request that is fulfilled by a placemetn allocation candidate that will become an allocation
16:25:54 giblet sean-k-mooney: and that forcing the PF
16:26:05 giblet sean-k-mooney: because the bandwidth is allocated from an RP that representes a PF
16:26:06 sean-k-mooney the port cant force the PF
16:26:34 sean-k-mooney it can ask for bandwith but its nova/placemtn aht has to match the port and bandwith up
16:26:54 giblet sean-k-mooney: if the bandwidth is represented in placement then nova, by selectin an a_c, actually forces a PF
16:27:23 sean-k-mooney yes it will...
16:28:26 sean-k-mooney sice we have a bug in our pci affinity policies it wont break existing workloads
16:28:47 giblet so when the allocation is done in placement nova needs to drive the pci claim mechanism to select VF from the same PF as the bandwidth is allocated from
16:29:30 sean-k-mooney yes i know i raised that point a few tiems when i argured we could not remove the reosuce tracker
16:29:51 giblet one way to drive it is to know which RP was used for the bandwidth, get the name of it, and from the name learn the pf device name
16:30:20 sean-k-mooney giblet: so the request in the neutron port does not contain a devename right
16:30:29 giblet sean-k-mooney: right.
16:30:49 sean-k-mooney ok so its placemetn that choses the bandwtih resouce provider and we are just corralting it
16:31:17 giblet sean-k-mooney: yes
16:31:56 giblet sean-k-mooney: we have to make sure the pci claim chooses the same PF as placement
16:32:02 sean-k-mooney i would still prefer if we looked up the pf name automatically in the driver and populated the tag rather then making it a manual step in the whitelist
16:32:05 stephenfin sean-k-mooney: Out of curiosity, doesn't systemd solve the "attach to guest, detach, and still have the same name" issue? https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
16:32:31 stephenfin sean-k-mooney, giblet: I must say, so would I
16:32:52 sean-k-mooney no not in all cases
16:33:02 giblet sean-k-mooney: for that nova needs to select the same device name as the neutron admin selected in the neutron config
16:33:13 stephenfin Given that it's information that's very easy to extract, and the only reason I see not to do that is because of a weird bug in an effectively EOL card
16:33:22 sean-k-mooney if the pf is not rebound to the nic driver after its bound to vfio-pci it will not have a netdev anymore
16:34:18 sean-k-mooney giblet: yes that is not that hard to do
16:34:50 giblet sean-k-mooney, stephenfin: is it doable in 4 effective day before FF?
16:35:04 sean-k-mooney we have similar code to do that in os-vif already https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L313-L334
16:36:42 cfriesen could I get some eyes on https://review.openstack.org/#/c/620706/ ? It's on a runway.
16:36:59 sean-k-mooney giblet: we need to look up netdev names form pci address to make hardware offload ovs work
16:38:22 sean-k-mooney giblet: you jsut do open( "/sys/bus/pci/devices/%s/physfn/net" % (pci_addr)).readline().strip()
16:39:10 giblet sean-k-mooney: is it fine to do that during the startup of the nova-compute service while parsing the passthrough_whitelist?
16:39:51 sean-k-mooney i dont know. we do it every time we plug a hardwar offloaed ovs-port and we plug all ports on startup
16:40:58 sean-k-mooney granted its os-vif that executing it but os-vif is run as part of the nova comptue agent so it shoudl be fine
16:41:47 stephenfin giblet: I'm guessing we could just slot it in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5926-L5940
16:44:00 giblet stephenfin, sean-k-mooney: thanks for the pointers I have to sit down with a clear mind and play with that codepath to see how can I inject the extra info there in a way that it will be used by the pci claim
16:44:33 sean-k-mooney giblet we also have a call to lookup device by name just after that
16:44:34 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5958
16:44:41 sean-k-mooney so doing the invers should be fine
16:44:42 openstackgerrit Ben Nemec proposed openstack/nova master: Add oslo.privsep to config-generator list https://review.openstack.org/640404
16:45:51 sean-k-mooney giblet: im not really objecting to useing a tag by the way
16:46:04 sean-k-mooney giblet: i think we can just automate it
16:46:37 stephenfin giblet: Sounds good. If you can't make progress fairly quickly, I'd be willing to hold my nose so we could get it in. Would be good to try avoid it though, given the few issues there are already
16:46:41 giblet sean-k-mooney: I think I understand. I agree in general to automate it if possible.
16:47:08 giblet sean-k-mooney: I'm more affraid that I will not be able to make it work
16:47:45 giblet stephenfin: alternatively I can remove this patch from the series, declare the two PF on the same compute on the same physnet as not-supported in Stein
16:47:49 stephenfin e.g. confusion with 'devname', issues if you whitelist more than PF and need for extra (possibly unnecessary) configuration, the first two of which probably need additional code to fix
16:48:33 stephenfin giblet: Aye, I was wondering if it was possible to shove this to the end and de-risk it. If that's an option, it might be a good one
16:48:43 sean-k-mooney giblet: i think that is what we did for vgpus
16:49:18 sean-k-mooney its not ideal but it would work. i would be good to ask leakypipes or dansmith
16:49:32 sean-k-mooney they tend to have stong opipions on this topic
16:50:07 giblet stephenfin: yeah, I think the sortness of time will push me to remove this patch from the critical path
16:50:10 sean-k-mooney actully maybe you shouldnt ask them :)
16:51:45 sean-k-mooney giblet: i can try and hack something up at the weekend to do auto tagging and then you can decide on monday

Earlier   Later