Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-01
16:08:24 stephenfin sean-k-mooney: I love release notes and think we should have them for all (user-facing) bugs. However, there's a user visible change being introduced here so I'd definitely say add one, yeah
16:08:28 sean-k-mooney stephenfin: if the pf is directly assignied to a guest and then readdd to the host its may or may not have a netdev associated with it and it may our may not have the same name
16:08:51 stephenfin So
16:09:36 sean-k-mooney so the sriov agent already depend on the pci address in the port profile so why not use that instad of a dev name
16:10:08 giblet sean-k-mooney: the sriov agent is configured with device names in neutron
16:10:11 sean-k-mooney i have not read the full spec
16:10:21 lyarwood mdbooth_: https://review.openstack.org/#/q/topic:bug/1803961+(status:open+OR+status:merged) - would you mind clearing your reviews out of that topic and hitting mine again early next week?
16:10:24 sean-k-mooney giblet: yes and it looks up the pci address
16:10:35 stephenfin Could we add code that says if 'pf_interface_name': 'eth0' is set, that same device can't be included whitelisted itself?
16:10:47 sean-k-mooney giblet: nova passes only the pci address to neutron in the port profile for the vf
16:11:03 giblet sean-k-mooney: yes, that is true
16:12:04 giblet sean-k-mooney: I need the neutron devs of this feature to argue why the device name was choosen for the RP nameing
16:12:37 giblet stephenfin: you want some kind of sanity check on the configured value to see if there are contradictions?
16:13:00 sean-k-mooney probaly becasue fo the connetx-3 nic form mellononx which has two pfs on teh same pciaddress
16:13:20 stephenfin giblet: I'm not sure yet. I'm trying to decide if it's a good idea or not :)
16:13:40 sean-k-mooney but that didnt work with openstack until verry verry recently and has been supperseeded
16:13:52 sean-k-mooney maybe it was connectx-2
16:13:52 mdbooth lyarwood: phase 1 complete
16:14:41 sean-k-mooney giblet: there may also be confusion between teh existing "devname" field and teh new "pf_interface_name"
16:14:41 giblet stephenfin: if the check only depends on the actula configured value and does not need to call out to the virt driver or other 3rd parties then I can be convinced to add a check
16:14:52 lyarwood mdbooth: ack thanks
16:15:11 giblet sean-k-mooney: jeah the existing devname is a filtering field while the new pf_interface_name is a tag
16:15:11 cfriesen can anyone figure out why this is failing in zuul? http://logs.openstack.org/63/631363/14/check/openstack-tox-py27/f6aa9a3/testr_results.html.gz It works fine for me locally.
16:15:41 sean-k-mooney giblet: ya didnt we say we were not going to add more tag to the whitelist like 3 or 4 cycles ago
16:15:46 giblet sean-k-mooney: if there is devname provided in the filter nova could copy that as a pf_inteface_name tag automatically
16:16:11 giblet sean-k-mooney: I did not know about that rule. what should I use instead to drive the pci claim?
16:16:15 stephenfin giblet: The devname would only match a single PF or VF though
16:16:16 sean-k-mooney giblet: well we can also lookup the pf name via sysfs
16:16:45 giblet sean-k-mooney: if there is more devname for a PF then we don't know which one to use
16:16:47 stephenfin sean-k-mooney: Yeah, but as giblet pointed out, there can be a mismatch for those ConnectX-3 NICs
16:17:19 sean-k-mooney yes that is true but do we want to cater for that edgecase
16:17:21 stephenfin bauzas: Before you knock off for the weekend, fancy hitting https://review.openstack.org/#/c/626932/ up?
16:17:32 stephenfin sean-k-mooney: Aye, that was going to be my question
16:17:40 bauzas stephenfin: looking
16:17:43 sean-k-mooney we could but it feels like we are workign around a silicon/driver bug
16:17:49 bauzas ah this one
16:17:51 stephenfin yuuup.
16:17:51 bauzas +d
16:18:36 sean-k-mooney giblet: stephenfin by the way only the PF had a duplicate pci adress
16:18:50 giblet sean-k-mooney: if nova looks up the name form sysfs and then selects one then the neutron admin needs to use the same one in the config, but we cannot really tell the admin which one nova selected
16:19:01 sean-k-mooney the vf could still be assocated with the pf netdev via sysfs
16:19:37 sean-k-mooney giblet: the admin already has to set up the config correctly on both the nova and neturon side
16:20:13 stephenfin Let's double check this
16:20:32 giblet sean-k-mooney: no not really as so far if the neutron admin selected any of the devname it worked as the ip tool accepted any of the devname. But now the name became a key between the nova and neutron communication to decide which PF the VF is allocated from
16:20:34 sean-k-mooney if we add the pf_interface_name all pci whitelist will have to be updated to make the bandwith based schduling work
16:21:13 sean-k-mooney giblet: but its not a key we send today and modifying the whitelist iis an upgrade impact
16:21:25 giblet sean-k-mooney: yes, as well as every neutron config needs to be updated to enable bandwidth
16:21:33 sean-k-mooney what we use as a key today is the pci address
16:22:08 stephenfin sean-k-mooney: I doubt there's much that can be done about that at this point though
16:22:18 sean-k-mooney what you are proposing would work but it feels like a hack that we should be able to avoid without modifing the config
16:22:24 giblet sean-k-mooney: if the admin does not add the bandwith inventory to the neutron agent config then she dont need to modify the whitelist either
16:22:30 openstackgerrit Chris Friesen proposed openstack/nova master: Add emulated TPM support to Nova https://review.openstack.org/631363
16:22:30 openstackgerrit Chris Friesen proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.openstack.org/639934
16:22:52 sean-k-mooney giblet: yes but this can be done without modifying the whitelist at all
16:23:09 stephenfin sean-k-mooney: Let's just make sure it can though
16:23:24 stephenfin We can't access sysfs outside of a driver, right?
16:23:28 giblet sean-k-mooney: yes, if neutron changes the naming to the RPs to use pci address for example
16:24:14 sean-k-mooney stephenfin: the driver could add the pf_interface_tag itself if we really wanted it too automatically
16:24:16 giblet sean-k-mooney: and we change pci claim to allow selecting PF for a PCIrequest by PF pci address
16:24:57 sean-k-mooney so neutron should not force nova to select a pf
16:25:14 sean-k-mooney it does not know the numa affintity so it cant make the right choice
16:25:16 giblet sean-k-mooney: it is nova and placement that forces the pf selection
16:25:50 giblet sean-k-mooney: port has resource request that is fulfilled by a placemetn allocation candidate that will become an allocation
16:25:54 giblet sean-k-mooney: and that forcing the PF
16:26:05 giblet sean-k-mooney: because the bandwidth is allocated from an RP that representes a PF
16:26:06 sean-k-mooney the port cant force the PF
16:26:34 sean-k-mooney it can ask for bandwith but its nova/placemtn aht has to match the port and bandwith up
16:26:54 giblet sean-k-mooney: if the bandwidth is represented in placement then nova, by selectin an a_c, actually forces a PF
16:27:23 sean-k-mooney yes it will...
16:28:26 sean-k-mooney sice we have a bug in our pci affinity policies it wont break existing workloads
16:28:47 giblet so when the allocation is done in placement nova needs to drive the pci claim mechanism to select VF from the same PF as the bandwidth is allocated from
16:29:30 sean-k-mooney yes i know i raised that point a few tiems when i argured we could not remove the reosuce tracker
16:29:51 giblet one way to drive it is to know which RP was used for the bandwidth, get the name of it, and from the name learn the pf device name
16:30:20 sean-k-mooney giblet: so the request in the neutron port does not contain a devename right
16:30:29 giblet sean-k-mooney: right.
16:30:49 sean-k-mooney ok so its placemetn that choses the bandwtih resouce provider and we are just corralting it
16:31:17 giblet sean-k-mooney: yes
16:31:56 giblet sean-k-mooney: we have to make sure the pci claim chooses the same PF as placement
16:32:02 sean-k-mooney i would still prefer if we looked up the pf name automatically in the driver and populated the tag rather then making it a manual step in the whitelist
16:32:05 stephenfin sean-k-mooney: Out of curiosity, doesn't systemd solve the "attach to guest, detach, and still have the same name" issue? https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
16:32:31 stephenfin sean-k-mooney, giblet: I must say, so would I
16:32:52 sean-k-mooney no not in all cases
16:33:02 giblet sean-k-mooney: for that nova needs to select the same device name as the neutron admin selected in the neutron config
16:33:13 stephenfin Given that it's information that's very easy to extract, and the only reason I see not to do that is because of a weird bug in an effectively EOL card
16:33:22 sean-k-mooney if the pf is not rebound to the nic driver after its bound to vfio-pci it will not have a netdev anymore
16:34:18 sean-k-mooney giblet: yes that is not that hard to do
16:34:50 giblet sean-k-mooney, stephenfin: is it doable in 4 effective day before FF?
16:35:04 sean-k-mooney we have similar code to do that in os-vif already https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L313-L334
16:36:42 cfriesen could I get some eyes on https://review.openstack.org/#/c/620706/ ? It's on a runway.
16:36:59 sean-k-mooney giblet: we need to look up netdev names form pci address to make hardware offload ovs work
16:38:22 sean-k-mooney giblet: you jsut do open( "/sys/bus/pci/devices/%s/physfn/net" % (pci_addr)).readline().strip()
16:39:10 giblet sean-k-mooney: is it fine to do that during the startup of the nova-compute service while parsing the passthrough_whitelist?
16:39:51 sean-k-mooney i dont know. we do it every time we plug a hardwar offloaed ovs-port and we plug all ports on startup
16:40:58 sean-k-mooney granted its os-vif that executing it but os-vif is run as part of the nova comptue agent so it shoudl be fine
16:41:47 stephenfin giblet: I'm guessing we could just slot it in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5926-L5940
16:44:00 giblet stephenfin, sean-k-mooney: thanks for the pointers I have to sit down with a clear mind and play with that codepath to see how can I inject the extra info there in a way that it will be used by the pci claim
16:44:33 sean-k-mooney giblet we also have a call to lookup device by name just after that
16:44:34 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5958
16:44:41 sean-k-mooney so doing the invers should be fine
16:44:42 openstackgerrit Ben Nemec proposed openstack/nova master: Add oslo.privsep to config-generator list https://review.openstack.org/640404

Earlier   Later