Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-01
16:17:43 sean-k-mooney we could but it feels like we are workign around a silicon/driver bug
16:17:49 bauzas ah this one
16:17:51 bauzas +d
16:17:51 stephenfin yuuup.
16:18:36 sean-k-mooney giblet: stephenfin by the way only the PF had a duplicate pci adress
16:18:50 giblet sean-k-mooney: if nova looks up the name form sysfs and then selects one then the neutron admin needs to use the same one in the config, but we cannot really tell the admin which one nova selected
16:19:01 sean-k-mooney the vf could still be assocated with the pf netdev via sysfs
16:19:37 sean-k-mooney giblet: the admin already has to set up the config correctly on both the nova and neturon side
16:20:13 stephenfin Let's double check this
16:20:32 giblet sean-k-mooney: no not really as so far if the neutron admin selected any of the devname it worked as the ip tool accepted any of the devname. But now the name became a key between the nova and neutron communication to decide which PF the VF is allocated from
16:20:34 sean-k-mooney if we add the pf_interface_name all pci whitelist will have to be updated to make the bandwith based schduling work
16:21:13 sean-k-mooney giblet: but its not a key we send today and modifying the whitelist iis an upgrade impact
16:21:25 giblet sean-k-mooney: yes, as well as every neutron config needs to be updated to enable bandwidth
16:21:33 sean-k-mooney what we use as a key today is the pci address
16:22:08 stephenfin sean-k-mooney: I doubt there's much that can be done about that at this point though
16:22:18 sean-k-mooney what you are proposing would work but it feels like a hack that we should be able to avoid without modifing the config
16:22:24 giblet sean-k-mooney: if the admin does not add the bandwith inventory to the neutron agent config then she dont need to modify the whitelist either
16:22:30 openstackgerrit Chris Friesen proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.openstack.org/639934
16:22:30 openstackgerrit Chris Friesen proposed openstack/nova master: Add emulated TPM support to Nova https://review.openstack.org/631363
16:22:52 sean-k-mooney giblet: yes but this can be done without modifying the whitelist at all
16:23:09 stephenfin sean-k-mooney: Let's just make sure it can though
16:23:24 stephenfin We can't access sysfs outside of a driver, right?
16:23:28 giblet sean-k-mooney: yes, if neutron changes the naming to the RPs to use pci address for example
16:24:14 sean-k-mooney stephenfin: the driver could add the pf_interface_tag itself if we really wanted it too automatically
16:24:16 giblet sean-k-mooney: and we change pci claim to allow selecting PF for a PCIrequest by PF pci address
16:24:57 sean-k-mooney so neutron should not force nova to select a pf
16:25:14 sean-k-mooney it does not know the numa affintity so it cant make the right choice
16:25:16 giblet sean-k-mooney: it is nova and placement that forces the pf selection
16:25:50 giblet sean-k-mooney: port has resource request that is fulfilled by a placemetn allocation candidate that will become an allocation
16:25:54 giblet sean-k-mooney: and that forcing the PF
16:26:05 giblet sean-k-mooney: because the bandwidth is allocated from an RP that representes a PF
16:26:06 sean-k-mooney the port cant force the PF
16:26:34 sean-k-mooney it can ask for bandwith but its nova/placemtn aht has to match the port and bandwith up
16:26:54 giblet sean-k-mooney: if the bandwidth is represented in placement then nova, by selectin an a_c, actually forces a PF
16:27:23 sean-k-mooney yes it will...
16:28:26 sean-k-mooney sice we have a bug in our pci affinity policies it wont break existing workloads
16:28:47 giblet so when the allocation is done in placement nova needs to drive the pci claim mechanism to select VF from the same PF as the bandwidth is allocated from
16:29:30 sean-k-mooney yes i know i raised that point a few tiems when i argured we could not remove the reosuce tracker
16:29:51 giblet one way to drive it is to know which RP was used for the bandwidth, get the name of it, and from the name learn the pf device name
16:30:20 sean-k-mooney giblet: so the request in the neutron port does not contain a devename right
16:30:29 giblet sean-k-mooney: right.
16:30:49 sean-k-mooney ok so its placemetn that choses the bandwtih resouce provider and we are just corralting it
16:31:17 giblet sean-k-mooney: yes
16:31:56 giblet sean-k-mooney: we have to make sure the pci claim chooses the same PF as placement
16:32:02 sean-k-mooney i would still prefer if we looked up the pf name automatically in the driver and populated the tag rather then making it a manual step in the whitelist
16:32:05 stephenfin sean-k-mooney: Out of curiosity, doesn't systemd solve the "attach to guest, detach, and still have the same name" issue? https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
16:32:31 stephenfin sean-k-mooney, giblet: I must say, so would I
16:32:52 sean-k-mooney no not in all cases
16:33:02 giblet sean-k-mooney: for that nova needs to select the same device name as the neutron admin selected in the neutron config
16:33:13 stephenfin Given that it's information that's very easy to extract, and the only reason I see not to do that is because of a weird bug in an effectively EOL card
16:33:22 sean-k-mooney if the pf is not rebound to the nic driver after its bound to vfio-pci it will not have a netdev anymore
16:34:18 sean-k-mooney giblet: yes that is not that hard to do
16:34:50 giblet sean-k-mooney, stephenfin: is it doable in 4 effective day before FF?
16:35:04 sean-k-mooney we have similar code to do that in os-vif already https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L313-L334
16:36:42 cfriesen could I get some eyes on https://review.openstack.org/#/c/620706/ ? It's on a runway.
16:36:59 sean-k-mooney giblet: we need to look up netdev names form pci address to make hardware offload ovs work
16:38:22 sean-k-mooney giblet: you jsut do open( "/sys/bus/pci/devices/%s/physfn/net" % (pci_addr)).readline().strip()
16:39:10 giblet sean-k-mooney: is it fine to do that during the startup of the nova-compute service while parsing the passthrough_whitelist?
16:39:51 sean-k-mooney i dont know. we do it every time we plug a hardwar offloaed ovs-port and we plug all ports on startup
16:40:58 sean-k-mooney granted its os-vif that executing it but os-vif is run as part of the nova comptue agent so it shoudl be fine
16:41:47 stephenfin giblet: I'm guessing we could just slot it in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5926-L5940
16:44:00 giblet stephenfin, sean-k-mooney: thanks for the pointers I have to sit down with a clear mind and play with that codepath to see how can I inject the extra info there in a way that it will be used by the pci claim
16:44:33 sean-k-mooney giblet we also have a call to lookup device by name just after that
16:44:34 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5958
16:44:41 sean-k-mooney so doing the invers should be fine
16:44:42 openstackgerrit Ben Nemec proposed openstack/nova master: Add oslo.privsep to config-generator list https://review.openstack.org/640404
16:45:51 sean-k-mooney giblet: im not really objecting to useing a tag by the way
16:46:04 sean-k-mooney giblet: i think we can just automate it
16:46:37 stephenfin giblet: Sounds good. If you can't make progress fairly quickly, I'd be willing to hold my nose so we could get it in. Would be good to try avoid it though, given the few issues there are already
16:46:41 giblet sean-k-mooney: I think I understand. I agree in general to automate it if possible.
16:47:08 giblet sean-k-mooney: I'm more affraid that I will not be able to make it work
16:47:45 giblet stephenfin: alternatively I can remove this patch from the series, declare the two PF on the same compute on the same physnet as not-supported in Stein
16:47:49 stephenfin e.g. confusion with 'devname', issues if you whitelist more than PF and need for extra (possibly unnecessary) configuration, the first two of which probably need additional code to fix
16:48:33 stephenfin giblet: Aye, I was wondering if it was possible to shove this to the end and de-risk it. If that's an option, it might be a good one
16:48:43 sean-k-mooney giblet: i think that is what we did for vgpus
16:49:18 sean-k-mooney its not ideal but it would work. i would be good to ask leakypipes or dansmith
16:49:32 sean-k-mooney they tend to have stong opipions on this topic
16:50:07 giblet stephenfin: yeah, I think the sortness of time will push me to remove this patch from the critical path
16:50:10 sean-k-mooney actully maybe you shouldnt ask them :)
16:51:45 sean-k-mooney giblet: i can try and hack something up at the weekend to do auto tagging and then you can decide on monday
16:52:46 giblet sean-k-mooney: if you hack someting up then I will definitly look at it on Monday but please don't spend your free time on this crazy things :)
16:54:03 sean-k-mooney i spent my free time last weekend trying to autoamte a tox nutika enviorment ot automaticlaly compile nova and then execute the unit test.
16:54:11 sean-k-mooney this might be a less crazy
16:54:20 giblet sean-k-mooney: :)
16:54:34 sean-k-mooney i got it to work for os-vif but nova make it explode
16:55:44 leakypipes sean-k-mooney, stephenfin: ask me what?
16:56:45 stephenfin leakypipes: We're not overly happy about this patch and think pf_interface_name could be figured out dynamically by the driver https://review.openstack.org/#/c/625311/29
16:57:21 sean-k-mooney noting you will like. unless your opipion on adding tags to the pci whitelist has changed recently
16:57:31 stephenfin leakypipes: giblet is concerned about time constraints though and think maybe declaring that two PF on the same compute on the same physnet as not-supported in Stein would be a safer option
16:58:05 stephenfin *maybe declaring two PFs
16:58:30 leakypipes stephenfin: ok, that patch is near the top of my queue. will try to comment on it this afternoon.
16:59:11 giblet I'm not too affraid of saying that this edge case is not supported in Stein
16:59:12 sean-k-mooney we are already doing auto discovery of nic offloads exctra on startup and we have code to discover the name of pf form vfs in os-vif so i was suggesing just looking up the pf name on startup as part of the pci device discovery
17:01:42 giblet however I will only be able to declare this case unsupported in the documentation as I don't see an easy way to detect the case from code and reject or just fail the boot request
17:02:26 giblet so if the admin ignores the doc then she can create a situation when the bandwidth is allocated from a PF while the VF is allocated from other PF for the same port
17:04:13 stephenfin giblet: Is there a file missing from https://review.openstack.org/#/c/623543/ ?
17:04:48 stephenfin I'm trying to figure out where "Then the pci claim will enforce that the PF interface name in the request matches the interface name of the PF from where the VF is selected from." happens
17:04:56 stephenfin (from the commit message, line 53)
17:05:21 giblet stephenfin: I think the pci claim matches the request with the device spec field by field
17:05:25 stephenfin Oh, wait, tags

Earlier   Later