| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 16:23:41 | openstack | Launchpad bug 1795920 in OpenStack Compute (nova) "SR-IOV shared PCI numa not working " [Undecided,New] | |
| 16:23:41 | spatel_ | sean-k-mooney: currently i am dealing with this issue :( https://bugs.launchpad.net/nova/+bug/1795920 | |
| 16:23:58 | spatel_ | Do you know what wrong i am doing here | |
| 16:24:14 | spatel_ | I have 2 NUMA node and running SR-IOV with shared PCI | |
| 16:24:18 | sean-k-mooney | https://docs.openstack.org/mitaka/networking-guide/config-sriov.html has the doc | |
| 16:24:23 | sean-k-mooney | am let me look | |
| 16:24:33 | spatel_ | I am only able to use one side of NUMA | |
| 16:25:27 | sean-k-mooney | by default unless you set a pci numa affinity policy in the flavor or image we require strict numa afinity | |
| 16:25:32 | spatel_ | Its not allowing me to launch SR-IOV instance on NUMA-2 ( because PCI is attach to NUMA-1 ) | |
| 16:25:52 | spatel_ | All i did is hw:pci_numa_affinity_policy=preferred in flavor | |
| 16:25:57 | spatel_ | what else i need to do? | |
| 16:26:39 | sean-k-mooney | spatel_: let me check. i taught that was enough but you might also need to set the policy in the whitelist | |
| 16:26:40 | spatel_ | I do have aggregate_instance_extra_specs:pinned='true', hw:cpu_policy='dedicated' in flavor | |
| 16:27:10 | spatel_ | I think document isn't clear in blueprint so i am totally confused :( | |
| 16:28:06 | spatel_ | If i remove "aggregate_instance_extra_specs:pinned='true', hw:cpu_policy='dedicated'" from flavor then i am able to launch instance anywhere in NUMA with SR-IOV support | |
| 16:29:03 | sean-k-mooney | aggregate_instance_extra_specs:pinned='true' is not a standard thing | |
| 16:30:03 | spatel_ | hmmm! i didn't get it | |
| 16:30:20 | sean-k-mooney | spatel_: you should not need to set anything in the aggragte to use the pci policies | |
| 16:30:29 | openstackgerrit | Jay Pipes proposed openstack/nova stable/ocata: Re-use existing ComputeNode on ironic rebalance https://review.openstack.org/607626 | |
| 16:30:46 | spatel_ | oh! so you are saying i should remove aggregate_instance_extra_specs:pinned | |
| 16:30:55 | jaypipes | mriedem: k, hopefully correct now. | |
| 16:31:02 | jaypipes | thx for the help again. | |
| 16:31:20 | sean-k-mooney | spatel_: yes | |
| 16:31:31 | spatel_ | lets say if i remove "aggragte" then does my vCPU get Pinned ? | |
| 16:33:57 | spatel_ | removing aggrate and going to launch instance | |
| 16:36:25 | spatel_ | sean-k-mooney: didn't work error 'No valid host was found. There are not enough hosts available' | |
| 16:36:49 | spatel_ | look like something is missing.. | |
| 16:38:38 | sean-k-mooney | spatel_: so jsut to confirm you dont have any aggragte metadata set and have hw:pci_numa_affinity_policy=preferred set | |
| 16:38:48 | sean-k-mooney | in the flavor | |
| 16:39:15 | spatel_ | This is what i have currently in flavor -> properties | hw:cpu_policy='dedicated', hw:numa_nodes='2', hw:pci_numa_affinity_policy='preferred' | |
| 16:39:57 | stephenfin | spatel_, sean-k-mooney: We didn't implement it with a flavor extra spec in the end | |
| 16:39:57 | spatel_ | If i remove all 3 option then i am successfully able to launch instance | |
| 16:40:13 | sean-k-mooney | so this has be set in the pci whitelist then | |
| 16:40:17 | stephenfin | spatel_: Yes | |
| 16:40:23 | stephenfin | Oops, sean-k-mooney ^ | |
| 16:40:45 | spatel_ | oh!! wait wait.. so what i need to do in pci whitelist ? | |
| 16:40:49 | sean-k-mooney | spatel_: https://github.com/openstack/nova/blob/master/nova/pci/request.py#L16-L25 | |
| 16:41:13 | sean-k-mooney | sorry thats not what you want but yes you do | |
| 16:41:28 | spatel_ | so i need to add that snippet in compute nova.conf in [PCI] section ? | |
| 16:41:29 | stephenfin | spatel_: Have you seen this? https://docs.openstack.org/nova/latest/admin/networking.html#numa-affinity | |
| 16:41:41 | stephenfin | spatel_: Ignore that - wrong feature :) | |
| 16:41:47 | spatel_ | ok | |
| 16:41:52 | spatel_ | I am running queens | |
| 16:42:09 | stephenfin | spatel_: https://docs.openstack.org/nova/latest/configuration/config.html#pci | |
| 16:42:30 | stephenfin | See the alias configuration key | |
| 16:42:42 | stephenfin | spatel_: But, to be clear, is this for a PCI device or an SR-IOV device? | |
| 16:42:52 | spatel_ | SR-IOV device | |
| 16:43:20 | spatel_ | We are running high performance network application and need high speed network or high PPS rate | |
| 16:43:29 | sean-k-mooney | stephenfin: looking at the whitelist code i dont think we supprot it in the whitelist | |
| 16:44:12 | stephenfin | sean-k-mooney: Doesn't seem like it. I'm trying to think why it was done that way | |
| 16:44:28 | sean-k-mooney | which would mean the polices only work for device requeted via flavor alias which would be dumb | |
| 16:44:47 | sean-k-mooney | are you sure we did not supprot this in the flavor extraspecs /image metadata | |
| 16:44:52 | spatel_ | I am going to add alias and get back to you.. | |
| 16:44:53 | stephenfin | definitely not | |
| 16:45:19 | sean-k-mooney | stephenfin: was the whole point of this feature to fix neutron sriov | |
| 16:45:53 | spatel_ | is product_id and vendore_id mandatory because in i am using devname here "pci_passthrough_whitelist = "{ "physical_network":"vlan", "devname":"eno2" }"" | |
| 16:46:23 | sean-k-mooney | spatel_: no | |
| 16:47:58 | sean-k-mooney | a white list can be in any of these forms https://github.com/openstack/nova/blob/master/nova/pci/devspec.py#L182-L192 | |
| 16:48:20 | sean-k-mooney | actully the alias yes that need to use vendor_id and product_id | |
| 16:48:34 | stephenfin | sean-k-mooney: If it was, it seems something may have slipped through the cracks here | |
| 16:48:48 | sean-k-mooney | alias are not for networking they are for passthrough devices | |
| 16:48:56 | stephenfin | Yup, I get that | |
| 16:49:12 | spatel_ | type-PCI, type-PF and type-VF what i should pick ? | |
| 16:49:17 | spatel_ | VF ? | |
| 16:49:19 | stephenfin | From the quick glance here, it should really be configured via the whitelist. I'm not sure why I went with the alias | |
| 16:49:22 | stephenfin | spatel_: yup | |
| 16:49:27 | spatel_ | doing it.. | |
| 16:49:33 | sean-k-mooney | stephenfin: ya i am go to confirm https://bugs.launchpad.net/nova/+bug/1795920 | |
| 16:49:34 | openstack | Launchpad bug 1795920 in OpenStack Compute (nova) "SR-IOV shared PCI numa not working " [Undecided,Confirmed] | |
| 16:50:05 | sean-k-mooney | stephenfin: interested in working on this? if not ill add it to my list but this need to be fixed | |
| 16:50:15 | sean-k-mooney | and backported | |
| 16:50:21 | stephenfin | I won't tackle it tonight but I can do so, yeah | |
| 16:50:36 | stephenfin | not sure if we can backport though. It'll be a config file change | |
| 16:50:39 | sean-k-mooney | cool we liekly need to repreose the old spec | |
| 16:51:10 | sean-k-mooney | stephenfin: i was suggesting we need to add the flavor and image extraspecs so no config file change | |
| 16:51:47 | stephenfin | sean-k-mooney: Possibly, but before doing so I'd suggest going back and reading the spec reviews | |
| 16:51:59 | stephenfin | There was a reason we didn't do that, though I don't recall it now :/ | |
| 16:52:18 | spatel | sean-k-mooney: this is what i change in nova.conf http://paste.openstack.org/show/731417/ | |
| 16:52:22 | stephenfin | If it's image metadata changes, we can't backport those due to object changes | |
| 16:52:24 | spatel | can you verify | |
| 16:52:33 | sean-k-mooney | yes i rembere i was very against using the alisa but i never recalled dropping the extra specs | |
| 16:53:12 | spatel | going to launch instance now, figure cross | |
| 16:53:32 | sean-k-mooney | spatel: its goning to fail. | |
| 16:53:39 | spatel | ?? | |
| 16:53:43 | spatel | why? | |
| 16:53:48 | sean-k-mooney | looking at the code the feature was not finished | |
| 16:54:01 | spatel | Damn it :( | |
| 16:54:12 | spatel | so what is the deal here ? | |
| 16:54:25 | sean-k-mooney | the numa policies are only repected for flavor based pci device passhtoruh e.g. for things like gpu or acllorator cards | |
| 16:54:44 | sean-k-mooney | spatel: the spec was approve and the full feature was not merged | |
| 16:55:09 | spatel | in my case its VF | |
| 16:55:34 | spatel | currently i can't utilize both numa :( | |
| 16:55:47 | sean-k-mooney | spatel: in your case its a neutron port with vnic_type direct correct | |
| 16:55:52 | spatel | yes | |
| 16:56:04 | sean-k-mooney | spatel: the workaround is to make the guest have 2 numa nodes | |
| 16:56:11 | melwitt | sean-k-mooney: which spec was not properly finished? | |
| 16:56:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/607650 | |
| 16:56:34 | mriedem | spatel: fyi ^ | |
| 16:56:47 | sean-k-mooney | melwitt: the numa pci polices | |
| 16:57:00 | sean-k-mooney | ill get the link on sec | |