| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 16:19:46 | sean-k-mooney | spatel_: i would say config issue. i would guess the default options for the gernel module cahnged and or you are using a different driver by default | |
| 16:19:47 | jaypipes | mriedem: apologies. how can I fix appropriately? do I need to re-do the git cherry-pick with -x? or can/should I just edit the commit message with seomthing? | |
| 16:20:27 | sean-k-mooney | spatel_: for example if the device was bound to vfio_pci instead fo the broadcom driver then it would existit in lspci but not have a netdev | |
| 16:20:59 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Service version check for NUMA live migration https://review.openstack.org/566723 | |
| 16:20:59 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Libvirt live migration: update NUMA XML for dest https://review.openstack.org/575179 | |
| 16:21:01 | spatel_ | hmmm! | |
| 16:21:02 | sean-k-mooney | spatel_: do you whitelist devices using the devname option? | |
| 16:21:04 | mriedem | jaypipes: see my other comment in the ocata backport about documenting the conflicts? | |
| 16:21:18 | spatel_ | sean-k-mooney: yes i am using devname option to specify my interface | |
| 16:21:29 | spatel_ | pci_passthrough_whitelist = "{ "physical_network":"vlan", "devname":"eno2" }" | |
| 16:21:36 | sean-k-mooney | spatel_: in general i advise against that for this exact reason | |
| 16:21:52 | sean-k-mooney | spatel_: if you use the pci adddress instead 4.18 would likely be fine | |
| 16:22:14 | spatel_ | pci address ? | |
| 16:22:27 | sean-k-mooney | spatel_: the whitelist can have 3 modes of whitelisting | |
| 16:22:35 | spatel_ | you mean vendor_id or product_id ? | |
| 16:23:09 | sean-k-mooney | spatel_: you can used devname, (vendor_id and product_id) or you can pass a pci address | |
| 16:23:24 | spatel_ | sean-k-mooney: i will give it a try and report back to BUG | |
| 16:23:41 | openstack | Launchpad bug 1795920 in OpenStack Compute (nova) "SR-IOV shared PCI numa not working " [Undecided,New] | |
| 16:23:41 | spatel_ | sean-k-mooney: currently i am dealing with this issue :( https://bugs.launchpad.net/nova/+bug/1795920 | |
| 16:23:58 | spatel_ | Do you know what wrong i am doing here | |
| 16:24:14 | spatel_ | I have 2 NUMA node and running SR-IOV with shared PCI | |
| 16:24:18 | sean-k-mooney | https://docs.openstack.org/mitaka/networking-guide/config-sriov.html has the doc | |
| 16:24:23 | sean-k-mooney | am let me look | |
| 16:24:33 | spatel_ | I am only able to use one side of NUMA | |
| 16:25:27 | sean-k-mooney | by default unless you set a pci numa affinity policy in the flavor or image we require strict numa afinity | |
| 16:25:32 | spatel_ | Its not allowing me to launch SR-IOV instance on NUMA-2 ( because PCI is attach to NUMA-1 ) | |
| 16:25:52 | spatel_ | All i did is hw:pci_numa_affinity_policy=preferred in flavor | |
| 16:25:57 | spatel_ | what else i need to do? | |
| 16:26:39 | sean-k-mooney | spatel_: let me check. i taught that was enough but you might also need to set the policy in the whitelist | |
| 16:26:40 | spatel_ | I do have aggregate_instance_extra_specs:pinned='true', hw:cpu_policy='dedicated' in flavor | |
| 16:27:10 | spatel_ | I think document isn't clear in blueprint so i am totally confused :( | |
| 16:28:06 | spatel_ | If i remove "aggregate_instance_extra_specs:pinned='true', hw:cpu_policy='dedicated'" from flavor then i am able to launch instance anywhere in NUMA with SR-IOV support | |
| 16:29:03 | sean-k-mooney | aggregate_instance_extra_specs:pinned='true' is not a standard thing | |
| 16:30:03 | spatel_ | hmmm! i didn't get it | |
| 16:30:20 | sean-k-mooney | spatel_: you should not need to set anything in the aggragte to use the pci policies | |
| 16:30:29 | openstackgerrit | Jay Pipes proposed openstack/nova stable/ocata: Re-use existing ComputeNode on ironic rebalance https://review.openstack.org/607626 | |
| 16:30:46 | spatel_ | oh! so you are saying i should remove aggregate_instance_extra_specs:pinned | |
| 16:30:55 | jaypipes | mriedem: k, hopefully correct now. | |
| 16:31:02 | jaypipes | thx for the help again. | |
| 16:31:20 | sean-k-mooney | spatel_: yes | |
| 16:31:31 | spatel_ | lets say if i remove "aggragte" then does my vCPU get Pinned ? | |
| 16:33:57 | spatel_ | removing aggrate and going to launch instance | |
| 16:36:25 | spatel_ | sean-k-mooney: didn't work error 'No valid host was found. There are not enough hosts available' | |
| 16:36:49 | spatel_ | look like something is missing.. | |
| 16:38:38 | sean-k-mooney | spatel_: so jsut to confirm you dont have any aggragte metadata set and have hw:pci_numa_affinity_policy=preferred set | |
| 16:38:48 | sean-k-mooney | in the flavor | |
| 16:39:15 | spatel_ | This is what i have currently in flavor -> properties | hw:cpu_policy='dedicated', hw:numa_nodes='2', hw:pci_numa_affinity_policy='preferred' | |
| 16:39:57 | stephenfin | spatel_, sean-k-mooney: We didn't implement it with a flavor extra spec in the end | |
| 16:39:57 | spatel_ | If i remove all 3 option then i am successfully able to launch instance | |
| 16:40:13 | sean-k-mooney | so this has be set in the pci whitelist then | |
| 16:40:17 | stephenfin | spatel_: Yes | |
| 16:40:23 | stephenfin | Oops, sean-k-mooney ^ | |
| 16:40:45 | spatel_ | oh!! wait wait.. so what i need to do in pci whitelist ? | |
| 16:40:49 | sean-k-mooney | spatel_: https://github.com/openstack/nova/blob/master/nova/pci/request.py#L16-L25 | |
| 16:41:13 | sean-k-mooney | sorry thats not what you want but yes you do | |
| 16:41:28 | spatel_ | so i need to add that snippet in compute nova.conf in [PCI] section ? | |
| 16:41:29 | stephenfin | spatel_: Have you seen this? https://docs.openstack.org/nova/latest/admin/networking.html#numa-affinity | |
| 16:41:41 | stephenfin | spatel_: Ignore that - wrong feature :) | |
| 16:41:47 | spatel_ | ok | |
| 16:41:52 | spatel_ | I am running queens | |
| 16:42:09 | stephenfin | spatel_: https://docs.openstack.org/nova/latest/configuration/config.html#pci | |
| 16:42:30 | stephenfin | See the alias configuration key | |
| 16:42:42 | stephenfin | spatel_: But, to be clear, is this for a PCI device or an SR-IOV device? | |
| 16:42:52 | spatel_ | SR-IOV device | |
| 16:43:20 | spatel_ | We are running high performance network application and need high speed network or high PPS rate | |
| 16:43:29 | sean-k-mooney | stephenfin: looking at the whitelist code i dont think we supprot it in the whitelist | |
| 16:44:12 | stephenfin | sean-k-mooney: Doesn't seem like it. I'm trying to think why it was done that way | |
| 16:44:28 | sean-k-mooney | which would mean the polices only work for device requeted via flavor alias which would be dumb | |
| 16:44:47 | sean-k-mooney | are you sure we did not supprot this in the flavor extraspecs /image metadata | |
| 16:44:52 | spatel_ | I am going to add alias and get back to you.. | |
| 16:44:53 | stephenfin | definitely not | |
| 16:45:19 | sean-k-mooney | stephenfin: was the whole point of this feature to fix neutron sriov | |
| 16:45:53 | spatel_ | is product_id and vendore_id mandatory because in i am using devname here "pci_passthrough_whitelist = "{ "physical_network":"vlan", "devname":"eno2" }"" | |
| 16:46:23 | sean-k-mooney | spatel_: no | |
| 16:47:58 | sean-k-mooney | a white list can be in any of these forms https://github.com/openstack/nova/blob/master/nova/pci/devspec.py#L182-L192 | |
| 16:48:20 | sean-k-mooney | actully the alias yes that need to use vendor_id and product_id | |
| 16:48:34 | stephenfin | sean-k-mooney: If it was, it seems something may have slipped through the cracks here | |
| 16:48:48 | sean-k-mooney | alias are not for networking they are for passthrough devices | |
| 16:48:56 | stephenfin | Yup, I get that | |
| 16:49:12 | spatel_ | type-PCI, type-PF and type-VF what i should pick ? | |
| 16:49:17 | spatel_ | VF ? | |
| 16:49:19 | stephenfin | From the quick glance here, it should really be configured via the whitelist. I'm not sure why I went with the alias | |
| 16:49:22 | stephenfin | spatel_: yup | |
| 16:49:27 | spatel_ | doing it.. | |
| 16:49:33 | sean-k-mooney | stephenfin: ya i am go to confirm https://bugs.launchpad.net/nova/+bug/1795920 | |
| 16:49:34 | openstack | Launchpad bug 1795920 in OpenStack Compute (nova) "SR-IOV shared PCI numa not working " [Undecided,Confirmed] | |
| 16:50:05 | sean-k-mooney | stephenfin: interested in working on this? if not ill add it to my list but this need to be fixed | |
| 16:50:15 | sean-k-mooney | and backported | |
| 16:50:21 | stephenfin | I won't tackle it tonight but I can do so, yeah | |
| 16:50:36 | stephenfin | not sure if we can backport though. It'll be a config file change | |
| 16:50:39 | sean-k-mooney | cool we liekly need to repreose the old spec | |
| 16:51:10 | sean-k-mooney | stephenfin: i was suggesting we need to add the flavor and image extraspecs so no config file change | |
| 16:51:47 | stephenfin | sean-k-mooney: Possibly, but before doing so I'd suggest going back and reading the spec reviews | |
| 16:51:59 | stephenfin | There was a reason we didn't do that, though I don't recall it now :/ | |
| 16:52:18 | spatel | sean-k-mooney: this is what i change in nova.conf http://paste.openstack.org/show/731417/ | |
| 16:52:22 | stephenfin | If it's image metadata changes, we can't backport those due to object changes | |
| 16:52:24 | spatel | can you verify | |
| 16:52:33 | sean-k-mooney | yes i rembere i was very against using the alisa but i never recalled dropping the extra specs | |
| 16:53:12 | spatel | going to launch instance now, figure cross | |
| 16:53:32 | sean-k-mooney | spatel: its goning to fail. | |