| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 16:49:27 | spatel_ | doing it.. | |
| 16:49:33 | sean-k-mooney | stephenfin: ya i am go to confirm https://bugs.launchpad.net/nova/+bug/1795920 | |
| 16:49:34 | openstack | Launchpad bug 1795920 in OpenStack Compute (nova) "SR-IOV shared PCI numa not working " [Undecided,Confirmed] | |
| 16:50:05 | sean-k-mooney | stephenfin: interested in working on this? if not ill add it to my list but this need to be fixed | |
| 16:50:15 | sean-k-mooney | and backported | |
| 16:50:21 | stephenfin | I won't tackle it tonight but I can do so, yeah | |
| 16:50:36 | stephenfin | not sure if we can backport though. It'll be a config file change | |
| 16:50:39 | sean-k-mooney | cool we liekly need to repreose the old spec | |
| 16:51:10 | sean-k-mooney | stephenfin: i was suggesting we need to add the flavor and image extraspecs so no config file change | |
| 16:51:47 | stephenfin | sean-k-mooney: Possibly, but before doing so I'd suggest going back and reading the spec reviews | |
| 16:51:59 | stephenfin | There was a reason we didn't do that, though I don't recall it now :/ | |
| 16:52:18 | spatel | sean-k-mooney: this is what i change in nova.conf http://paste.openstack.org/show/731417/ | |
| 16:52:22 | stephenfin | If it's image metadata changes, we can't backport those due to object changes | |
| 16:52:24 | spatel | can you verify | |
| 16:52:33 | sean-k-mooney | yes i rembere i was very against using the alisa but i never recalled dropping the extra specs | |
| 16:53:12 | spatel | going to launch instance now, figure cross | |
| 16:53:32 | sean-k-mooney | spatel: its goning to fail. | |
| 16:53:39 | spatel | ?? | |
| 16:53:43 | spatel | why? | |
| 16:53:48 | sean-k-mooney | looking at the code the feature was not finished | |
| 16:54:01 | spatel | Damn it :( | |
| 16:54:12 | spatel | so what is the deal here ? | |
| 16:54:25 | sean-k-mooney | the numa policies are only repected for flavor based pci device passhtoruh e.g. for things like gpu or acllorator cards | |
| 16:54:44 | sean-k-mooney | spatel: the spec was approve and the full feature was not merged | |
| 16:55:09 | spatel | in my case its VF | |
| 16:55:34 | spatel | currently i can't utilize both numa :( | |
| 16:55:47 | sean-k-mooney | spatel: in your case its a neutron port with vnic_type direct correct | |
| 16:55:52 | spatel | yes | |
| 16:56:04 | sean-k-mooney | spatel: the workaround is to make the guest have 2 numa nodes | |
| 16:56:11 | melwitt | sean-k-mooney: which spec was not properly finished? | |
| 16:56:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/607650 | |
| 16:56:34 | mriedem | spatel: fyi ^ | |
| 16:56:47 | sean-k-mooney | melwitt: the numa pci polices | |
| 16:57:00 | sean-k-mooney | ill get the link on sec | |
| 16:57:19 | melwitt | thanks. I didn't see a link in the backscroll | |
| 16:57:23 | spatel | sean-k-mooney: "guest have 2 numa nodes" can you explain this? | |
| 16:57:40 | sean-k-mooney | melwitt: https://review.openstack.org/#/c/361140/ | |
| 16:57:54 | openstackgerrit | Vlad Gusev proposed openstack/nova master: Not set instance to ERROR if set_admin_password failed https://review.openstack.org/555160 | |
| 16:57:55 | sean-k-mooney | spatel: in the flavor set hw:numa_nodes=2 | |
| 16:58:08 | spatel | let me try hold on... | |
| 16:58:23 | sean-k-mooney | this will create a guest with 2 numa nodes with half the cpus and ram on each | |
| 16:58:44 | spatel | sean-k-mooney: FYI, i have tried this and it failed "hw:cpu_policy='dedicated', hw:numa_nodes='2', hw:pci_numa_affinity_policy='preferred'" | |
| 16:59:11 | spatel | now i am going to remove "hw:cpu_policy='dedicated" and "hw:pci_numa_affinity_policy='preferred'" to see if that work | |
| 16:59:13 | sean-k-mooney | melwitt: the flavor and image extraspecs are appently not implemente meaning that this does not work for neutron sriov ports as it should | |
| 16:59:18 | melwitt | sean-k-mooney: ok, according to the notes on the blueprint, people thought it was completed "The last functional patch for this was merged on Dec 30, 2017" https://blueprints.launchpad.net/openstack/nova/+spec/share-pci-between-numa-nodes | |
| 16:59:55 | spatel | melwitt: that is why i am chasing that blueprint because it says completed 2017 | |
| 17:00:12 | sean-k-mooney | melwitt: i taught it was complted im checking the code to confirm but apparently its not working | |
| 17:00:30 | spatel | Do we have any ETA because i have 100 compute node in racks and here i am stuck with this issue :( | |
| 17:00:52 | melwitt | I feel like someone has asked me about this bp before, asking if it applies to SRIOV too, and I thought since it never mentions SRIOV, that it doesn't | |
| 17:01:05 | sean-k-mooney | i can proably test this locally too i have jsut set up a sriov host | |
| 17:01:18 | melwitt | or wasn't meant to. and that adding SRIOV support would be additional work outside the scope of this particular blueprint | |
| 17:01:49 | sean-k-mooney | melwitt: its primary usecase was sriov specifics for telcos where they had to numa node server but all nics were connecte to one numa node due to space constriants in there rack | |
| 17:01:51 | melwitt | i.e. a new blueprint would be, for example, "add SRIOV support for sharing PCI devices between NUMA nodes" | |
| 17:03:06 | spatel | melwitt: its much clear now.. :) | |
| 17:03:31 | openstackgerrit | Merged openstack/nova stable/rocky: Explicitly fail if trying to attach SR-IOV port https://review.openstack.org/605118 | |
| 17:04:48 | mriedem | known issue yeah? https://bugs.launchpad.net/nova/+bug/1794717 | |
| 17:04:48 | openstack | Launchpad bug 1794717 in OpenStack Compute (nova) "rocky: ephemeral disk can not be resized" [Undecided,New] | |
| 17:05:11 | sean-k-mooney | mriedem: not being able to resise ephemeral disk ya | |
| 17:05:50 | sean-k-mooney | mriedem: i mean i thk i some very specific edgecase it can work today but there is no generic way to enable it | |
| 17:06:20 | sean-k-mooney | e.g. how to you resize form 1 500G disk to 2 400G disk so we just decided not to support it at all | |
| 17:06:47 | mriedem | yup, bug 1558880 | |
| 17:06:47 | openstack | bug 1558880 in OpenStack Compute (nova) "instance can not resize ephemeral in mitaka" [Medium,Confirmed] https://launchpad.net/bugs/1558880 | |
| 17:07:18 | melwitt | sean-k-mooney: ok, I don't know anything about that. if that spec scope is actually incomplete, then we need to decide how we deal with it. open another spec for this cycle to finish it or treat them as bugs | |
| 17:08:20 | sean-k-mooney | melwitt: proably reporpose the spec is the best way and jsut add the flavor exra specs and image metadata values that were orginaly proposed | |
| 17:08:55 | sean-k-mooney | melwitt: unless you think we can backport them in which case it could be a bug | |
| 17:09:47 | sean-k-mooney | backporting woudl be the only reason to make it a bug in my mind but its also addign new fuctionality e.g. tuning off numa affinty for pci devices | |
| 17:15:26 | sean-k-mooney | melwitt: i or stephen will repropsoe the spec | |
| 17:15:56 | sean-k-mooney | melwitt: stephenfin is heading home so i will likely do it later today | |
| 17:17:15 | melwitt | sean-k-mooney: I'd run the idea by mriedem too, in case he has another opinion on how to handle this | |
| 17:17:34 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208 | |
| 17:17:54 | sean-k-mooney | melwitt: sure i just felt a little checky to sneak it in as a bug fix :) | |
| 17:18:56 | bauzas | dansmith: mriedem: I tested the vgpu reshape for allocations too and good news it works ! I just fixed a few things that I discovered when testing ^ | |
| 17:19:06 | melwitt | sean-k-mooney: yeah, you're probably right. I didn't think much about it | |
| 17:19:13 | bauzas | now, call it a day | |
| 17:23:58 | sean-k-mooney | spatel: where you able to use a multi numa node guest to spawn the instance. that funtionality definetly works | |
| 17:24:54 | spatel | Testing it now..should i add "hw:cpu_policy='dedicated'" too for pinning ? | |
| 17:25:02 | dansmith | bauzas: ack | |
| 17:25:17 | sean-k-mooney | spatel: yes if you want cpu pinning then add hw:cpu_policy='dedicated' | |
| 17:25:38 | spatel | doing it.. hold on.. soon report back | |
| 17:27:26 | sean-k-mooney | no rush | |
| 17:35:38 | spatel | sean-k-mooney: i am able to launch two VM with 10vCPU core each ( i have 32 core compute node with 16+16 numa) but look like it didn't pin down CPU | |
| 17:35:41 | spatel | check this out http://paste.openstack.org/show/731420/ | |
| 17:35:55 | spatel | I can see it pin CPU cross numa | |
| 17:37:00 | sean-k-mooney | spatel: can you run virs dumpxml <instance> | |
| 17:37:27 | sean-k-mooney | spatel: i think it pinned everythin correctly | |
| 17:37:57 | spatel | http://paste.openstack.org/show/731423/ | |
| 17:37:58 | sean-k-mooney | spatel: it looks like each | |
| 17:38:27 | spatel | I thought it should pin all vCPU core with same NUMA node CPU right? | |
| 17:38:40 | sean-k-mooney | no | |
| 17:38:44 | spatel | hmmmm? | |
| 17:39:10 | sean-k-mooney | by setting hw:numa_nodes=2 you will have half the cpus on one numa node and half on the other | |
| 17:39:23 | sean-k-mooney | memory will also be equally split | |
| 17:40:13 | sean-k-mooney | provided there is a pci device free on at least one of the 2 numa nodes assocaicated with the vm vcpus then we will allow the vm to boot | |
| 17:40:26 | spatel | if i remove numa_node=2 then it will pin down all vCPU on same node right? | |
| 17:40:57 | sean-k-mooney | correct addint hw:cpu_policy=dedicated implictly adds hw:numa_nodes=1 | |
| 17:41:25 | spatel | hmm! interesting.. | |
| 17:41:53 | spatel | using numa_node=2 will have some performing issue right? | |
| 17:41:53 | sean-k-mooney | by explcitly setting hw:numa_nodes=2 it will allow both numa nodes on the host to be used but it will also limit the vm to host with 2+ numa nodes | |
| 17:42:25 | sean-k-mooney | spatel: it can if the application in the gust itself does not understand numa affinity | |
| 17:43:02 | sean-k-mooney | it can also improve the performacne as you doubles your memory bandwith as the vm will now use memroy form 2 host numa noes/memory controlers | |