| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 16:54:44 | sean-k-mooney | spatel: the spec was approve and the full feature was not merged | |
| 16:55:09 | spatel | in my case its VF | |
| 16:55:34 | spatel | currently i can't utilize both numa :( | |
| 16:55:47 | sean-k-mooney | spatel: in your case its a neutron port with vnic_type direct correct | |
| 16:55:52 | spatel | yes | |
| 16:56:04 | sean-k-mooney | spatel: the workaround is to make the guest have 2 numa nodes | |
| 16:56:11 | melwitt | sean-k-mooney: which spec was not properly finished? | |
| 16:56:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/607650 | |
| 16:56:34 | mriedem | spatel: fyi ^ | |
| 16:56:47 | sean-k-mooney | melwitt: the numa pci polices | |
| 16:57:00 | sean-k-mooney | ill get the link on sec | |
| 16:57:19 | melwitt | thanks. I didn't see a link in the backscroll | |
| 16:57:23 | spatel | sean-k-mooney: "guest have 2 numa nodes" can you explain this? | |
| 16:57:40 | sean-k-mooney | melwitt: https://review.openstack.org/#/c/361140/ | |
| 16:57:54 | openstackgerrit | Vlad Gusev proposed openstack/nova master: Not set instance to ERROR if set_admin_password failed https://review.openstack.org/555160 | |
| 16:57:55 | sean-k-mooney | spatel: in the flavor set hw:numa_nodes=2 | |
| 16:58:08 | spatel | let me try hold on... | |
| 16:58:23 | sean-k-mooney | this will create a guest with 2 numa nodes with half the cpus and ram on each | |
| 16:58:44 | spatel | sean-k-mooney: FYI, i have tried this and it failed "hw:cpu_policy='dedicated', hw:numa_nodes='2', hw:pci_numa_affinity_policy='preferred'" | |
| 16:59:11 | spatel | now i am going to remove "hw:cpu_policy='dedicated" and "hw:pci_numa_affinity_policy='preferred'" to see if that work | |
| 16:59:13 | sean-k-mooney | melwitt: the flavor and image extraspecs are appently not implemente meaning that this does not work for neutron sriov ports as it should | |
| 16:59:18 | melwitt | sean-k-mooney: ok, according to the notes on the blueprint, people thought it was completed "The last functional patch for this was merged on Dec 30, 2017" https://blueprints.launchpad.net/openstack/nova/+spec/share-pci-between-numa-nodes | |
| 16:59:55 | spatel | melwitt: that is why i am chasing that blueprint because it says completed 2017 | |
| 17:00:12 | sean-k-mooney | melwitt: i taught it was complted im checking the code to confirm but apparently its not working | |
| 17:00:30 | spatel | Do we have any ETA because i have 100 compute node in racks and here i am stuck with this issue :( | |
| 17:00:52 | melwitt | I feel like someone has asked me about this bp before, asking if it applies to SRIOV too, and I thought since it never mentions SRIOV, that it doesn't | |
| 17:01:05 | sean-k-mooney | i can proably test this locally too i have jsut set up a sriov host | |
| 17:01:18 | melwitt | or wasn't meant to. and that adding SRIOV support would be additional work outside the scope of this particular blueprint | |
| 17:01:49 | sean-k-mooney | melwitt: its primary usecase was sriov specifics for telcos where they had to numa node server but all nics were connecte to one numa node due to space constriants in there rack | |
| 17:01:51 | melwitt | i.e. a new blueprint would be, for example, "add SRIOV support for sharing PCI devices between NUMA nodes" | |
| 17:03:06 | spatel | melwitt: its much clear now.. :) | |
| 17:03:31 | openstackgerrit | Merged openstack/nova stable/rocky: Explicitly fail if trying to attach SR-IOV port https://review.openstack.org/605118 | |
| 17:04:48 | openstack | Launchpad bug 1794717 in OpenStack Compute (nova) "rocky: ephemeral disk can not be resized" [Undecided,New] | |
| 17:04:48 | mriedem | known issue yeah? https://bugs.launchpad.net/nova/+bug/1794717 | |
| 17:05:11 | sean-k-mooney | mriedem: not being able to resise ephemeral disk ya | |
| 17:05:50 | sean-k-mooney | mriedem: i mean i thk i some very specific edgecase it can work today but there is no generic way to enable it | |
| 17:06:20 | sean-k-mooney | e.g. how to you resize form 1 500G disk to 2 400G disk so we just decided not to support it at all | |
| 17:06:47 | openstack | bug 1558880 in OpenStack Compute (nova) "instance can not resize ephemeral in mitaka" [Medium,Confirmed] https://launchpad.net/bugs/1558880 | |
| 17:06:47 | mriedem | yup, bug 1558880 | |
| 17:07:18 | melwitt | sean-k-mooney: ok, I don't know anything about that. if that spec scope is actually incomplete, then we need to decide how we deal with it. open another spec for this cycle to finish it or treat them as bugs | |
| 17:08:20 | sean-k-mooney | melwitt: proably reporpose the spec is the best way and jsut add the flavor exra specs and image metadata values that were orginaly proposed | |
| 17:08:55 | sean-k-mooney | melwitt: unless you think we can backport them in which case it could be a bug | |
| 17:09:47 | sean-k-mooney | backporting woudl be the only reason to make it a bug in my mind but its also addign new fuctionality e.g. tuning off numa affinty for pci devices | |
| 17:15:26 | sean-k-mooney | melwitt: i or stephen will repropsoe the spec | |
| 17:15:56 | sean-k-mooney | melwitt: stephenfin is heading home so i will likely do it later today | |
| 17:17:15 | melwitt | sean-k-mooney: I'd run the idea by mriedem too, in case he has another opinion on how to handle this | |
| 17:17:34 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208 | |
| 17:17:54 | sean-k-mooney | melwitt: sure i just felt a little checky to sneak it in as a bug fix :) | |
| 17:18:56 | bauzas | dansmith: mriedem: I tested the vgpu reshape for allocations too and good news it works ! I just fixed a few things that I discovered when testing ^ | |
| 17:19:06 | melwitt | sean-k-mooney: yeah, you're probably right. I didn't think much about it | |
| 17:19:13 | bauzas | now, call it a day | |
| 17:23:58 | sean-k-mooney | spatel: where you able to use a multi numa node guest to spawn the instance. that funtionality definetly works | |
| 17:24:54 | spatel | Testing it now..should i add "hw:cpu_policy='dedicated'" too for pinning ? | |
| 17:25:02 | dansmith | bauzas: ack | |
| 17:25:17 | sean-k-mooney | spatel: yes if you want cpu pinning then add hw:cpu_policy='dedicated' | |
| 17:25:38 | spatel | doing it.. hold on.. soon report back | |
| 17:27:26 | sean-k-mooney | no rush | |
| 17:35:38 | spatel | sean-k-mooney: i am able to launch two VM with 10vCPU core each ( i have 32 core compute node with 16+16 numa) but look like it didn't pin down CPU | |
| 17:35:41 | spatel | check this out http://paste.openstack.org/show/731420/ | |
| 17:35:55 | spatel | I can see it pin CPU cross numa | |
| 17:37:00 | sean-k-mooney | spatel: can you run virs dumpxml <instance> | |
| 17:37:27 | sean-k-mooney | spatel: i think it pinned everythin correctly | |
| 17:37:57 | spatel | http://paste.openstack.org/show/731423/ | |
| 17:37:58 | sean-k-mooney | spatel: it looks like each | |
| 17:38:27 | spatel | I thought it should pin all vCPU core with same NUMA node CPU right? | |
| 17:38:40 | sean-k-mooney | no | |
| 17:38:44 | spatel | hmmmm? | |
| 17:39:10 | sean-k-mooney | by setting hw:numa_nodes=2 you will have half the cpus on one numa node and half on the other | |
| 17:39:23 | sean-k-mooney | memory will also be equally split | |
| 17:40:13 | sean-k-mooney | provided there is a pci device free on at least one of the 2 numa nodes assocaicated with the vm vcpus then we will allow the vm to boot | |
| 17:40:26 | spatel | if i remove numa_node=2 then it will pin down all vCPU on same node right? | |
| 17:40:57 | sean-k-mooney | correct addint hw:cpu_policy=dedicated implictly adds hw:numa_nodes=1 | |
| 17:41:25 | spatel | hmm! interesting.. | |
| 17:41:53 | sean-k-mooney | by explcitly setting hw:numa_nodes=2 it will allow both numa nodes on the host to be used but it will also limit the vm to host with 2+ numa nodes | |
| 17:41:53 | spatel | using numa_node=2 will have some performing issue right? | |
| 17:42:25 | sean-k-mooney | spatel: it can if the application in the gust itself does not understand numa affinity | |
| 17:43:02 | sean-k-mooney | it can also improve the performacne as you doubles your memory bandwith as the vm will now use memroy form 2 host numa noes/memory controlers | |
| 17:43:26 | spatel | I think time to run some test... | |
| 17:43:38 | spatel | We are media company and using VoIP base application | |
| 17:43:57 | sean-k-mooney | testing is always a good idea :) | |
| 17:44:19 | spatel | First i build openstack without SR-IOV and found performance was horrible (PPS rate was only 50k after that it start dropping packets) | |
| 17:44:20 | sean-k-mooney | as a comunity we have done a lot of work to improve numa affinity over the years | |
| 17:45:14 | spatel | I have just started learning numa stuff so i am new but it looks interesting.. | |
| 17:45:18 | sean-k-mooney | spatel: the stict pci numa affinity was added for telco usescases wehre they could not tolerate cross numa pci/sriov | |
| 17:45:56 | sean-k-mooney | spatel: it certenly is .... interesting. its also a pain in the ass but give better performace when you get it right | |
| 17:46:11 | spatel | I have some legacy hardware and i have to stick to them | |
| 17:46:27 | spatel | other side i am planning to test DPDK if its better | |
| 17:46:31 | sean-k-mooney | numa is not going away infact its become more common | |
| 17:46:44 | sean-k-mooney | dpdk is much better then kernel ovs | |
| 17:46:51 | sean-k-mooney | but its more complicated too | |
| 17:47:05 | spatel | but it doesn't need hardware dependency atleast | |
| 17:47:37 | spatel | I spent thousand of $$$$ to get SR-IOV supported card | |
| 17:47:38 | sean-k-mooney | spatel: not in the same way it requires that the guests use hugepages and that there is a dpdk driver for your nic | |
| 17:48:19 | spatel | Does it perform like SR-IOV ? | |
| 17:48:22 | sean-k-mooney | spatel: ya dpdk will be cheaeper in that sense but you will have to dedicate 1-2 cores to handel trafic for ovs-dpdk | |
| 17:48:50 | sean-k-mooney | spatel: in some cases yes. in general not quite | |
| 17:49:14 | sean-k-mooney | what data rates / traffic profiles are you targeting? | |
| 17:49:31 | spatel | currently i am deploying VoIP application on 1U server with 32 core / 32G memory. and i have 1000 servers... | |
| 17:49:34 | sean-k-mooney | 10G small packets 40G jumbo frames? a mix | |
| 17:50:01 | spatel | my peak in production 200 to 230kpps UDP packet rate | |