| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 19:05:45 | sean-k-mooney | but that is slightly differnt | |
| 19:05:50 | artom | Tbh, I think having to set an extra spec that you have no choice for is bad UX, no? | |
| 19:06:11 | artom | What's preventing Nova from detecting VDPA devices and adding the required XML? | |
| 19:06:20 | sean-k-mooney | am kind of but we shoudl not be changing the memory we sare using based on a neutorn port | |
| 19:06:42 | sean-k-mooney | artom: tl;dr our current memory tracking is really broken | |
| 19:06:44 | artom | Ah, because not all VDPA devices require it? | |
| 19:07:20 | sean-k-mooney | and vm that vfio(sriov port or pci passthough) vgpu or nvmeof device is being locked in memory by libvirt | |
| 19:07:32 | sean-k-mooney | meaning oversubcript does not work | |
| 19:08:07 | sean-k-mooney | artom: am the simulator does not require it | |
| 19:08:20 | sean-k-mooney | artom: its not clear if dpdk based vdpa devices would | |
| 19:09:36 | sean-k-mooney | so i dont know maybe we could auto add it | |
| 19:11:12 | artom | You're saying "let's not change the memory based on Neutron port", except libvirt kinda does that already for vGPU, to use your own example | |
| 19:11:19 | sean-k-mooney | my conern right now is to track locked memory proertly we might need severly restrict what type of vms can use neutron sriov port or passthough of pci or vgpus deivces | |
| 19:11:39 | sean-k-mooney | artom: well it does it for neutron VF ports | |
| 19:11:58 | sean-k-mooney | so we are already inadvertely doing it based on ports | |
| 19:12:11 | artom | Yeah | |
| 19:12:24 | sean-k-mooney | well libvirt is | |
| 19:12:29 | artom | UX-wise, if something needs doing regardless, we should be asking the user to do it for us | |
| 19:12:54 | artom | The fact that our memory tracking is broken is a tangential problem :P | |
| 19:13:06 | sean-k-mooney | well im concerned we might need to block vms that dont use hw:mem_page_size form using sriov port in the future to solve this issue | |
| 19:14:06 | sean-k-mooney | artom: ya it is but i was trying not to boil the ocean i guess doing it based on the port type make sense | |
| 19:14:22 | sean-k-mooney | i was orginaly hoping this was just a bug that would go away | |
| 19:14:31 | artom | I get it, the deadline is looming and you're rushing | |
| 19:14:43 | sean-k-mooney | partly that | |
| 19:15:04 | sean-k-mooney | and partly in theory mellonx/nvidia coudl fix this is there driver suppport page faults | |
| 19:15:19 | sean-k-mooney | they dont right now and may never but if they did it would not need to be locked | |
| 19:15:54 | sean-k-mooney | the vdpa sim module doe snot need locking but it likely either support page fause or just does not use dma memory | |
| 19:16:54 | artom | My paranoid conservative opinion is that this needs to be figured out in a spec next cycle ;) | |
| 19:17:01 | artom | Instead of panic-merging stuff ;) | |
| 19:17:36 | sean-k-mooney | well we discussed locked memory before for sev and realtime but did not have a usecase for it. | |
| 19:17:45 | sean-k-mooney | i could also just mark the guest as realtime today | |
| 19:18:19 | sean-k-mooney | that requirement would go away when libvirt start treating vdpa like a vf and upping the mlock limit | |
| 19:18:37 | sean-k-mooney | maybe that a better idea for now. | |
| 19:19:07 | sean-k-mooney | so no new extra spec but require a newer libvirt or a realtime guest. | |
| 19:19:18 | sean-k-mooney | i agree though i would like to not rush this | |
| 19:28:16 | sean-k-mooney | artom: for what its worth to use ovs-dpdk and vhost user you have to set hw:mem_page_size=large | |
| 19:28:29 | sean-k-mooney | other wise it wont work the same way that vdpa breaks today | |
| 19:28:36 | sean-k-mooney | if you dont add locked | |
| 19:28:48 | artom | Yeah, I agree there's precedent | |
| 19:29:11 | artom | (In terms for breaking unless the user does something they strictly-speaking should not have to do) | |
| 19:29:23 | artom | But... doesn't mean we shouldn't strive to improve on that :) | |
| 19:30:01 | sean-k-mooney | well again we did not wnat reqouce usage to change basked on port type | |
| 19:30:15 | sean-k-mooney | that is why you were required to enable hugepages in the flavor or image | |
| 19:30:43 | sean-k-mooney | there are othere issue this create for attach too | |
| 19:31:15 | artom | Well you clearly shouldn't be allowed to attach ports that require locked memory to a running instance :) | |
| 19:31:37 | artom | (Unless it already has locked memory) | |
| 19:32:23 | sean-k-mooney | yep | |
| 19:32:30 | artom | Which we would need to track, and it could come from not only the extra spec, but also other source that caused libvirt to do it for us, etc etc | |
| 19:32:33 | sean-k-mooney | but not just running | |
| 19:32:34 | artom | Hence: spec discussion :) | |
| 19:32:43 | sean-k-mooney | it would be incorrect to change it on hard reboot | |
| 19:32:52 | sean-k-mooney | or iff it was off | |
| 19:33:06 | sean-k-mooney | e.g. you cant add to any instance already on the host | |
| 19:33:13 | sean-k-mooney | as it change the memory usage of the guest | |
| 19:33:30 | sean-k-mooney | same way addign a vhost-user port shoudl not make the guest suddenly use hugepages | |
| 19:57:00 | openstackgerrit | sean mooney proposed openstack/nova master: Support per port numa policies with SR-IOV https://review.opendev.org/c/openstack/nova/+/773792 | |
| 20:05:02 | lyarwood | sean-k-mooney: sorry had to go afk, so devstack is still running somehow. Is your host pretty loaded at the moment? | |
| 20:06:59 | sean-k-mooney | no load is at 4.38 on the host | |
| 20:07:10 | sean-k-mooney | it has 24cores/48threads | |
| 20:07:13 | lyarwood | weird | |
| 20:07:18 | sean-k-mooney | it might be ipv6 being slow | |
| 20:07:25 | lyarwood | I'm using the medium flavor with 12 vcpus | |
| 20:07:48 | sean-k-mooney | i dont have native ipv6 and the tunnel is sometimes slow | |
| 20:07:53 | lyarwood | I'm not using the async stuff but still, it's been running for over an hour now | |
| 20:08:00 | lyarwood | kk | |
| 20:08:17 | sean-k-mooney | ya that weird what does the load look like in the vm | |
| 20:09:01 | sean-k-mooney | io wait is also not that high in the host %0.90 | |
| 20:10:45 | lyarwood | kk let me try again with DEVSTACK_PARALLEL=true | |
| 20:11:08 | sean-k-mooney | it should only take about 20 mings without it | |
| 20:12:45 | lyarwood | takes about 6 on a local f32 vm | |
| 20:12:54 | lyarwood | kk, it's looking quicker tbh | |
| 20:19:03 | sean-k-mooney | ya something is wrong with my ipv6 routing http://paste.openstack.org/show/803402/ | |
| 20:19:15 | sean-k-mooney | dns is working but i cant ping | |
| 20:19:28 | sean-k-mooney | thats form the server that is hosting the vm | |
| 20:30:08 | openstackgerrit | Merged openstack/nova master: pci manager: replace node_id parameter with compute_node https://review.opendev.org/c/openstack/nova/+/778747 | |
| 20:30:42 | openstackgerrit | Merged openstack/nova master: apidb: Compact Queens database migrations https://review.opendev.org/c/openstack/nova/+/759404 | |
| 20:31:08 | sean-k-mooney | lyarwood: problems on my routeer the tunnel is not working properly. for now if you do | |
| 20:31:11 | sean-k-mooney | sysctl -w net.ipv6.conf.all.disable_ipv6=1 | |
| 20:31:16 | sean-k-mooney | sysctl -w net.ipv6.conf.default.disable_ipv6=1 | |
| 20:31:20 | sean-k-mooney | it will work around the issue | |
| 20:31:33 | sean-k-mooney | with sudo of course | |
| 20:31:50 | sean-k-mooney | it shoudl be falling back to ipv4 anyway but that can slow down installs | |
| 20:32:20 | sean-k-mooney | this is one of the reason im going to be reinstallin my cloud in a month or too | |
| 20:32:29 | openstackgerrit | Merged openstack/nova master: pci: track host NUMA topology in stats https://review.opendev.org/c/openstack/nova/+/774149 | |
| 20:32:40 | sean-k-mooney | ipv6 is nice but somethiem it has issue so untill i have it nativly im going to remove it | |
| 20:32:59 | sean-k-mooney | am what? ^ | |
| 20:34:54 | sean-k-mooney | stephenfin: gibi the numa node is already in the pci_stats in the compute node table | |
| 20:35:11 | sean-k-mooney | as is the host numa toplogy blob | |
| 20:36:34 | sean-k-mooney | {"product_id": "101e", "vendor_id": "15b3", "numa_node": 0, "tags": {"dev_type": "vdpa", "physical_network": null, "parent_ifname": "enp6s0f0_0"}, "count": 3} | |
| 20:39:53 | sean-k-mooney | artom: i really wish you asked me to review that | |
| 20:40:14 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: fakelibvirt: make kB_mem default not laughable https://review.opendev.org/c/openstack/nova/+/779559 | |
| 20:40:16 | sean-k-mooney | these were class method because we did not want to store this data in an object | |
| 20:40:34 | artom | sean-k-mooney, the NUMA node yes, but not the socket | |
| 20:41:06 | sean-k-mooney | right but the socket you were goint to store/lookup seperately | |
| 20:41:27 | artom | sean-k-mooney, I save the host numa_topology in the pci stats object | |
| 20:41:34 | artom | (I forget what it's called exactly) | |
| 20:41:37 | sean-k-mooney | im reviewing https://review.opendev.org/c/openstack/nova/+/774149 now to fiture out what you cahgned but this is going to conflcit with most of my patches | |
| 20:42:24 | openstackgerrit | Merged openstack/nova master: conf: Clean up docs for scheduler options https://review.opendev.org/c/openstack/nova/+/773639 | |
| 20:42:57 | artom | Storing it (the numa_topology) in the object was cleaner than passing it through | |
| 20:43:24 | artom | Only the socket policy filter() method needs it, but it'd have been passed through like 42 other methods to get there | |