| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 18:39:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Follow up from bp/pci-socket-affinity series https://review.opendev.org/c/openstack/nova/+/779556 | |
| 18:45:40 | lyarwood | sean-k-mooney: so it looks like the issue is with focal nodes still using tgtadm, just going to use your cloud to rebuild a multinode env if that's okay | |
| 18:46:01 | sean-k-mooney | ya it shoudl be fine | |
| 18:46:16 | sean-k-mooney | assuming i has free space go for it if not tell me an i can shelve something | |
| 18:47:19 | lyarwood | sean-k-mooney: ah nvm, virt-builder supports focal now | |
| 18:47:20 | sean-k-mooney | ah ya it has about 20-30G of hugepages free | |
| 18:47:54 | sean-k-mooney | well there is space if you want to boot a coulpel of 8G vms you should be able ot spawn 3-4 | |
| 18:48:56 | sean-k-mooney | lyarwood: im planning to redpeloy the cloud in a month or two and enable memory over subsciption. curretly everything uses hugepages but for how lightly used the vms are it proably makes snese to not do that | |
| 18:56:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: fakelibvirt: make kB_mem default not laughable https://review.opendev.org/c/openstack/nova/+/779559 | |
| 18:58:37 | lyarwood | sean-k-mooney: kk | |
| 18:59:09 | sean-k-mooney | artom: lyarwood since ye appare to be around could ye way in on https://review.opendev.org/c/openstack/nova/+/778347/4 just trying to get more input before i respin | |
| 18:59:51 | sean-k-mooney | artom: lyarwood basicaly too questions should we use a diffferent name e.g.hw:mem_lock or hw:locked_memoy instead of hw:mlock | |
| 19:00:04 | sean-k-mooney | and should that require hw:mem_page_size to be set | |
| 19:00:37 | lyarwood | sean-k-mooney: I'll look once I've kicked off devstack | |
| 19:00:42 | sean-k-mooney | thanks | |
| 19:00:44 | lyarwood | sean-k-mooney: did you move your jump host again btw? | |
| 19:01:28 | sean-k-mooney | no it should still be dyn.seanmooney.info | |
| 19:01:47 | sean-k-mooney | openstack.seanmooney.info is loadbalanced by cloudflares cdn | |
| 19:02:48 | lyarwood | yeah there we go, I still had openstack.seanmooney.info | |
| 19:02:55 | lyarwood | .ssh/config updated | |
| 19:02:59 | sean-k-mooney | dyn.seanmooney.info seams to be working for me but because of nat i cant really test that properly | |
| 19:03:03 | sean-k-mooney | ah ok | |
| 19:04:30 | artom | sean-k-mooney, I'm obviously missing context here, but do we need that patch at all right now? | |
| 19:04:45 | sean-k-mooney | artom: yep | |
| 19:04:46 | artom | As in, why not continue to add it implicitly when we detect a VDPA device? | |
| 19:05:00 | artom | Sorry, not "continue to add", but just "add implicitly" | |
| 19:05:01 | sean-k-mooney | libvirt does not do it right now | |
| 19:05:15 | artom | Like you're saying we do for SEV and realtime | |
| 19:05:18 | sean-k-mooney | and the way libvirt currently does it is a big problem for us | |
| 19:05:23 | artom | Ah, so Nova doesn't do it, *libvirt* does it | |
| 19:05:28 | sean-k-mooney | yep | |
| 19:05:29 | artom | (For SEV and realtime) | |
| 19:05:40 | sean-k-mooney | for sev and realtime nova does it | |
| 19:05:45 | sean-k-mooney | but that is slightly differnt | |
| 19:05:50 | artom | Tbh, I think having to set an extra spec that you have no choice for is bad UX, no? | |
| 19:06:11 | artom | What's preventing Nova from detecting VDPA devices and adding the required XML? | |
| 19:06:20 | sean-k-mooney | am kind of but we shoudl not be changing the memory we sare using based on a neutorn port | |
| 19:06:42 | sean-k-mooney | artom: tl;dr our current memory tracking is really broken | |
| 19:06:44 | artom | Ah, because not all VDPA devices require it? | |
| 19:07:20 | sean-k-mooney | and vm that vfio(sriov port or pci passthough) vgpu or nvmeof device is being locked in memory by libvirt | |
| 19:07:32 | sean-k-mooney | meaning oversubcript does not work | |
| 19:08:07 | sean-k-mooney | artom: am the simulator does not require it | |
| 19:08:20 | sean-k-mooney | artom: its not clear if dpdk based vdpa devices would | |
| 19:09:36 | sean-k-mooney | so i dont know maybe we could auto add it | |
| 19:11:12 | artom | You're saying "let's not change the memory based on Neutron port", except libvirt kinda does that already for vGPU, to use your own example | |
| 19:11:19 | sean-k-mooney | my conern right now is to track locked memory proertly we might need severly restrict what type of vms can use neutron sriov port or passthough of pci or vgpus deivces | |
| 19:11:39 | sean-k-mooney | artom: well it does it for neutron VF ports | |
| 19:11:58 | sean-k-mooney | so we are already inadvertely doing it based on ports | |
| 19:12:11 | artom | Yeah | |
| 19:12:24 | sean-k-mooney | well libvirt is | |
| 19:12:29 | artom | UX-wise, if something needs doing regardless, we should be asking the user to do it for us | |
| 19:12:54 | artom | The fact that our memory tracking is broken is a tangential problem :P | |
| 19:13:06 | sean-k-mooney | well im concerned we might need to block vms that dont use hw:mem_page_size form using sriov port in the future to solve this issue | |
| 19:14:06 | sean-k-mooney | artom: ya it is but i was trying not to boil the ocean i guess doing it based on the port type make sense | |
| 19:14:22 | sean-k-mooney | i was orginaly hoping this was just a bug that would go away | |
| 19:14:31 | artom | I get it, the deadline is looming and you're rushing | |
| 19:14:43 | sean-k-mooney | partly that | |
| 19:15:04 | sean-k-mooney | and partly in theory mellonx/nvidia coudl fix this is there driver suppport page faults | |
| 19:15:19 | sean-k-mooney | they dont right now and may never but if they did it would not need to be locked | |
| 19:15:54 | sean-k-mooney | the vdpa sim module doe snot need locking but it likely either support page fause or just does not use dma memory | |
| 19:16:54 | artom | My paranoid conservative opinion is that this needs to be figured out in a spec next cycle ;) | |
| 19:17:01 | artom | Instead of panic-merging stuff ;) | |
| 19:17:36 | sean-k-mooney | well we discussed locked memory before for sev and realtime but did not have a usecase for it. | |
| 19:17:45 | sean-k-mooney | i could also just mark the guest as realtime today | |
| 19:18:19 | sean-k-mooney | that requirement would go away when libvirt start treating vdpa like a vf and upping the mlock limit | |
| 19:18:37 | sean-k-mooney | maybe that a better idea for now. | |
| 19:19:07 | sean-k-mooney | so no new extra spec but require a newer libvirt or a realtime guest. | |
| 19:19:18 | sean-k-mooney | i agree though i would like to not rush this | |
| 19:28:16 | sean-k-mooney | artom: for what its worth to use ovs-dpdk and vhost user you have to set hw:mem_page_size=large | |
| 19:28:29 | sean-k-mooney | other wise it wont work the same way that vdpa breaks today | |
| 19:28:36 | sean-k-mooney | if you dont add locked | |
| 19:28:48 | artom | Yeah, I agree there's precedent | |
| 19:29:11 | artom | (In terms for breaking unless the user does something they strictly-speaking should not have to do) | |
| 19:29:23 | artom | But... doesn't mean we shouldn't strive to improve on that :) | |
| 19:30:01 | sean-k-mooney | well again we did not wnat reqouce usage to change basked on port type | |
| 19:30:15 | sean-k-mooney | that is why you were required to enable hugepages in the flavor or image | |
| 19:30:43 | sean-k-mooney | there are othere issue this create for attach too | |
| 19:31:15 | artom | Well you clearly shouldn't be allowed to attach ports that require locked memory to a running instance :) | |
| 19:31:37 | artom | (Unless it already has locked memory) | |
| 19:32:23 | sean-k-mooney | yep | |
| 19:32:30 | artom | Which we would need to track, and it could come from not only the extra spec, but also other source that caused libvirt to do it for us, etc etc | |
| 19:32:33 | sean-k-mooney | but not just running | |
| 19:32:34 | artom | Hence: spec discussion :) | |
| 19:32:43 | sean-k-mooney | it would be incorrect to change it on hard reboot | |
| 19:32:52 | sean-k-mooney | or iff it was off | |
| 19:33:06 | sean-k-mooney | e.g. you cant add to any instance already on the host | |
| 19:33:13 | sean-k-mooney | as it change the memory usage of the guest | |
| 19:33:30 | sean-k-mooney | same way addign a vhost-user port shoudl not make the guest suddenly use hugepages | |
| 19:57:00 | openstackgerrit | sean mooney proposed openstack/nova master: Support per port numa policies with SR-IOV https://review.opendev.org/c/openstack/nova/+/773792 | |
| 20:05:02 | lyarwood | sean-k-mooney: sorry had to go afk, so devstack is still running somehow. Is your host pretty loaded at the moment? | |
| 20:06:59 | sean-k-mooney | no load is at 4.38 on the host | |
| 20:07:10 | sean-k-mooney | it has 24cores/48threads | |
| 20:07:13 | lyarwood | weird | |
| 20:07:18 | sean-k-mooney | it might be ipv6 being slow | |
| 20:07:25 | lyarwood | I'm using the medium flavor with 12 vcpus | |
| 20:07:48 | sean-k-mooney | i dont have native ipv6 and the tunnel is sometimes slow | |
| 20:07:53 | lyarwood | I'm not using the async stuff but still, it's been running for over an hour now | |
| 20:08:00 | lyarwood | kk | |
| 20:08:17 | sean-k-mooney | ya that weird what does the load look like in the vm | |
| 20:09:01 | sean-k-mooney | io wait is also not that high in the host %0.90 | |