| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 17:21:15 | sean-k-mooney | i never bother to check what exacatly add the iptables ruels athat break it but the fix was simple so i never felt the need | |
| 17:28:19 | kashyap | lyarwood: gibi: The above patch (I resolved conflicts from a rebase) is chunky ... but solves a real live migration problem. In the past it was a blueprint and I wrote some docs in a "spec". But there's a case to be made for it to be an "advanced bug-fix" | |
| 17:28:47 | kashyap | I now need to go out, perhaps I can bring this up on the upstream meeting for discussion | |
| 17:31:41 | kashyap | [That needs to be split up for easier review] | |
| 17:32:09 | gibi | kashyap: ack, I have to look at it tomorrow | |
| 18:33:15 | sean-k-mooney | stephenfin: gibi i am doing some testing of vdpa for different api actions tl;dr i need to block shelve in addtion to livemigrate which was already planned. | |
| 18:33:47 | sean-k-mooney | but the reason i need to block shelve is because the vdpa change shared the same code for shelve as normal neutron vf ports | |
| 18:33:54 | sean-k-mooney | e.g. vnic_type=direct | |
| 18:33:58 | sean-k-mooney | so it also hits https://bugzilla.redhat.com/show_bug.cgi?id=1767797 | |
| 18:34:00 | openstack | bugzilla.redhat.com bug 1767797 in openstack-nova "When unshelving an SR-IOV instance, the binding profile isn't reclaimed or rescheduled, and this might cause PCI-PT conflicts" [High,Assigned] - Assigned to alifshit | |
| 18:34:40 | sean-k-mooney | nova is correctly claiming the device in the pci tracker but not updating the neutron port | |
| 18:34:47 | sean-k-mooney | before we regenerate the xml | |
| 18:35:21 | sean-k-mooney | im wondering if i should block shelve for all sriov port types including vdpa,vf and pfs | |
| 18:35:32 | stephenfin | I think that would be wise | |
| 18:35:42 | sean-k-mooney | or even do it in two patches? | |
| 18:35:58 | sean-k-mooney | 1 for the vdpa related things and one for the other sriov port type so that could be backported? | |
| 18:36:41 | sean-k-mooney | if i only have one vm unshelve works fine but its just because it can use the same vf/vdpa device | |
| 18:36:50 | sean-k-mooney | so if we fix that bug it will work | |
| 18:37:53 | stephenfin | Two patches make sense | |
| 18:37:59 | sean-k-mooney | i kind of feel this is like the numa live migration case. it sort of works but fundementally its broken, although this time i think we can backport a fix for unshleve with sriov | |
| 18:39:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Follow up from bp/pci-socket-affinity series https://review.opendev.org/c/openstack/nova/+/779556 | |
| 18:45:40 | lyarwood | sean-k-mooney: so it looks like the issue is with focal nodes still using tgtadm, just going to use your cloud to rebuild a multinode env if that's okay | |
| 18:46:01 | sean-k-mooney | ya it shoudl be fine | |
| 18:46:16 | sean-k-mooney | assuming i has free space go for it if not tell me an i can shelve something | |
| 18:47:19 | lyarwood | sean-k-mooney: ah nvm, virt-builder supports focal now | |
| 18:47:20 | sean-k-mooney | ah ya it has about 20-30G of hugepages free | |
| 18:47:54 | sean-k-mooney | well there is space if you want to boot a coulpel of 8G vms you should be able ot spawn 3-4 | |
| 18:48:56 | sean-k-mooney | lyarwood: im planning to redpeloy the cloud in a month or two and enable memory over subsciption. curretly everything uses hugepages but for how lightly used the vms are it proably makes snese to not do that | |
| 18:56:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: fakelibvirt: make kB_mem default not laughable https://review.opendev.org/c/openstack/nova/+/779559 | |
| 18:58:37 | lyarwood | sean-k-mooney: kk | |
| 18:59:09 | sean-k-mooney | artom: lyarwood since ye appare to be around could ye way in on https://review.opendev.org/c/openstack/nova/+/778347/4 just trying to get more input before i respin | |
| 18:59:51 | sean-k-mooney | artom: lyarwood basicaly too questions should we use a diffferent name e.g.hw:mem_lock or hw:locked_memoy instead of hw:mlock | |
| 19:00:04 | sean-k-mooney | and should that require hw:mem_page_size to be set | |
| 19:00:37 | lyarwood | sean-k-mooney: I'll look once I've kicked off devstack | |
| 19:00:42 | sean-k-mooney | thanks | |
| 19:00:44 | lyarwood | sean-k-mooney: did you move your jump host again btw? | |
| 19:01:28 | sean-k-mooney | no it should still be dyn.seanmooney.info | |
| 19:01:47 | sean-k-mooney | openstack.seanmooney.info is loadbalanced by cloudflares cdn | |
| 19:02:48 | lyarwood | yeah there we go, I still had openstack.seanmooney.info | |
| 19:02:55 | lyarwood | .ssh/config updated | |
| 19:02:59 | sean-k-mooney | dyn.seanmooney.info seams to be working for me but because of nat i cant really test that properly | |
| 19:03:03 | sean-k-mooney | ah ok | |
| 19:04:30 | artom | sean-k-mooney, I'm obviously missing context here, but do we need that patch at all right now? | |
| 19:04:45 | sean-k-mooney | artom: yep | |
| 19:04:46 | artom | As in, why not continue to add it implicitly when we detect a VDPA device? | |
| 19:05:00 | artom | Sorry, not "continue to add", but just "add implicitly" | |
| 19:05:01 | sean-k-mooney | libvirt does not do it right now | |
| 19:05:15 | artom | Like you're saying we do for SEV and realtime | |
| 19:05:18 | sean-k-mooney | and the way libvirt currently does it is a big problem for us | |
| 19:05:23 | artom | Ah, so Nova doesn't do it, *libvirt* does it | |
| 19:05:28 | sean-k-mooney | yep | |
| 19:05:29 | artom | (For SEV and realtime) | |
| 19:05:40 | sean-k-mooney | for sev and realtime nova does it | |
| 19:05:45 | sean-k-mooney | but that is slightly differnt | |
| 19:05:50 | artom | Tbh, I think having to set an extra spec that you have no choice for is bad UX, no? | |
| 19:06:11 | artom | What's preventing Nova from detecting VDPA devices and adding the required XML? | |
| 19:06:20 | sean-k-mooney | am kind of but we shoudl not be changing the memory we sare using based on a neutorn port | |
| 19:06:42 | sean-k-mooney | artom: tl;dr our current memory tracking is really broken | |
| 19:06:44 | artom | Ah, because not all VDPA devices require it? | |
| 19:07:20 | sean-k-mooney | and vm that vfio(sriov port or pci passthough) vgpu or nvmeof device is being locked in memory by libvirt | |
| 19:07:32 | sean-k-mooney | meaning oversubcript does not work | |
| 19:08:07 | sean-k-mooney | artom: am the simulator does not require it | |
| 19:08:20 | sean-k-mooney | artom: its not clear if dpdk based vdpa devices would | |
| 19:09:36 | sean-k-mooney | so i dont know maybe we could auto add it | |
| 19:11:12 | artom | You're saying "let's not change the memory based on Neutron port", except libvirt kinda does that already for vGPU, to use your own example | |
| 19:11:19 | sean-k-mooney | my conern right now is to track locked memory proertly we might need severly restrict what type of vms can use neutron sriov port or passthough of pci or vgpus deivces | |
| 19:11:39 | sean-k-mooney | artom: well it does it for neutron VF ports | |
| 19:11:58 | sean-k-mooney | so we are already inadvertely doing it based on ports | |
| 19:12:11 | artom | Yeah | |
| 19:12:24 | sean-k-mooney | well libvirt is | |
| 19:12:29 | artom | UX-wise, if something needs doing regardless, we should be asking the user to do it for us | |
| 19:12:54 | artom | The fact that our memory tracking is broken is a tangential problem :P | |
| 19:13:06 | sean-k-mooney | well im concerned we might need to block vms that dont use hw:mem_page_size form using sriov port in the future to solve this issue | |
| 19:14:06 | sean-k-mooney | artom: ya it is but i was trying not to boil the ocean i guess doing it based on the port type make sense | |
| 19:14:22 | sean-k-mooney | i was orginaly hoping this was just a bug that would go away | |
| 19:14:31 | artom | I get it, the deadline is looming and you're rushing | |
| 19:14:43 | sean-k-mooney | partly that | |
| 19:15:04 | sean-k-mooney | and partly in theory mellonx/nvidia coudl fix this is there driver suppport page faults | |
| 19:15:19 | sean-k-mooney | they dont right now and may never but if they did it would not need to be locked | |
| 19:15:54 | sean-k-mooney | the vdpa sim module doe snot need locking but it likely either support page fause or just does not use dma memory | |
| 19:16:54 | artom | My paranoid conservative opinion is that this needs to be figured out in a spec next cycle ;) | |
| 19:17:01 | artom | Instead of panic-merging stuff ;) | |
| 19:17:36 | sean-k-mooney | well we discussed locked memory before for sev and realtime but did not have a usecase for it. | |
| 19:17:45 | sean-k-mooney | i could also just mark the guest as realtime today | |
| 19:18:19 | sean-k-mooney | that requirement would go away when libvirt start treating vdpa like a vf and upping the mlock limit | |
| 19:18:37 | sean-k-mooney | maybe that a better idea for now. | |
| 19:19:07 | sean-k-mooney | so no new extra spec but require a newer libvirt or a realtime guest. | |
| 19:19:18 | sean-k-mooney | i agree though i would like to not rush this | |
| 19:28:16 | sean-k-mooney | artom: for what its worth to use ovs-dpdk and vhost user you have to set hw:mem_page_size=large | |
| 19:28:29 | sean-k-mooney | other wise it wont work the same way that vdpa breaks today | |
| 19:28:36 | sean-k-mooney | if you dont add locked | |
| 19:28:48 | artom | Yeah, I agree there's precedent | |
| 19:29:11 | artom | (In terms for breaking unless the user does something they strictly-speaking should not have to do) | |
| 19:29:23 | artom | But... doesn't mean we shouldn't strive to improve on that :) | |
| 19:30:01 | sean-k-mooney | well again we did not wnat reqouce usage to change basked on port type | |
| 19:30:15 | sean-k-mooney | that is why you were required to enable hugepages in the flavor or image | |
| 19:30:43 | sean-k-mooney | there are othere issue this create for attach too | |
| 19:31:15 | artom | Well you clearly shouldn't be allowed to attach ports that require locked memory to a running instance :) | |
| 19:31:37 | artom | (Unless it already has locked memory) | |
| 19:32:23 | sean-k-mooney | yep | |