| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 16:55:08 | sean-k-mooney | i almost never ead release notes outside of git | |
| 16:55:36 | stephenfin | Yeah, figured you would | |
| 16:55:43 | stephenfin | you could do a comment | |
| 16:56:24 | sean-k-mooney | hum i guess i could. whats the yaml comment syntax // | |
| 16:56:32 | sean-k-mooney | it follows c right | |
| 16:56:40 | stephenfin | as I noted, release notes should avoid version information in general since it doesn't make sense for backported fixes. That's not applicable here but in general, it's a good guide | |
| 16:56:42 | sean-k-mooney | i think ill just drop it for now | |
| 16:56:55 | stephenfin | '#' I think, but I'm not sure | |
| 16:56:56 | sean-k-mooney | well this is a feature | |
| 16:57:02 | sean-k-mooney | so it wont be backported but sure | |
| 16:57:11 | stephenfin | yeah, like I said, not applicable here but a good guide | |
| 16:57:28 | stephenfin | just never include version information in the release note and you never need to think about it | |
| 16:57:56 | sean-k-mooney | im not sure i actully agree on the backport thing but i also dont wnat to spend time debating it FF week :) | |
| 17:17:25 | lyarwood | has anyone deployed a local multinode devstack env recently? for some reason I can't curl keystone on the controller from the compute but I can ssh and ping between the hosts just fine | |
| 17:18:16 | sean-k-mooney | lyarwood: likely iptables | |
| 17:18:34 | lyarwood | sean-k-mooney: disabled, as is firewalld | |
| 17:18:36 | sean-k-mooney | do "sudo iptables -F; sudo iptables -X" | |
| 17:18:56 | lyarwood | what the flying | |
| 17:18:59 | lyarwood | >< | |
| 17:19:07 | lyarwood | ggwp systemd | |
| 17:19:27 | lyarwood | the service was dead but for what ever reason systemd didn't flush the rules | |
| 17:19:31 | sean-k-mooney | devstack/neutorn? add some iptable rules directly | |
| 17:19:51 | sean-k-mooney | i used to keep that command in my local.sh | |
| 17:19:51 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Use improved guest CPU config APIs https://review.opendev.org/c/openstack/nova/+/762330 | |
| 17:20:00 | sean-k-mooney | so that devstack would just do it every time i stacked | |
| 17:20:07 | sean-k-mooney | i assume its working now? | |
| 17:20:08 | lyarwood | yeah not a bad idea | |
| 17:20:30 | lyarwood | yeah hopefully just stacking the compute again | |
| 17:21:15 | sean-k-mooney | i never bother to check what exacatly add the iptables ruels athat break it but the fix was simple so i never felt the need | |
| 17:28:19 | kashyap | lyarwood: gibi: The above patch (I resolved conflicts from a rebase) is chunky ... but solves a real live migration problem. In the past it was a blueprint and I wrote some docs in a "spec". But there's a case to be made for it to be an "advanced bug-fix" | |
| 17:28:47 | kashyap | I now need to go out, perhaps I can bring this up on the upstream meeting for discussion | |
| 17:31:41 | kashyap | [That needs to be split up for easier review] | |
| 17:32:09 | gibi | kashyap: ack, I have to look at it tomorrow | |
| 18:33:15 | sean-k-mooney | stephenfin: gibi i am doing some testing of vdpa for different api actions tl;dr i need to block shelve in addtion to livemigrate which was already planned. | |
| 18:33:47 | sean-k-mooney | but the reason i need to block shelve is because the vdpa change shared the same code for shelve as normal neutron vf ports | |
| 18:33:54 | sean-k-mooney | e.g. vnic_type=direct | |
| 18:33:58 | sean-k-mooney | so it also hits https://bugzilla.redhat.com/show_bug.cgi?id=1767797 | |
| 18:34:00 | openstack | bugzilla.redhat.com bug 1767797 in openstack-nova "When unshelving an SR-IOV instance, the binding profile isn't reclaimed or rescheduled, and this might cause PCI-PT conflicts" [High,Assigned] - Assigned to alifshit | |
| 18:34:40 | sean-k-mooney | nova is correctly claiming the device in the pci tracker but not updating the neutron port | |
| 18:34:47 | sean-k-mooney | before we regenerate the xml | |
| 18:35:21 | sean-k-mooney | im wondering if i should block shelve for all sriov port types including vdpa,vf and pfs | |
| 18:35:32 | stephenfin | I think that would be wise | |
| 18:35:42 | sean-k-mooney | or even do it in two patches? | |
| 18:35:58 | sean-k-mooney | 1 for the vdpa related things and one for the other sriov port type so that could be backported? | |
| 18:36:41 | sean-k-mooney | if i only have one vm unshelve works fine but its just because it can use the same vf/vdpa device | |
| 18:36:50 | sean-k-mooney | so if we fix that bug it will work | |
| 18:37:53 | stephenfin | Two patches make sense | |
| 18:37:59 | sean-k-mooney | i kind of feel this is like the numa live migration case. it sort of works but fundementally its broken, although this time i think we can backport a fix for unshleve with sriov | |
| 18:39:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Follow up from bp/pci-socket-affinity series https://review.opendev.org/c/openstack/nova/+/779556 | |
| 18:45:40 | lyarwood | sean-k-mooney: so it looks like the issue is with focal nodes still using tgtadm, just going to use your cloud to rebuild a multinode env if that's okay | |
| 18:46:01 | sean-k-mooney | ya it shoudl be fine | |
| 18:46:16 | sean-k-mooney | assuming i has free space go for it if not tell me an i can shelve something | |
| 18:47:19 | lyarwood | sean-k-mooney: ah nvm, virt-builder supports focal now | |
| 18:47:20 | sean-k-mooney | ah ya it has about 20-30G of hugepages free | |
| 18:47:54 | sean-k-mooney | well there is space if you want to boot a coulpel of 8G vms you should be able ot spawn 3-4 | |
| 18:48:56 | sean-k-mooney | lyarwood: im planning to redpeloy the cloud in a month or two and enable memory over subsciption. curretly everything uses hugepages but for how lightly used the vms are it proably makes snese to not do that | |
| 18:56:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: fakelibvirt: make kB_mem default not laughable https://review.opendev.org/c/openstack/nova/+/779559 | |
| 18:58:37 | lyarwood | sean-k-mooney: kk | |
| 18:59:09 | sean-k-mooney | artom: lyarwood since ye appare to be around could ye way in on https://review.opendev.org/c/openstack/nova/+/778347/4 just trying to get more input before i respin | |
| 18:59:51 | sean-k-mooney | artom: lyarwood basicaly too questions should we use a diffferent name e.g.hw:mem_lock or hw:locked_memoy instead of hw:mlock | |
| 19:00:04 | sean-k-mooney | and should that require hw:mem_page_size to be set | |
| 19:00:37 | lyarwood | sean-k-mooney: I'll look once I've kicked off devstack | |
| 19:00:42 | sean-k-mooney | thanks | |
| 19:00:44 | lyarwood | sean-k-mooney: did you move your jump host again btw? | |
| 19:01:28 | sean-k-mooney | no it should still be dyn.seanmooney.info | |
| 19:01:47 | sean-k-mooney | openstack.seanmooney.info is loadbalanced by cloudflares cdn | |
| 19:02:48 | lyarwood | yeah there we go, I still had openstack.seanmooney.info | |
| 19:02:55 | lyarwood | .ssh/config updated | |
| 19:02:59 | sean-k-mooney | dyn.seanmooney.info seams to be working for me but because of nat i cant really test that properly | |
| 19:03:03 | sean-k-mooney | ah ok | |
| 19:04:30 | artom | sean-k-mooney, I'm obviously missing context here, but do we need that patch at all right now? | |
| 19:04:45 | sean-k-mooney | artom: yep | |
| 19:04:46 | artom | As in, why not continue to add it implicitly when we detect a VDPA device? | |
| 19:05:00 | artom | Sorry, not "continue to add", but just "add implicitly" | |
| 19:05:01 | sean-k-mooney | libvirt does not do it right now | |
| 19:05:15 | artom | Like you're saying we do for SEV and realtime | |
| 19:05:18 | sean-k-mooney | and the way libvirt currently does it is a big problem for us | |
| 19:05:23 | artom | Ah, so Nova doesn't do it, *libvirt* does it | |
| 19:05:28 | sean-k-mooney | yep | |
| 19:05:29 | artom | (For SEV and realtime) | |
| 19:05:40 | sean-k-mooney | for sev and realtime nova does it | |
| 19:05:45 | sean-k-mooney | but that is slightly differnt | |
| 19:05:50 | artom | Tbh, I think having to set an extra spec that you have no choice for is bad UX, no? | |
| 19:06:11 | artom | What's preventing Nova from detecting VDPA devices and adding the required XML? | |
| 19:06:20 | sean-k-mooney | am kind of but we shoudl not be changing the memory we sare using based on a neutorn port | |
| 19:06:42 | sean-k-mooney | artom: tl;dr our current memory tracking is really broken | |
| 19:06:44 | artom | Ah, because not all VDPA devices require it? | |
| 19:07:20 | sean-k-mooney | and vm that vfio(sriov port or pci passthough) vgpu or nvmeof device is being locked in memory by libvirt | |
| 19:07:32 | sean-k-mooney | meaning oversubcript does not work | |
| 19:08:07 | sean-k-mooney | artom: am the simulator does not require it | |
| 19:08:20 | sean-k-mooney | artom: its not clear if dpdk based vdpa devices would | |
| 19:09:36 | sean-k-mooney | so i dont know maybe we could auto add it | |
| 19:11:12 | artom | You're saying "let's not change the memory based on Neutron port", except libvirt kinda does that already for vGPU, to use your own example | |
| 19:11:19 | sean-k-mooney | my conern right now is to track locked memory proertly we might need severly restrict what type of vms can use neutron sriov port or passthough of pci or vgpus deivces | |
| 19:11:39 | sean-k-mooney | artom: well it does it for neutron VF ports | |
| 19:11:58 | sean-k-mooney | so we are already inadvertely doing it based on ports | |
| 19:12:11 | artom | Yeah | |
| 19:12:24 | sean-k-mooney | well libvirt is | |
| 19:12:29 | artom | UX-wise, if something needs doing regardless, we should be asking the user to do it for us | |
| 19:12:54 | artom | The fact that our memory tracking is broken is a tangential problem :P | |