| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-26 | |||
| 11:11:04 | sean-k-mooney[m] | ack | |
| 11:11:17 | sean-k-mooney[m] | i have not reviewd that patch closely by the way | |
| 11:13:31 | sean-k-mooney[m] | at the end of the day i still think we should be tracking the mdevs in novas db and recreating form the db record instead of trying to use the guest xml to store the mdevs used | |
| 11:13:38 | opendevreview | Merged openstack/nova master: Updating tests with Yoga testing runtime https://review.opendev.org/c/openstack/nova/+/819194 | |
| 11:14:19 | sean-k-mooney[m] | e.g. right now we are relying on the domain to exist to prevent other vms from using the mdevs which is fragile | |
| 11:16:15 | sean-k-mooney[m] | at least in comparison to storing them in the resoucs table or pci devices table. it has the advantage of being local i guess but it make reaning about this harder since we dont track the arent device ectra in other edge cases like host reboot | |
| 11:16:40 | sean-k-mooney[m] | so i hope we can make this simpler and just recored it in the db eventurally | |
| 11:18:27 | lyarwood | Yeah agreed, I think for now this is a valid backportable fix but really it needs to be replaced by something from our DB or placement or something | |
| 11:19:11 | lyarwood | It's just super confusing given we detach from the active and then use the persistent domain to work out what we need to reattach before we then restore the managedSave state of the domain | |
| 11:20:17 | lyarwood | I guess an alternative is for libvirt to support managedSave somehow for mdevs, likely by the caller telling it not to care about their state or something? | |
| 11:20:44 | sean-k-mooney[m] | so i think it dose now right ? | |
| 11:21:14 | sean-k-mooney[m] | oh no | |
| 11:21:40 | sean-k-mooney[m] | what has changed is we blocked suspend before becaue we could not unplug and replug the mdev | |
| 11:21:44 | sean-k-mooney[m] | now we can | |
| 11:23:41 | sean-k-mooney[m] | oh there is another nasty bug there | |
| 11:24:02 | sean-k-mooney[m] | so we previously detach the interface on suspend right | |
| 11:24:49 | sean-k-mooney[m] | now we are reattaching them when we did not before which is good | |
| 11:25:07 | sean-k-mooney[m] | but we sometimes suspend the guest when we do snapshot | |
| 11:25:36 | sean-k-mooney[m] | so as a side effect if you could not use live snap shots snapshotting would remove the mdev and not reattach them | |
| 11:25:51 | sean-k-mooney[m] | so this will fix that too | |
| 11:26:32 | opendevreview | Lee Yarwood proposed openstack/nova master: fup: Require python >= 3.8 from Yoga https://review.opendev.org/c/openstack/nova/+/819415 | |
| 11:27:27 | lyarwood | do we use suspend for that? | |
| 11:27:36 | lyarwood | I thought that was a pause on the domain itself | |
| 11:28:02 | lyarwood | ah it's suspend | |
| 11:28:03 | lyarwood | TIL | |
| 11:28:53 | lyarwood | okay yeah resume needs to be updated | |
| 11:29:02 | sean-k-mooney[m] | ya i tought it would be pause too untile i read it again just now. | |
| 11:29:06 | lyarwood | https://github.com/openstack/nova/blob/d630615a02469442fb50ed4aa7e092206a28166a/nova/virt/libvirt/driver.py#L3043-L3063 that even | |
| 11:29:10 | lyarwood | nice catch | |
| 11:29:42 | lyarwood | tbh the TODO from stephenfin is right, that could just be a single call to self.resume | |
| 11:30:07 | sean-k-mooney[m] | actully i tought that just called resume like the suppoed_for_snappshot calls suspend | |
| 11:30:22 | sean-k-mooney[m] | ya | |
| 11:30:41 | sean-k-mooney[m] | it should be. | |
| 11:38:57 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM Testing python >= 3.8 requirement https://review.opendev.org/c/openstack/nova/+/819432 | |
| 11:50:49 | sean-k-mooney | lyarwood:... unfortunetly we might need to keey requires_python>=3.6 for rdo this cycle due to upgrdes | |
| 11:51:12 | sean-k-mooney | which makes me think we cant actully turn off testin 3.6 until z | |
| 11:51:21 | lyarwood | Why would we need to keep it? | |
| 11:51:45 | lyarwood | Can't we just ship py38 in the Yoga containers? | |
| 11:51:45 | sean-k-mooney | because they will be supprot yoga on centos 8 and 9 for upgrades | |
| 11:52:18 | sean-k-mooney | we can but they ahve to supprot rpm install. and to be fair 3.8 is avaiable on cents too | |
| 11:52:27 | sean-k-mooney | * centos8 too | |
| 11:52:31 | sean-k-mooney | its just not the default | |
| 11:53:08 | lyarwood | Sigh why didn't the RDO folks raise that with the TC before they moved the runtimes for Yoga | |
| 11:53:17 | lyarwood | brb | |
| 11:54:42 | sean-k-mooney | ya i dont know i really dont like the idea of our testing and min requires being differnet so i dotn know what to do to be honest | |
| 11:55:09 | lyarwood | Well for RDO at least they can workaround it easily enough | |
| 11:55:21 | lyarwood | by installing py38 alongside py36 | |
| 11:55:53 | gibi | lyarwood, bauzas: re py36 support: sorry I also missed that we are discussing that on an older PS. :/ | |
| 11:56:07 | gibi | lyarwood bauzas: I agree to land the fup on setup.cfg | |
| 11:56:17 | lyarwood | np my language here didn't help, I confused things massively | |
| 12:03:58 | gibi | and I do believ that RDO missed the deadline not to raise the issue on the TC ruling | |
| 12:09:50 | sean-k-mooney | im also kind of confused why RDO have not been testign with 3.8 as well | |
| 12:10:11 | sean-k-mooney | it has been a testing requirement for some time | |
| 12:10:38 | sean-k-mooney | i guess they were usign the lower requiremtn for centos of 3.6 | |
| 12:12:02 | sean-k-mooney | but give our downstream will be using 3.9 form wallaby+ that feels like a gap someone form redhat would have been woking on fixing | |
| 12:13:47 | sean-k-mooney | py 3.9.8 currrenlty py 3.8 will not be supproted on centos 9 | |
| 12:20:28 | lyarwood | sorry back | |
| 12:20:41 | lyarwood | just noticed I missed stephenfin 's thread about this on the ML | |
| 12:30:06 | gibi | thread even | |
| 14:19:29 | artom | Huh, looks like at some point between... victoria? and master ovs+hybrid_plug became plug-time Neutron events... | |
| 14:20:55 | artom | https://zuul.opendev.org/t/openstack/build/2f75f41da4df48f8a8abc5087fd29efb/log/controller/logs/screen-n-cpu.txt#9233 | |
| 14:23:18 | tobias-urdin | lyarwood: should we ping in somebody from RDO on the above topic or did u take it already? | |
| 14:23:51 | lyarwood | tobias-urdin: Looks like folks from RDO already raised this on the ML, I hadn't seen it until the end of the discussion earlier | |
| 14:30:52 | tobias-urdin | lyarwood: ack, good to hear | |
| 14:36:09 | sean-k-mooney | artom: ovs hybrid plug was always a plugtime event | |
| 14:36:21 | artom | sean-k-mooney, not for revert resize | |
| 14:36:33 | artom | Remember 7a7a223602ca5aa0aca8f65a6ab143f1d8f8ec1b? | |
| 14:36:34 | sean-k-mooney | for revert its already plugged | |
| 14:36:39 | artom | Looks like we don't need it anymore | |
| 14:36:40 | sean-k-mooney | so its bind time | |
| 14:37:12 | artom | Apparently that's no longer the case, see the log link | |
| 14:37:23 | artom | It's from https://review.opendev.org/c/openstack/nova/+/819349 | |
| 14:38:22 | sean-k-mooney | for resize we dont touch the netwroking on the souce host until you confrim | |
| 14:38:43 | sean-k-mooney | we do update the port binding (there is only one) before we go to resize verify | |
| 14:38:53 | sean-k-mooney | then on revert we update the host back to the soruce | |
| 14:38:56 | artom | Well I have no idea what happened, but we now get plug-time events, and not bind-time events | |
| 14:39:08 | sean-k-mooney | but we should not need to set up the networking on the host again | |
| 14:39:23 | artom | I mean, I don't disagree | |
| 14:39:44 | artom | But based on what my CI job run is telling me, revert resize is now plug-time like everything else (again) | |
| 14:39:56 | sean-k-mooney | we are going to get plug time events as well i think | |
| 14:40:03 | sean-k-mooney | but we should get bind time events first | |
| 14:40:12 | artom | Never get the former | |
| 14:40:22 | sean-k-mooney | we will call effectivly hard reboot as part of starting the instance again on the source | |
| 14:40:29 | sean-k-mooney | that is where the plug time event comes form | |
| 14:40:46 | sean-k-mooney | at least that is what im guessing | |
| 14:41:01 | sean-k-mooney | have not looked at that code regently how are we starting the souce vm | |
| 14:42:04 | artom | Tbh I'm sick of the whole thing, want to get CI coverage on it so when it changes again under us we at least notice it, and revert 7a7a223602ca5aa0aca8f65a6ab143f1d8f8ec1b because we apparently don't need it anymore | |
| 14:42:51 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11238-L11240 | |
| 14:43:33 | sean-k-mooney | artom: im not sure we should | |
| 14:43:37 | sean-k-mooney | revert 7a7a223602ca5aa0aca8f65a6ab143f1d8f8ec1b | |
| 14:43:46 | sean-k-mooney | to me this is a neutron regresssion | |
| 14:44:30 | sean-k-mooney | its a bug that it is not sendign a event at bind time when we change the host_id | |
| 14:48:00 | artom | sean-k-mooney, hellz yeah, less work for us | |
| 14:48:09 | artom | :P | |
| 14:48:10 | sean-k-mooney | artom: so ya https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7262 we call plug vifs again when starting the ports on the source but that is idempotent and will only update the network backend (ovs) if the ports are not present | |
| 14:49:49 | sean-k-mooney | artom: i suspect the issue is the l2 agent recice the port update event but does not set the port status as up because its already up and therefor does not trigger the event | |
| 14:52:25 | artom | sean-k-mooney, huh, so I can actually hand this over to #neutron via the downstream CIX call, as originally it was a CIX | |
| 14:54:30 | sean-k-mooney | it might be related to this logic https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L2696-L2699 | |
| 14:54:52 | sean-k-mooney | that looks wrong to me as it will filter out host chagnes on the port | |
| 14:57:25 | artom | sean-k-mooney, we can try a quick DNM change to verify that... what should we change it to? | |