| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-04 | |||
| 14:18:10 | lyarwood | we've got some workarounds of removing the heartbeat timeout (-1) between libvirtd's and/or updating libvirt to >= 6.2.0 but I'd still like to address the underlying issue | |
| 14:18:26 | lyarwood | that looking at the tempest test I think might actually be there and not within Nova | |
| 14:18:48 | gmann | +1, done | |
| 14:19:20 | gibi | needing 240 sec to destroy a VM is definitely interesting | |
| 14:36:35 | tridde | So, traits... Consider a instance spun up from an image and flavor that at the time has no traits set. Later on a new CUSTOM trait is implemented, for example for compute nodes with local fast storage. That trait is set as required on some new flavors and as forbidden on all "normal" flavors including the flavor the mentioned instance was spun up from. | |
| 14:38:47 | tridde | Later the compute node where the instance is located breaks and is evacuated. Due to the fact (if I read the code correctly) that the forbidden trait was not set in the original request_spec as it was not in the flavor at the time, it doesn't provide placement with the forbidden trait when it's to be evacuated and it might very well end up on one of those local fast storage nodes. | |
| 14:39:11 | tridde | Would you consider this a bug, or rather the expected behaviour? | |
| 14:40:22 | sean-k-mooney | tridde: that is expected behavior | |
| 14:40:56 | sean-k-mooney | tridde: we make a copy of the image properties and flavor extra specs including the traits when teh instance is used | |
| 14:41:00 | sean-k-mooney | *created | |
| 14:41:17 | sean-k-mooney | and intentionally do not allow change made to flavors and images to affect exising instnaces | |
| 14:42:38 | sean-k-mooney | tridde: this general topic has come up before but basically image properties and flavor extra specs shoudl operationally be treaded as read only | |
| 14:42:51 | sean-k-mooney | once a vm has been created that uses the flavor or image | |
| 14:43:34 | sean-k-mooney | the intended way to deal with this is to create a v2 version fo the flaovr and phase out the use of the old flavor | |
| 14:43:53 | tridde | sean-k-mooney: Yeah, that's what I figured. It kind of makes sense, but at the same time makes traits somewhat less useful for steering workloads to the correct nodes. | |
| 14:44:01 | sean-k-mooney | tridde: i have proght up the idea of allowing resize to same falvor to just update the extra specs and move the vm | |
| 14:44:30 | sean-k-mooney | tridde: yes and no | |
| 14:44:48 | sean-k-mooney | you are using them correctly however you need to do more forward lookign planing when usign them | |
| 14:48:18 | tridde | Yeah, I guess the issue is when you have a long running openstack setup, with very different workloads, some instances that are very long running and some that are not. When the setup was originally done with (Liberty or Kilo IIRC), traits didn't even exist and now it's on Victoria. And I am sure there are instances still left from Liberty/Kilo time... | |
| 14:48:19 | sean-k-mooney | tridde: can i assume you would like to see the ablity to update exising instnace with new flavor/image definition and or get a list of instnace that do not match | |
| 14:49:19 | tridde | sean-k-mooney: Yeah, that would make sense and sounds like a good balance between the current design while still adding some more flexibility. | |
| 14:49:35 | sean-k-mooney | in generally ^ has not been accpeted in the past but its a common pain point so it is something we are thinking about but have not a solution for yet | |
| 14:50:31 | sean-k-mooney | tridde: link 613 is the most recent conversation on this topic https://etherpad.opendev.org/p/nova-wallaby-ptg | |
| 14:51:02 | sean-k-mooney | tridde: i did not have time to work on any of the itmes in the agreed section since then | |
| 14:52:41 | tridde | sean-k-mooney: Do I read correctly that recreate --flavor would not require a restart of the instance, rather just a live migration? | |
| 14:53:22 | sean-k-mooney | no it would recreate would be a resize in that case | |
| 14:54:01 | sean-k-mooney | i could see an enhancement later that if it was just a traits change perhaps it could live migrate | |
| 14:54:21 | sean-k-mooney | but in general that woudl not work for arbiatry extra specs | |
| 14:55:51 | sean-k-mooney | this is not currently resouce downstream for me or other in redhat to work on but it is something i would liek to do eventually | |
| 15:03:37 | tridde | Ok. In our case it's a pain with lots of customer contact any time we are to reboot a customer instance in a planned fashion, so it would be very desireable not to have to do that every time we implement a new trait... | |
| 16:57:06 | lyarwood | https://bugs.launchpad.net/nova/+bug/1945983/comments/2 - lyarwood-- hopefully this is easy to resolve | |
| 17:12:15 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM - Test devstack fix for bug #1945983 https://review.opendev.org/c/openstack/nova/+/812392 | |
| 21:23:55 | _jralbert | We recently upgraded from Train to Ussuri; since then, our compute nodes which rely on libvirt and vfio mdev to provide access to vGPU slices of physical GPUs can no longer start nova-compute after a reboot. The nova-compute code around recreating assigned mdevs appears to have changed fairly significantly between these releases, and I'm wondering if anyone here knows the details to chat about them before I get into creating a bug repor | |
| #openstack-nova - 2021-10-05 | |||
| 06:45:45 | gibi | _jralbert: you need bauzas to wake up (he is in EU timezone) | |
| 07:17:08 | bauzas | good morning Nova | |
| 07:17:38 | bauzas | gibi, _jralbert: what's up ? | |
| 07:17:58 | bauzas | oh the reboot case | |
| 07:18:22 | bauzas | there is an open bug I need to correctly care https://bugs.launchpad.net/nova/+bug/1900800 | |
| 08:37:37 | gibi | bauzas: I knew that you know more | |
| 08:38:33 | bauzas | gibi: I know you knew I know ;) | |
| 08:38:53 | gibi | :) | |
| 08:55:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM Trigger tests with new eventlet and dnspython https://review.opendev.org/c/openstack/nova/+/812456 | |
| 09:46:56 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Move live migration jobs back to voting https://review.opendev.org/c/openstack/nova/+/812392 | |
| 09:46:56 | opendevreview | Lee Yarwood proposed openstack/nova master: Revert "zuul: Skip block migration with attached volumes tests due to bug #1931702" https://review.opendev.org/c/openstack/nova/+/812473 | |
| 09:47:10 | lyarwood | ^ okay this is looking good now | |
| 09:47:20 | lyarwood | I can't believe I didn't see this earlier but there we go | |
| 09:47:50 | gibi | better later than never ;) | |
| 10:28:41 | bauzas | lyarwood: I promise I'll look at your changes | |
| 10:34:04 | bauzas | lyarwood: actually, I had a comment for the devstack change https://review.opendev.org/c/openstack/devstack/+/812391/4/lib/nova#302 | |
| 10:36:33 | lyarwood | done | |
| 11:24:49 | lyarwood | actually I'm going to drop for a while this afternoon, could use the air ahead of being locked down until surgery next Tuesday, back online this evening UK time | |
| 11:25:03 | lyarwood | if anything comes up with the initiator change feel free to respin before I'm back | |
| 13:12:51 | kashyap | gibi: For later: Does chengsheng's response on this help any bit? -- https://review.opendev.org/c/openstack/nova/+/762330/ | |
| 13:13:34 | kashyap | I'd like to make some time to get this to completion. The lack of these two newer and better APIs is causing subtle live migration bugs :-( | |
| 13:17:53 | gibi | kashyap: added to my queue but wont reach it today | |
| 13:18:02 | kashyap | Does not have to be today :) | |
| 13:19:05 | gibi | kashyap: but thank you for the ping, I already forgot about his patch | |
| 13:53:42 | bauzas | artom_: sean-k-mooney: looks to me some SRIOV live migration issue https://bugs.launchpad.net/nova/+bug/1944619 | |
| 13:54:56 | sean-k-mooney | ill take a look but this is technially hardwoar offloade ovs if they are usign switch dev | |
| 13:55:10 | artom_ | bauzas, hrmm, looks we're not cleaning up properly in some cases | |
| 13:55:11 | sean-k-mooney | currently we do not support live migration with vdpa | |
| 13:55:23 | artom_ | Though I don't like "provoke an error in pre_live_migration" | |
| 13:55:26 | bauzas | artom_: can we triage this one ? | |
| 13:55:32 | sean-k-mooney | so if they are usign the sriov nic agent they may not use switchdev mode | |
| 13:55:48 | sean-k-mooney | if they are using switchdev mode then they have to use ovs hardware offload instead | |
| 13:58:07 | artom | bauzas, it'd be cool if they could upload full nova compute logs that include the error, ideally without the failure being "provoked" manually | |
| 13:59:33 | bauzas | they gave us some pastebin https://paste.ubuntu.com/p/ThQmDYtdSS/ | |
| 13:59:36 | bauzas | artom: ^ | |
| 14:02:29 | artom | bauzas, yeah, I'm not a fan of them triggering the failure by manually creating the destination instance disk | |
| 14:02:47 | artom | But I suppose it *could* happen organically, so Nova needs to handle it correctly? | |
| 14:03:51 | bauzas | artom: here, I don't really want to discuss about the bug, but rather about its upstream triage | |
| 14:04:29 | bauzas | artom: if you think they need to give us some other verification, could you ask it in the bug report and punt it to the 'Incomplete' status ? | |
| 14:05:11 | artom | bauzas, valid, I suppose, then | |
| 14:05:31 | bauzas | artom: ack, could you then triage it ? | |
| 14:05:34 | artom | Tbh, I don't know the gritty hardware details to understand what sean-k-mooney was saying about various levels of support in variou backends | |
| 14:06:55 | sean-k-mooney | im currently responding to it to level set what shoudl be supproted and what is not ssupproted | |
| 14:06:57 | sean-k-mooney | Check the logs for: libvirt.libvirtError: Requested operation is not valid: PCI device 0000:03:04.1 is in use by driver QEMU, domain instance-00000001 | |
| 14:07:14 | sean-k-mooney | ^ that seams to be the error they are reporting that breaks the roolback | |
| 14:07:20 | sean-k-mooney | that is not related to sriov live migration | |
| 14:07:36 | sean-k-mooney | that is cause by previou incorrect unshleves and possible cold migrations | |
| 14:08:06 | sean-k-mooney | bauzas: artom is aredally working on a fix for the in use device issue | |
| 14:08:17 | bauzas | kk | |
| 14:08:36 | artom | The unshelve fix? It merged in master, currently sitting on Victoria in the backport train | |
| 14:09:06 | artom | Depending on lyarwood's "controversial" helper backport here: https://review.opendev.org/c/openstack/nova/+/791481/1 | |
| 14:09:13 | sean-k-mooney | artom: that is part of it yes | |
| 14:09:18 | sean-k-mooney | the unsleve fix | |
| 14:09:48 | sean-k-mooney | although gibi also has another upstream bug for it which i am looking for currently | |
| 14:14:29 | sean-k-mooney | oh actully no that was a different issue | |
| 14:14:53 | sean-k-mooney | so the in use port i am pretty sure was cause by the unshelve of a vm previously | |
| 14:15:04 | sean-k-mooney | its not related to the sriov live migration | |
| 14:29:23 | sean-k-mooney | bauzas: artom im 99% sure this is hardware offload ovs not standard sriov by the way | |
| 14:38:53 | artom | sean-k-mooney, what's the difference? | |
| 15:03:15 | sean-k-mooney | artom: they have very different code paths | |
| 15:03:49 | sean-k-mooney | artom: for standard sriov all the interface configuretion is doen by libvirt | |
| 15:03:51 | opendevreview | Merged openstack/nova master: nova-manage: Ensure mountpoint is passed when updating attachment https://review.opendev.org/c/openstack/nova/+/811713 | |
| 15:04:34 | sean-k-mooney | for hardware offload ovs os-vif need to create ovs ports and add representor netdevs to ovs and the neutron l2 agent or ovn need to configure ovs to do thinks like vlan tagging | |
| 15:05:04 | sean-k-mooney | so the neutron contol path gose via ovn or the l2 agent not the sriov nic agent | |
| 15:05:17 | sean-k-mooney | the nic is in swtichdev mode not in legacy | |
| 15:06:18 | sean-k-mooney | vlan enfromanet is done via ovs not libvirt and you can use geneve/vxlan networks not just flat/vlan | |
| 15:11:34 | sean-k-mooney | there are some other difference too but those are the main ones | |