| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-04 | |||
| 10:27:09 | sean-k-mooney | must have missed that | |
| 10:29:34 | gibi | this could be a global conspiracy | |
| 10:29:48 | gibi | sean-k-mooney: probably you know something too important | |
| 10:31:10 | sean-k-mooney | ya like dbus sucks org.freedesktop.DBus.Error.ServiceUnknown | |
| 10:33:59 | gibi | ohh they infected your dbus too, they are really after you :] | |
| 10:35:31 | sean-k-mooney | Process 6662 (evolution.bin) of user 1000 dumped core. maybe i should just reboot that will totally fix it right | |
| 10:36:10 | sean-k-mooney | althouhg secretly i think that also likely tthe problem i rebooted my laptop for the first time in 4+ months | |
| 10:37:58 | sean-k-mooney | im going to update evertyting installed via flatpack im guessing the gnome runtime is outdated and i am only now seeing that after the reboot | |
| 10:39:49 | gibi | I always imagine dumping core like https://www.youtube.com/watch?v=xIuUEwat6aM | |
| 10:40:02 | sean-k-mooney | nice | |
| 10:45:04 | lyarwood | risky click of the day | |
| 10:45:52 | gibi | I promise it is safe | |
| 10:48:34 | sean-k-mooney | and reboot did not work so backup and reinstall the os it is just not today | |
| 10:49:16 | sean-k-mooney | im pretty sure the flatpack version of evolution is not impoatiable with the dbus form fedora 32 | |
| 10:49:36 | sean-k-mooney | given fedora 32 is also now unsupported its as good a time to change as any | |
| 12:17:35 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Remove trunk live migration testing https://review.opendev.org/c/openstack/nova/+/812266 | |
| 13:00:18 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM: Test libvirtd with increased keepalive_interval https://review.opendev.org/c/openstack/nova/+/798069 | |
| 13:08:53 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM: Test nova-live-migration with openstack-two-node-centos-8-stream https://review.opendev.org/c/openstack/nova/+/812351 | |
| 13:11:35 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM: Test nova-live-migration with openstack-two-node-centos-8-stream https://review.opendev.org/c/openstack/nova/+/812351 | |
| 13:17:03 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM: Test nova-live-migration with openstack-two-node-centos-8-stream https://review.opendev.org/c/openstack/nova/+/812351 | |
| 13:30:30 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Remove trunk live migration testing https://review.opendev.org/c/openstack/nova/+/812266 | |
| 13:35:30 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Remove trunk live migration testing https://review.opendev.org/c/openstack/nova/+/812266 | |
| 14:15:46 | gmann | lyarwood: gibi : I did not get backport things. you mean these one? enabling trunk on stable to unhide the bug ? https://review.opendev.org/q/topic:%22bug%252F1945346%22+(status:open%20OR%20status:merged) | |
| 14:16:10 | gmann | and once bug is resolved we keep enabling it from nova side | |
| 14:16:39 | lyarwood | yeah don't enable it on stable within the grenade jobs for now | |
| 14:16:50 | gibi | yepp | |
| 14:16:58 | gmann | lyarwood: ok, let me put them in WIP | |
| 14:17:04 | gibi | lyarwood: just proposed disabling trunk testing on mater | |
| 14:17:13 | gibi | lyarwood just proposed disabling trunk testing on master | |
| 14:17:37 | lyarwood | yeah alongside live migration tests while we work out why it's causing instance removal to take so long | |
| 14:18:10 | lyarwood | we've got some workarounds of removing the heartbeat timeout (-1) between libvirtd's and/or updating libvirt to >= 6.2.0 but I'd still like to address the underlying issue | |
| 14:18:26 | lyarwood | that looking at the tempest test I think might actually be there and not within Nova | |
| 14:18:48 | gmann | +1, done | |
| 14:19:20 | gibi | needing 240 sec to destroy a VM is definitely interesting | |
| 14:36:35 | tridde | So, traits... Consider a instance spun up from an image and flavor that at the time has no traits set. Later on a new CUSTOM trait is implemented, for example for compute nodes with local fast storage. That trait is set as required on some new flavors and as forbidden on all "normal" flavors including the flavor the mentioned instance was spun up from. | |
| 14:38:47 | tridde | Later the compute node where the instance is located breaks and is evacuated. Due to the fact (if I read the code correctly) that the forbidden trait was not set in the original request_spec as it was not in the flavor at the time, it doesn't provide placement with the forbidden trait when it's to be evacuated and it might very well end up on one of those local fast storage nodes. | |
| 14:39:11 | tridde | Would you consider this a bug, or rather the expected behaviour? | |
| 14:40:22 | sean-k-mooney | tridde: that is expected behavior | |
| 14:40:56 | sean-k-mooney | tridde: we make a copy of the image properties and flavor extra specs including the traits when teh instance is used | |
| 14:41:00 | sean-k-mooney | *created | |
| 14:41:17 | sean-k-mooney | and intentionally do not allow change made to flavors and images to affect exising instnaces | |
| 14:42:38 | sean-k-mooney | tridde: this general topic has come up before but basically image properties and flavor extra specs shoudl operationally be treaded as read only | |
| 14:42:51 | sean-k-mooney | once a vm has been created that uses the flavor or image | |
| 14:43:34 | sean-k-mooney | the intended way to deal with this is to create a v2 version fo the flaovr and phase out the use of the old flavor | |
| 14:43:53 | tridde | sean-k-mooney: Yeah, that's what I figured. It kind of makes sense, but at the same time makes traits somewhat less useful for steering workloads to the correct nodes. | |
| 14:44:01 | sean-k-mooney | tridde: i have proght up the idea of allowing resize to same falvor to just update the extra specs and move the vm | |
| 14:44:30 | sean-k-mooney | tridde: yes and no | |
| 14:44:48 | sean-k-mooney | you are using them correctly however you need to do more forward lookign planing when usign them | |
| 14:48:18 | tridde | Yeah, I guess the issue is when you have a long running openstack setup, with very different workloads, some instances that are very long running and some that are not. When the setup was originally done with (Liberty or Kilo IIRC), traits didn't even exist and now it's on Victoria. And I am sure there are instances still left from Liberty/Kilo time... | |
| 14:48:19 | sean-k-mooney | tridde: can i assume you would like to see the ablity to update exising instnace with new flavor/image definition and or get a list of instnace that do not match | |
| 14:49:19 | tridde | sean-k-mooney: Yeah, that would make sense and sounds like a good balance between the current design while still adding some more flexibility. | |
| 14:49:35 | sean-k-mooney | in generally ^ has not been accpeted in the past but its a common pain point so it is something we are thinking about but have not a solution for yet | |
| 14:50:31 | sean-k-mooney | tridde: link 613 is the most recent conversation on this topic https://etherpad.opendev.org/p/nova-wallaby-ptg | |
| 14:51:02 | sean-k-mooney | tridde: i did not have time to work on any of the itmes in the agreed section since then | |
| 14:52:41 | tridde | sean-k-mooney: Do I read correctly that recreate --flavor would not require a restart of the instance, rather just a live migration? | |
| 14:53:22 | sean-k-mooney | no it would recreate would be a resize in that case | |
| 14:54:01 | sean-k-mooney | i could see an enhancement later that if it was just a traits change perhaps it could live migrate | |
| 14:54:21 | sean-k-mooney | but in general that woudl not work for arbiatry extra specs | |
| 14:55:51 | sean-k-mooney | this is not currently resouce downstream for me or other in redhat to work on but it is something i would liek to do eventually | |
| 15:03:37 | tridde | Ok. In our case it's a pain with lots of customer contact any time we are to reboot a customer instance in a planned fashion, so it would be very desireable not to have to do that every time we implement a new trait... | |
| 16:57:06 | lyarwood | https://bugs.launchpad.net/nova/+bug/1945983/comments/2 - lyarwood-- hopefully this is easy to resolve | |
| 17:12:15 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM - Test devstack fix for bug #1945983 https://review.opendev.org/c/openstack/nova/+/812392 | |
| 21:23:55 | _jralbert | We recently upgraded from Train to Ussuri; since then, our compute nodes which rely on libvirt and vfio mdev to provide access to vGPU slices of physical GPUs can no longer start nova-compute after a reboot. The nova-compute code around recreating assigned mdevs appears to have changed fairly significantly between these releases, and I'm wondering if anyone here knows the details to chat about them before I get into creating a bug repor | |
| #openstack-nova - 2021-10-05 | |||
| 06:45:45 | gibi | _jralbert: you need bauzas to wake up (he is in EU timezone) | |
| 07:17:08 | bauzas | good morning Nova | |
| 07:17:38 | bauzas | gibi, _jralbert: what's up ? | |
| 07:17:58 | bauzas | oh the reboot case | |
| 07:18:22 | bauzas | there is an open bug I need to correctly care https://bugs.launchpad.net/nova/+bug/1900800 | |
| 08:37:37 | gibi | bauzas: I knew that you know more | |
| 08:38:33 | bauzas | gibi: I know you knew I know ;) | |
| 08:38:53 | gibi | :) | |
| 08:55:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM Trigger tests with new eventlet and dnspython https://review.opendev.org/c/openstack/nova/+/812456 | |
| 09:46:56 | opendevreview | Lee Yarwood proposed openstack/nova master: Revert "zuul: Skip block migration with attached volumes tests due to bug #1931702" https://review.opendev.org/c/openstack/nova/+/812473 | |
| 09:46:56 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Move live migration jobs back to voting https://review.opendev.org/c/openstack/nova/+/812392 | |
| 09:47:10 | lyarwood | ^ okay this is looking good now | |
| 09:47:20 | lyarwood | I can't believe I didn't see this earlier but there we go | |
| 09:47:50 | gibi | better later than never ;) | |
| 10:28:41 | bauzas | lyarwood: I promise I'll look at your changes | |
| 10:34:04 | bauzas | lyarwood: actually, I had a comment for the devstack change https://review.opendev.org/c/openstack/devstack/+/812391/4/lib/nova#302 | |
| 10:36:33 | lyarwood | done | |
| 11:24:49 | lyarwood | actually I'm going to drop for a while this afternoon, could use the air ahead of being locked down until surgery next Tuesday, back online this evening UK time | |
| 11:25:03 | lyarwood | if anything comes up with the initiator change feel free to respin before I'm back | |
| 13:12:51 | kashyap | gibi: For later: Does chengsheng's response on this help any bit? -- https://review.opendev.org/c/openstack/nova/+/762330/ | |
| 13:13:34 | kashyap | I'd like to make some time to get this to completion. The lack of these two newer and better APIs is causing subtle live migration bugs :-( | |
| 13:17:53 | gibi | kashyap: added to my queue but wont reach it today | |
| 13:18:02 | kashyap | Does not have to be today :) | |
| 13:19:05 | gibi | kashyap: but thank you for the ping, I already forgot about his patch | |
| 13:53:42 | bauzas | artom_: sean-k-mooney: looks to me some SRIOV live migration issue https://bugs.launchpad.net/nova/+bug/1944619 | |
| 13:54:56 | sean-k-mooney | ill take a look but this is technially hardwoar offloade ovs if they are usign switch dev | |
| 13:55:10 | artom_ | bauzas, hrmm, looks we're not cleaning up properly in some cases | |
| 13:55:11 | sean-k-mooney | currently we do not support live migration with vdpa | |
| 13:55:23 | artom_ | Though I don't like "provoke an error in pre_live_migration" | |
| 13:55:26 | bauzas | artom_: can we triage this one ? | |
| 13:55:32 | sean-k-mooney | so if they are usign the sriov nic agent they may not use switchdev mode | |
| 13:55:48 | sean-k-mooney | if they are using switchdev mode then they have to use ovs hardware offload instead | |
| 13:58:07 | artom | bauzas, it'd be cool if they could upload full nova compute logs that include the error, ideally without the failure being "provoked" manually | |
| 13:59:33 | bauzas | they gave us some pastebin https://paste.ubuntu.com/p/ThQmDYtdSS/ | |
| 13:59:36 | bauzas | artom: ^ | |
| 14:02:29 | artom | bauzas, yeah, I'm not a fan of them triggering the failure by manually creating the destination instance disk | |
| 14:02:47 | artom | But I suppose it *could* happen organically, so Nova needs to handle it correctly? | |