Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-04
09:17:38 lyarwood yup it's my rescheduled op thanks :)
09:18:27 bauzas :crossing_fingers_fake_emoji:
09:23:38 gibi lyarwood, gmann did we just enabled trunk testing in grenade too?
09:23:50 lyarwood We did yeah
09:23:59 gibi then we need to revert that due to https://launchpad.net/bugs/1912310
09:24:01 gibi as well
09:24:14 lyarwood yeah it's in the .zuul change
09:24:22 gibi ohh, then I missed that...
09:24:35 lyarwood just working on something else first but I'll write up a new bug for the trunk delete thing shortly
09:24:41 gibi lyarwood: thanksd
09:24:45 lyarwood just to make it easier for folks
09:24:48 gibi I see the grenade part now
09:25:01 gibi I guess we need to backport this zuul change to stable too
09:38:46 lyarwood Yeah I would think so
09:38:54 lyarwood until the underlying issue is resolved
10:04:38 sean-k-mooney well fedora has a unique way of dealing with a weeks worth of eamil while im on vaction.
10:05:12 sean-k-mooney evolation apparenly no longer has any of my email account set up
10:09:27 lyarwood A new welcome back to work feature
10:10:16 kashyap LOL
10:10:22 gibi :)
10:10:51 kashyap My personal happiness with email [if you can call it that] has shot up since I embraced Mutt some 6 years ago
10:11:07 kashyap It's a joy to use; and I can mark a mail thread with 70 emails as "read" in one keystroke.
10:18:31 sean-k-mooney this might be a google porblem
10:18:57 sean-k-mooney if i use minetime as a alternitive calandar app it als no longer has access to my calandar
10:19:24 sean-k-mooney ok that just started working
10:25:39 lyarwood did you not get the HR email?
10:25:44 lyarwood /s
10:26:09 gibi :D
10:26:47 lyarwood reminds me of when I joined RH and they managed to email my flight details for new hire to my redhat email address, the issue being that my first day was literally the first day of new hire in Munich.
10:27:01 sean-k-mooney that i abuse my eamial so badly the agree to mediate the dispute with google and terminate all my gmail account including my work and personal one
10:27:09 sean-k-mooney must have missed that
10:29:34 gibi this could be a global conspiracy
10:29:48 gibi sean-k-mooney: probably you know something too important
10:31:10 sean-k-mooney ya like dbus sucks org.freedesktop.DBus.Error.ServiceUnknown
10:33:59 gibi ohh they infected your dbus too, they are really after you :]
10:35:31 sean-k-mooney Process 6662 (evolution.bin) of user 1000 dumped core. maybe i should just reboot that will totally fix it right
10:36:10 sean-k-mooney althouhg secretly i think that also likely tthe problem i rebooted my laptop for the first time in 4+ months
10:37:58 sean-k-mooney im going to update evertyting installed via flatpack im guessing the gnome runtime is outdated and i am only now seeing that after the reboot
10:39:49 gibi I always imagine dumping core like https://www.youtube.com/watch?v=xIuUEwat6aM
10:40:02 sean-k-mooney nice
10:45:04 lyarwood risky click of the day
10:45:52 gibi I promise it is safe
10:48:34 sean-k-mooney and reboot did not work so backup and reinstall the os it is just not today
10:49:16 sean-k-mooney im pretty sure the flatpack version of evolution is not impoatiable with the dbus form fedora 32
10:49:36 sean-k-mooney given fedora 32 is also now unsupported its as good a time to change as any
12:17:35 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Remove trunk live migration testing https://review.opendev.org/c/openstack/nova/+/812266
13:00:18 opendevreview Lee Yarwood proposed openstack/nova master: DNM: Test libvirtd with increased keepalive_interval https://review.opendev.org/c/openstack/nova/+/798069
13:08:53 opendevreview Lee Yarwood proposed openstack/nova master: DNM: Test nova-live-migration with openstack-two-node-centos-8-stream https://review.opendev.org/c/openstack/nova/+/812351
13:11:35 opendevreview Lee Yarwood proposed openstack/nova master: DNM: Test nova-live-migration with openstack-two-node-centos-8-stream https://review.opendev.org/c/openstack/nova/+/812351
13:17:03 opendevreview Lee Yarwood proposed openstack/nova master: DNM: Test nova-live-migration with openstack-two-node-centos-8-stream https://review.opendev.org/c/openstack/nova/+/812351
13:30:30 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Remove trunk live migration testing https://review.opendev.org/c/openstack/nova/+/812266
13:35:30 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Remove trunk live migration testing https://review.opendev.org/c/openstack/nova/+/812266
14:15:46 gmann lyarwood: gibi : I did not get backport things. you mean these one? enabling trunk on stable to unhide the bug ? https://review.opendev.org/q/topic:%22bug%252F1945346%22+(status:open%20OR%20status:merged)
14:16:10 gmann and once bug is resolved we keep enabling it from nova side
14:16:39 lyarwood yeah don't enable it on stable within the grenade jobs for now
14:16:50 gibi yepp
14:16:58 gmann lyarwood: ok, let me put them in WIP
14:17:04 gibi lyarwood: just proposed disabling trunk testing on mater
14:17:13 gibi lyarwood just proposed disabling trunk testing on master
14:17:37 lyarwood yeah alongside live migration tests while we work out why it's causing instance removal to take so long
14:18:10 lyarwood we've got some workarounds of removing the heartbeat timeout (-1) between libvirtd's and/or updating libvirt to >= 6.2.0 but I'd still like to address the underlying issue
14:18:26 lyarwood that looking at the tempest test I think might actually be there and not within Nova
14:18:48 gmann +1, done
14:19:20 gibi needing 240 sec to destroy a VM is definitely interesting
14:36:35 tridde So, traits... Consider a instance spun up from an image and flavor that at the time has no traits set. Later on a new CUSTOM trait is implemented, for example for compute nodes with local fast storage. That trait is set as required on some new flavors and as forbidden on all "normal" flavors including the flavor the mentioned instance was spun up from.
14:38:47 tridde Later the compute node where the instance is located breaks and is evacuated. Due to the fact (if I read the code correctly) that the forbidden trait was not set in the original request_spec as it was not in the flavor at the time, it doesn't provide placement with the forbidden trait when it's to be evacuated and it might very well end up on one of those local fast storage nodes.
14:39:11 tridde Would you consider this a bug, or rather the expected behaviour?
14:40:22 sean-k-mooney tridde: that is expected behavior
14:40:56 sean-k-mooney tridde: we make a copy of the image properties and flavor extra specs including the traits when teh instance is used
14:41:00 sean-k-mooney *created
14:41:17 sean-k-mooney and intentionally do not allow change made to flavors and images to affect exising instnaces
14:42:38 sean-k-mooney tridde: this general topic has come up before but basically image properties and flavor extra specs shoudl operationally be treaded as read only
14:42:51 sean-k-mooney once a vm has been created that uses the flavor or image
14:43:34 sean-k-mooney the intended way to deal with this is to create a v2 version fo the flaovr and phase out the use of the old flavor
14:43:53 tridde sean-k-mooney: Yeah, that's what I figured. It kind of makes sense, but at the same time makes traits somewhat less useful for steering workloads to the correct nodes.
14:44:01 sean-k-mooney tridde: i have proght up the idea of allowing resize to same falvor to just update the extra specs and move the vm
14:44:30 sean-k-mooney tridde: yes and no
14:44:48 sean-k-mooney you are using them correctly however you need to do more forward lookign planing when usign them
14:48:18 tridde Yeah, I guess the issue is when you have a long running openstack setup, with very different workloads, some instances that are very long running and some that are not. When the setup was originally done with (Liberty or Kilo IIRC), traits didn't even exist and now it's on Victoria. And I am sure there are instances still left from Liberty/Kilo time...
14:48:19 sean-k-mooney tridde: can i assume you would like to see the ablity to update exising instnace with new flavor/image definition and or get a list of instnace that do not match
14:49:19 tridde sean-k-mooney: Yeah, that would make sense and sounds like a good balance between the current design while still adding some more flexibility.
14:49:35 sean-k-mooney in generally ^ has not been accpeted in the past but its a common pain point so it is something we are thinking about but have not a solution for yet
14:50:31 sean-k-mooney tridde: link 613 is the most recent conversation on this topic https://etherpad.opendev.org/p/nova-wallaby-ptg
14:51:02 sean-k-mooney tridde: i did not have time to work on any of the itmes in the agreed section since then
14:52:41 tridde sean-k-mooney: Do I read correctly that recreate --flavor would not require a restart of the instance, rather just a live migration?
14:53:22 sean-k-mooney no it would recreate would be a resize in that case
14:54:01 sean-k-mooney i could see an enhancement later that if it was just a traits change perhaps it could live migrate
14:54:21 sean-k-mooney but in general that woudl not work for arbiatry extra specs
14:55:51 sean-k-mooney this is not currently resouce downstream for me or other in redhat to work on but it is something i would liek to do eventually
15:03:37 tridde Ok. In our case it's a pain with lots of customer contact any time we are to reboot a customer instance in a planned fashion, so it would be very desireable not to have to do that every time we implement a new trait...
16:57:06 lyarwood https://bugs.launchpad.net/nova/+bug/1945983/comments/2 - lyarwood-- hopefully this is easy to resolve
17:12:15 opendevreview Lee Yarwood proposed openstack/nova master: DNM - Test devstack fix for bug #1945983 https://review.opendev.org/c/openstack/nova/+/812392
21:23:55 _jralbert We recently upgraded from Train to Ussuri; since then, our compute nodes which rely on libvirt and vfio mdev to provide access to vGPU slices of physical GPUs can no longer start nova-compute after a reboot. The nova-compute code around recreating assigned mdevs appears to have changed fairly significantly between these releases, and I'm wondering if anyone here knows the details to chat about them before I get into creating a bug repor
#openstack-nova - 2021-10-05
06:45:45 gibi _jralbert: you need bauzas to wake up (he is in EU timezone)
07:17:08 bauzas good morning Nova
07:17:38 bauzas gibi, _jralbert: what's up ?
07:17:58 bauzas oh the reboot case
07:18:22 bauzas there is an open bug I need to correctly care https://bugs.launchpad.net/nova/+bug/1900800
08:37:37 gibi bauzas: I knew that you know more
08:38:33 bauzas gibi: I know you knew I know ;)

Earlier   Later