Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-07-21
11:00:06 sean-k-mooney i woudl expect it to be the same with kolla
11:00:28 EugenMayer probably
11:00:39 sean-k-mooney EugenMayer: have you tried using virsh in the nova_compute container
11:00:45 sean-k-mooney and seeign if it can coonnect
11:01:12 EugenMayer i did not - running with sasl false right now (running reconfigure)
11:01:19 EugenMayer cannot test that rigtht now
11:01:24 sean-k-mooney no worries
11:01:25 EugenMayer important thing would be https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-daa292a9ef0f95cd3db9000779f7ae3caf56cc4628790fc37c361638d32e5861R12
11:01:47 EugenMayer if that was not set, it would not matter that the auth.conf was there and the linux user had the proper sasl password
11:02:07 sean-k-mooney right
11:02:51 sean-k-mooney presumable this is tested in the kolla ci
11:03:12 sean-k-mooney if not perhaps a recnet libvirt release changed something
11:03:51 EugenMayer without sasl it works, its all up and runnign again
11:04:56 sean-k-mooney ack.
11:05:11 EugenMayer my best guess right now is https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-daa292a9ef0f95cd3db9000779f7ae3caf56cc4628790fc37c361638d32e5861R12 as not set to sasl
11:05:25 EugenMayer right now it is none, but well i did not peak that file beforehand
11:05:37 EugenMayer just while i was applying the configuration with disabled sasl
11:05:51 sean-k-mooney kolla can generate teh config without pushing them right
11:05:52 EugenMayer i guess i will need to delay this, i did all this mess on prod
11:06:05 sean-k-mooney oh ya
11:06:05 EugenMayer no it generates them on the spot AFAIU
11:06:20 sean-k-mooney ok i tough tyou coudl run config without reconfigure
11:06:29 EugenMayer since it uses ansible, it will do all that remotely
11:06:29 sean-k-mooney its been a while so i may be wrong
11:06:42 sean-k-mooney in anycase i would play with this in a vm not prod
11:07:00 EugenMayer sure, but it is not easy to reproduce that
11:07:14 sean-k-mooney deploy Wallaby
11:07:19 EugenMayer i bet this is an upgrade issue and if a spin up a new openstack cluster with kolla it will work
11:07:20 sean-k-mooney then upgrade and it breaks no?
11:07:53 sean-k-mooney yep i woudl expect if you do singel node wallay in a vm
11:07:56 EugenMayer well that does not ensure we get into the same case. right now i upgraded in serias from xena x to xena >x, and this failed
11:07:58 sean-k-mooney and upgrade it it will break
11:08:14 sean-k-mooney oh ok
11:08:16 EugenMayer s/in serias/in series
11:08:25 sean-k-mooney it was a minor update with in xena
11:08:27 sean-k-mooney got it
11:08:43 sean-k-mooney very odd that this woudl be added mid series
11:08:43 EugenMayer yes, so that is why i actually used prod. i did not jump a series
11:09:02 EugenMayer they do that often, last mid series upgrade had a similar issue/breaking change
11:09:19 EugenMayer https://docs.openstack.org/releasenotes/kolla-ansible/xena.html#relnotes-13-0-0-stable-xena-upgrade-notes
11:09:20 sean-k-mooney add the feature maybe enabling it by default i woudl not expect
11:09:56 EugenMayer but back then i think the reason was i was running on non stable kolla for xena, since they has not been a release for the xena stable release for quiet some time after xena itself was released
11:10:01 sean-k-mooney basically in a minor update case i woudl expect to not need to update any config
11:10:25 sean-k-mooney oh you deploy xena with master
11:10:51 sean-k-mooney ya i have found issue doing that in the past
11:10:56 EugenMayer then i upgraded to 13.0 and had to deal with the adjustments, which i see justified. now it was kolla 13.0 to 13.1 mid xena, with again breaking changes which kind of scare me
11:11:25 sean-k-mooney ya i personally woudl not have defaulted this to enabled
11:11:29 EugenMayer well i deploy xena with the stable/xena branch of kolla - back then there was no alternativw
11:11:40 sean-k-mooney i may have backported it because deployment tools have different policies on that
11:11:46 sean-k-mooney but not requried any operator intervention
11:11:52 sean-k-mooney e.g. make it opt in
11:12:06 sean-k-mooney they may have wanted it to be secure by default
11:12:11 EugenMayer yeah that is how i would have done it, opt in for xena and default for yoga
11:12:18 sean-k-mooney but i woudl only have done that in yoga
11:12:29 EugenMayer still, release opt in, learn about the issues, make it default next one
11:12:48 EugenMayer do not release such a change and make it the default in one go .. that would be my mantra
11:13:04 EugenMayer as i see, we have the same ideas here
11:14:02 sean-k-mooney yep i perscibe to what is sometiem refered to as the grenade theroy of upgrade
11:14:18 sean-k-mooney https://opendev.org/openstack/grenade#Theory%20of%20Upgrade
11:20:35 EugenMayer interesting
11:23:10 sean-k-mooney if a project follow stable polciy ^ is the critia it needs to meet
11:23:28 sean-k-mooney or at least a sub set of it
11:23:41 sean-k-mooney when the project is evaulting uprade impact
11:24:33 sean-k-mooney stable policy is a liggel more inovvled but for nova for example we need to ensure feature we develop on master comply with the grenade theory of upgrade
11:25:14 EugenMayer well i guess it sometimes frustrates the developers to be limited by those rules
11:26:49 EugenMayer but in the end your audience will be a happier ones. Scaring our audience away from applying upgrades due to the expectation of 'every one of those will be problematic', will make it worse
11:27:20 sean-k-mooney i have never really found it limiting
11:27:45 sean-k-mooney we just wait a release to turns things on if it need operator intervention
11:27:53 sean-k-mooney it also lets us find bugs
11:28:17 sean-k-mooney we enable it in the nova-next jobs for a cycle and see if it has issues
11:29:09 EugenMayer feature flags often help if one is impatience here
13:08:45 sean-k-mooney gibi: bauzas is downstream irc down for ye
13:09:05 bauzas sean-k-mooney: nope
13:09:06 gibi nope
13:09:18 bauzas and I don't see a netsplit
13:09:34 sean-k-mooney looks like the vpn dropped again
13:09:42 sean-k-mooney ok ill fix that
13:54:26 opendevreview Rico Lin proposed openstack/nova master: Add locked_memory extra spec and image property https://review.opendev.org/c/openstack/nova/+/778347
13:54:27 opendevreview Rico Lin proposed openstack/nova master: libvirt: Add vIOMMU device to guest https://review.opendev.org/c/openstack/nova/+/830646
13:54:27 opendevreview Rico Lin proposed openstack/nova master: Add traits for viommu model https://review.opendev.org/c/openstack/nova/+/844507
14:05:19 bauzas folks, I need to disappear for a dentist appointment
14:15:45 EugenMayer sean-k-mooney thank you for helping once again, sorry forgot the mention the obvious!
14:23:44 sean-k-mooney EugenMayer: no worries sorry i did not see how to make it work end to end but happy your prod it working again
15:28:59 opendevreview Amit Uniyal proposed openstack/nova master: add regression test case for bug 1978983 https://review.opendev.org/c/openstack/nova/+/849104
15:28:59 opendevreview Amit Uniyal proposed openstack/nova master: For evacuation, ignore if task_state is not None https://review.opendev.org/c/openstack/nova/+/848886
15:48:04 colby_ sean-k-mooney: bauzas: Still trying to get our vGPU stuff working. Here is the bug I filed: https://bugs.launchpad.net/nova/+bug/1981631. Im happy to be a guinea pig for testing. I can also poke around and help gather any info you need to help identify the issue. I did some logging of python code variables to see what rp(pri device) its trying to use when they are all created already and its def trying to use a new
15:48:04 colby_ one instead of existing. Let me know if you need anything from me as manually deleting the mdevs is not ideal for production at this point.
15:48:30 bauzas colby_: fwiw, I opened a bug against libvirt for it
15:48:47 bauzas colby_: https://bugzilla.redhat.com/show_bug.cgi?id=2109450
15:51:30 colby_ ok thanks! Its good to know its on that end I guess. I guess that is the drawback of using centos8 stream which has the newer version of libvirt
15:52:59 colby_ So will using that new API be backported to older releases once you add that in the next release?
15:53:44 bauzas colby_: or libvirt will fix it
15:55:33 colby_ or are we stuck hoping redhat will add the old functionality back in?
15:55:37 colby_ ah ok
16:01:29 bauzas colby_: tbc, I'm working on a POC to use the new API
16:01:37 bauzas but ideally, the regression should be fixed
16:02:26 colby_ cool. thanks for pointing me to that bug. Ill keep an eye on it.
16:02:28 opendevreview Balazs Gibizer proposed openstack/nova master: Reproducer for bug 1982497 https://review.opendev.org/c/openstack/nova/+/850672
16:23:01 opendevreview Sylvain Bauza proposed openstack/nova master: Reproducer for bug 1951656 https://review.opendev.org/c/openstack/nova/+/850673
17:51:16 opendevreview ribaudr proposed openstack/nova master: Allow unshelve to a specific host (Compute API part) https://review.opendev.org/c/openstack/nova/+/831507

Earlier   Later