| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-21 | |||
| 10:52:48 | EugenMayer | host os would not make any sense here | |
| 10:53:04 | sean-k-mooney | ack | |
| 10:53:08 | EugenMayer | sicne nova_compute and nova_libvirt run in dedicated docker containers, so the os is picked by kolla | |
| 10:53:20 | sean-k-mooney | not nessisarly | |
| 10:53:23 | sean-k-mooney | they share the kernel | |
| 10:53:41 | EugenMayer | the container runs debian bullseye | |
| 10:53:41 | sean-k-mooney | and if md5 was disabeled by the kernel security policy it would not work | |
| 10:53:58 | EugenMayer | i see, well the host is what is expected by kolla, some version of ubuntu | |
| 10:54:15 | sean-k-mooney | ya should not be an issue | |
| 10:54:20 | EugenMayer | 11.4 | |
| 10:54:56 | sean-k-mooney | i just asked becasue i know in fips enforcing mode md5 is not allowed by the kernel and its default to DIGEST_MD5 when libvirt_tls is not enabled | |
| 10:55:22 | sean-k-mooney | that woudl likely give you a diffent error anyway | |
| 10:56:02 | sean-k-mooney | https://github.com/openstack/kolla-ansible/blob/stable/xena/ansible/roles/nova-cell/handlers/main.yml#L118-L129= feels like a bit of a hack but form what i can see you appear to have set it correctly manually too | |
| 10:57:56 | sean-k-mooney | ah i see | |
| 10:57:59 | sean-k-mooney | https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-4b6ee2f357ff265811c14e74dd6144d494c4f2baf9e00a9871207114b1172a4bR58 | |
| 10:58:05 | sean-k-mooney | that is how they are making this work | |
| 10:58:40 | EugenMayer | yes | |
| 10:58:51 | sean-k-mooney | EugenMayer: so they are modifying the default session config for the nova user | |
| 10:59:07 | sean-k-mooney | that is distro specific normally | |
| 10:59:32 | sean-k-mooney | as in rhel adn ubuntu have different default for how virsh functions | |
| 10:59:54 | sean-k-mooney | but since its the same debian image | |
| 11:00:06 | sean-k-mooney | i woudl expect it to be the same with kolla | |
| 11:00:28 | EugenMayer | probably | |
| 11:00:39 | sean-k-mooney | EugenMayer: have you tried using virsh in the nova_compute container | |
| 11:00:45 | sean-k-mooney | and seeign if it can coonnect | |
| 11:01:12 | EugenMayer | i did not - running with sasl false right now (running reconfigure) | |
| 11:01:19 | EugenMayer | cannot test that rigtht now | |
| 11:01:24 | sean-k-mooney | no worries | |
| 11:01:25 | EugenMayer | important thing would be https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-daa292a9ef0f95cd3db9000779f7ae3caf56cc4628790fc37c361638d32e5861R12 | |
| 11:01:47 | EugenMayer | if that was not set, it would not matter that the auth.conf was there and the linux user had the proper sasl password | |
| 11:02:07 | sean-k-mooney | right | |
| 11:02:51 | sean-k-mooney | presumable this is tested in the kolla ci | |
| 11:03:12 | sean-k-mooney | if not perhaps a recnet libvirt release changed something | |
| 11:03:51 | EugenMayer | without sasl it works, its all up and runnign again | |
| 11:04:56 | sean-k-mooney | ack. | |
| 11:05:11 | EugenMayer | my best guess right now is https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-daa292a9ef0f95cd3db9000779f7ae3caf56cc4628790fc37c361638d32e5861R12 as not set to sasl | |
| 11:05:25 | EugenMayer | right now it is none, but well i did not peak that file beforehand | |
| 11:05:37 | EugenMayer | just while i was applying the configuration with disabled sasl | |
| 11:05:51 | sean-k-mooney | kolla can generate teh config without pushing them right | |
| 11:05:52 | EugenMayer | i guess i will need to delay this, i did all this mess on prod | |
| 11:06:05 | sean-k-mooney | oh ya | |
| 11:06:05 | EugenMayer | no it generates them on the spot AFAIU | |
| 11:06:20 | sean-k-mooney | ok i tough tyou coudl run config without reconfigure | |
| 11:06:29 | EugenMayer | since it uses ansible, it will do all that remotely | |
| 11:06:29 | sean-k-mooney | its been a while so i may be wrong | |
| 11:06:42 | sean-k-mooney | in anycase i would play with this in a vm not prod | |
| 11:07:00 | EugenMayer | sure, but it is not easy to reproduce that | |
| 11:07:14 | sean-k-mooney | deploy Wallaby | |
| 11:07:19 | EugenMayer | i bet this is an upgrade issue and if a spin up a new openstack cluster with kolla it will work | |
| 11:07:20 | sean-k-mooney | then upgrade and it breaks no? | |
| 11:07:53 | sean-k-mooney | yep i woudl expect if you do singel node wallay in a vm | |
| 11:07:56 | EugenMayer | well that does not ensure we get into the same case. right now i upgraded in serias from xena x to xena >x, and this failed | |
| 11:07:58 | sean-k-mooney | and upgrade it it will break | |
| 11:08:14 | sean-k-mooney | oh ok | |
| 11:08:16 | EugenMayer | s/in serias/in series | |
| 11:08:25 | sean-k-mooney | it was a minor update with in xena | |
| 11:08:27 | sean-k-mooney | got it | |
| 11:08:43 | sean-k-mooney | very odd that this woudl be added mid series | |
| 11:08:43 | EugenMayer | yes, so that is why i actually used prod. i did not jump a series | |
| 11:09:02 | EugenMayer | they do that often, last mid series upgrade had a similar issue/breaking change | |
| 11:09:19 | EugenMayer | https://docs.openstack.org/releasenotes/kolla-ansible/xena.html#relnotes-13-0-0-stable-xena-upgrade-notes | |
| 11:09:20 | sean-k-mooney | add the feature maybe enabling it by default i woudl not expect | |
| 11:09:56 | EugenMayer | but back then i think the reason was i was running on non stable kolla for xena, since they has not been a release for the xena stable release for quiet some time after xena itself was released | |
| 11:10:01 | sean-k-mooney | basically in a minor update case i woudl expect to not need to update any config | |
| 11:10:25 | sean-k-mooney | oh you deploy xena with master | |
| 11:10:51 | sean-k-mooney | ya i have found issue doing that in the past | |
| 11:10:56 | EugenMayer | then i upgraded to 13.0 and had to deal with the adjustments, which i see justified. now it was kolla 13.0 to 13.1 mid xena, with again breaking changes which kind of scare me | |
| 11:11:25 | sean-k-mooney | ya i personally woudl not have defaulted this to enabled | |
| 11:11:29 | EugenMayer | well i deploy xena with the stable/xena branch of kolla - back then there was no alternativw | |
| 11:11:40 | sean-k-mooney | i may have backported it because deployment tools have different policies on that | |
| 11:11:46 | sean-k-mooney | but not requried any operator intervention | |
| 11:11:52 | sean-k-mooney | e.g. make it opt in | |
| 11:12:06 | sean-k-mooney | they may have wanted it to be secure by default | |
| 11:12:11 | EugenMayer | yeah that is how i would have done it, opt in for xena and default for yoga | |
| 11:12:18 | sean-k-mooney | but i woudl only have done that in yoga | |
| 11:12:29 | EugenMayer | still, release opt in, learn about the issues, make it default next one | |
| 11:12:48 | EugenMayer | do not release such a change and make it the default in one go .. that would be my mantra | |
| 11:13:04 | EugenMayer | as i see, we have the same ideas here | |
| 11:14:02 | sean-k-mooney | yep i perscibe to what is sometiem refered to as the grenade theroy of upgrade | |
| 11:14:18 | sean-k-mooney | https://opendev.org/openstack/grenade#Theory%20of%20Upgrade | |
| 11:20:35 | EugenMayer | interesting | |
| 11:23:10 | sean-k-mooney | if a project follow stable polciy ^ is the critia it needs to meet | |
| 11:23:28 | sean-k-mooney | or at least a sub set of it | |
| 11:23:41 | sean-k-mooney | when the project is evaulting uprade impact | |
| 11:24:33 | sean-k-mooney | stable policy is a liggel more inovvled but for nova for example we need to ensure feature we develop on master comply with the grenade theory of upgrade | |
| 11:25:14 | EugenMayer | well i guess it sometimes frustrates the developers to be limited by those rules | |
| 11:26:49 | EugenMayer | but in the end your audience will be a happier ones. Scaring our audience away from applying upgrades due to the expectation of 'every one of those will be problematic', will make it worse | |
| 11:27:20 | sean-k-mooney | i have never really found it limiting | |
| 11:27:45 | sean-k-mooney | we just wait a release to turns things on if it need operator intervention | |
| 11:27:53 | sean-k-mooney | it also lets us find bugs | |
| 11:28:17 | sean-k-mooney | we enable it in the nova-next jobs for a cycle and see if it has issues | |
| 11:29:09 | EugenMayer | feature flags often help if one is impatience here | |
| 13:08:45 | sean-k-mooney | gibi: bauzas is downstream irc down for ye | |
| 13:09:05 | bauzas | sean-k-mooney: nope | |
| 13:09:06 | gibi | nope | |
| 13:09:18 | bauzas | and I don't see a netsplit | |
| 13:09:34 | sean-k-mooney | looks like the vpn dropped again | |
| 13:09:42 | sean-k-mooney | ok ill fix that | |
| 13:54:26 | opendevreview | Rico Lin proposed openstack/nova master: Add locked_memory extra spec and image property https://review.opendev.org/c/openstack/nova/+/778347 | |
| 13:54:27 | opendevreview | Rico Lin proposed openstack/nova master: libvirt: Add vIOMMU device to guest https://review.opendev.org/c/openstack/nova/+/830646 | |