| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-21 | |||
| 10:44:28 | sean-k-mooney | lookign at the error | |
| 10:44:45 | sean-k-mooney | i wonder is this related to userids and group ids | |
| 10:44:55 | sean-k-mooney | libvirt is sayign the user does not exist | |
| 10:45:30 | sean-k-mooney | user | |
| 10:45:33 | sean-k-mooney | | not found: unable to canonify user and get auxprops | |
| 10:46:58 | sean-k-mooney | EugenMayer: kolla defiens the nova user id and group id https://github.com/openstack/kolla/blob/master/kolla/common/users.py#L148-L151= | |
| 10:47:08 | sean-k-mooney | but maybe that user is not create in the libvirt contaienr | |
| 10:47:28 | EugenMayer | the docs are wrong there anyway | |
| 10:47:56 | EugenMayer | The username is configured via ``libvirt_sasl_authname``, and defaults to | |
| 10:47:56 | EugenMayer | ``kolla``. The password is configured via ``libvirt_sasl_password``, and is | |
| 10:47:56 | EugenMayer | generated with other passwords using and stored in ``passwords.yml``. | |
| 10:48:14 | EugenMayer | which is wrong, the default user is nova | |
| 10:48:17 | EugenMayer | # Username for libvirt SASL. | |
| 10:48:17 | EugenMayer | libvirt_sasl_authname: "nova" | |
| 10:48:23 | sean-k-mooney | ack | |
| 10:48:24 | EugenMayer | and also in my auth.conf it is nova | |
| 10:48:26 | sean-k-mooney | https://github.com/openstack/kolla/blob/master/docker/nova/nova-libvirt/Dockerfile.j2#L10= | |
| 10:48:38 | sean-k-mooney | and nova should exist in the libvirt container | |
| 10:49:07 | EugenMayer | i did create it using saslpasswd2 -c -p -a libvirt nova | |
| 10:49:18 | EugenMayer | myself, just to ensure it has been provisioned | |
| 10:49:38 | sean-k-mooney | in the container | |
| 10:49:42 | sean-k-mooney | ok | |
| 10:50:04 | EugenMayer | you mean, there should be a linux user 'nova'? | |
| 10:50:12 | sean-k-mooney | yes | |
| 10:50:18 | sean-k-mooney | i think that is what the error is about | |
| 10:50:21 | EugenMayer | (nova-libvirt)[root@compute1 /]# cat /etc/passwd | grep nova | |
| 10:50:21 | EugenMayer | nova:x:42436:42436::/var/lib/nova:/usr/sbin/nologin | |
| 10:50:30 | sean-k-mooney | yep | |
| 10:50:33 | sean-k-mooney | that is what we expect | |
| 10:50:42 | EugenMayer | yes | |
| 10:51:28 | sean-k-mooney | what host os are you using | |
| 10:51:50 | sean-k-mooney | i assum this is not related to md5 beign diabled because of fips or similar | |
| 10:52:48 | EugenMayer | host os would not make any sense here | |
| 10:53:04 | sean-k-mooney | ack | |
| 10:53:08 | EugenMayer | sicne nova_compute and nova_libvirt run in dedicated docker containers, so the os is picked by kolla | |
| 10:53:20 | sean-k-mooney | not nessisarly | |
| 10:53:23 | sean-k-mooney | they share the kernel | |
| 10:53:41 | EugenMayer | the container runs debian bullseye | |
| 10:53:41 | sean-k-mooney | and if md5 was disabeled by the kernel security policy it would not work | |
| 10:53:58 | EugenMayer | i see, well the host is what is expected by kolla, some version of ubuntu | |
| 10:54:15 | sean-k-mooney | ya should not be an issue | |
| 10:54:20 | EugenMayer | 11.4 | |
| 10:54:56 | sean-k-mooney | i just asked becasue i know in fips enforcing mode md5 is not allowed by the kernel and its default to DIGEST_MD5 when libvirt_tls is not enabled | |
| 10:55:22 | sean-k-mooney | that woudl likely give you a diffent error anyway | |
| 10:56:02 | sean-k-mooney | https://github.com/openstack/kolla-ansible/blob/stable/xena/ansible/roles/nova-cell/handlers/main.yml#L118-L129= feels like a bit of a hack but form what i can see you appear to have set it correctly manually too | |
| 10:57:56 | sean-k-mooney | ah i see | |
| 10:57:59 | sean-k-mooney | https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-4b6ee2f357ff265811c14e74dd6144d494c4f2baf9e00a9871207114b1172a4bR58 | |
| 10:58:05 | sean-k-mooney | that is how they are making this work | |
| 10:58:40 | EugenMayer | yes | |
| 10:58:51 | sean-k-mooney | EugenMayer: so they are modifying the default session config for the nova user | |
| 10:59:07 | sean-k-mooney | that is distro specific normally | |
| 10:59:32 | sean-k-mooney | as in rhel adn ubuntu have different default for how virsh functions | |
| 10:59:54 | sean-k-mooney | but since its the same debian image | |
| 11:00:06 | sean-k-mooney | i woudl expect it to be the same with kolla | |
| 11:00:28 | EugenMayer | probably | |
| 11:00:39 | sean-k-mooney | EugenMayer: have you tried using virsh in the nova_compute container | |
| 11:00:45 | sean-k-mooney | and seeign if it can coonnect | |
| 11:01:12 | EugenMayer | i did not - running with sasl false right now (running reconfigure) | |
| 11:01:19 | EugenMayer | cannot test that rigtht now | |
| 11:01:24 | sean-k-mooney | no worries | |
| 11:01:25 | EugenMayer | important thing would be https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-daa292a9ef0f95cd3db9000779f7ae3caf56cc4628790fc37c361638d32e5861R12 | |
| 11:01:47 | EugenMayer | if that was not set, it would not matter that the auth.conf was there and the linux user had the proper sasl password | |
| 11:02:07 | sean-k-mooney | right | |
| 11:02:51 | sean-k-mooney | presumable this is tested in the kolla ci | |
| 11:03:12 | sean-k-mooney | if not perhaps a recnet libvirt release changed something | |
| 11:03:51 | EugenMayer | without sasl it works, its all up and runnign again | |
| 11:04:56 | sean-k-mooney | ack. | |
| 11:05:11 | EugenMayer | my best guess right now is https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-daa292a9ef0f95cd3db9000779f7ae3caf56cc4628790fc37c361638d32e5861R12 as not set to sasl | |
| 11:05:25 | EugenMayer | right now it is none, but well i did not peak that file beforehand | |
| 11:05:37 | EugenMayer | just while i was applying the configuration with disabled sasl | |
| 11:05:51 | sean-k-mooney | kolla can generate teh config without pushing them right | |
| 11:05:52 | EugenMayer | i guess i will need to delay this, i did all this mess on prod | |
| 11:06:05 | sean-k-mooney | oh ya | |
| 11:06:05 | EugenMayer | no it generates them on the spot AFAIU | |
| 11:06:20 | sean-k-mooney | ok i tough tyou coudl run config without reconfigure | |
| 11:06:29 | EugenMayer | since it uses ansible, it will do all that remotely | |
| 11:06:29 | sean-k-mooney | its been a while so i may be wrong | |
| 11:06:42 | sean-k-mooney | in anycase i would play with this in a vm not prod | |
| 11:07:00 | EugenMayer | sure, but it is not easy to reproduce that | |
| 11:07:14 | sean-k-mooney | deploy Wallaby | |
| 11:07:19 | EugenMayer | i bet this is an upgrade issue and if a spin up a new openstack cluster with kolla it will work | |
| 11:07:20 | sean-k-mooney | then upgrade and it breaks no? | |
| 11:07:53 | sean-k-mooney | yep i woudl expect if you do singel node wallay in a vm | |
| 11:07:56 | EugenMayer | well that does not ensure we get into the same case. right now i upgraded in serias from xena x to xena >x, and this failed | |
| 11:07:58 | sean-k-mooney | and upgrade it it will break | |
| 11:08:14 | sean-k-mooney | oh ok | |
| 11:08:16 | EugenMayer | s/in serias/in series | |
| 11:08:25 | sean-k-mooney | it was a minor update with in xena | |
| 11:08:27 | sean-k-mooney | got it | |
| 11:08:43 | sean-k-mooney | very odd that this woudl be added mid series | |
| 11:08:43 | EugenMayer | yes, so that is why i actually used prod. i did not jump a series | |
| 11:09:02 | EugenMayer | they do that often, last mid series upgrade had a similar issue/breaking change | |
| 11:09:19 | EugenMayer | https://docs.openstack.org/releasenotes/kolla-ansible/xena.html#relnotes-13-0-0-stable-xena-upgrade-notes | |
| 11:09:20 | sean-k-mooney | add the feature maybe enabling it by default i woudl not expect | |
| 11:09:56 | EugenMayer | but back then i think the reason was i was running on non stable kolla for xena, since they has not been a release for the xena stable release for quiet some time after xena itself was released | |
| 11:10:01 | sean-k-mooney | basically in a minor update case i woudl expect to not need to update any config | |
| 11:10:25 | sean-k-mooney | oh you deploy xena with master | |
| 11:10:51 | sean-k-mooney | ya i have found issue doing that in the past | |
| 11:10:56 | EugenMayer | then i upgraded to 13.0 and had to deal with the adjustments, which i see justified. now it was kolla 13.0 to 13.1 mid xena, with again breaking changes which kind of scare me | |
| 11:11:25 | sean-k-mooney | ya i personally woudl not have defaulted this to enabled | |