| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-21 | |||
| 10:38:18 | EugenMayer | sean-k-mooney what you linked is perfectly right, but you also see, they default to enabling it | |
| 10:38:19 | sean-k-mooney | so its nice that it works | |
| 10:38:36 | EugenMayer | i see | |
| 10:39:13 | sean-k-mooney | https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#sasl-authentication | |
| 10:40:04 | sean-k-mooney | so it looks like you need new passwoard in you passwords.yal | |
| 10:40:15 | sean-k-mooney | i assume you geneerated those and mreged them with your exsitng ones | |
| 10:41:10 | EugenMayer | did that already, the upgrade was not working without that | |
| 10:41:30 | sean-k-mooney | ya i would expect the templates to fail to generate the config | |
| 10:41:31 | EugenMayer | that is why i said on the compute, the auth.conf is created, has the right creds | |
| 10:41:37 | sean-k-mooney | ack | |
| 10:41:56 | sean-k-mooney | for now i would proably disabel the sasl auth and complete the upgrade | |
| 10:41:58 | EugenMayer | and i assured that the backend, libvirt, has set the sasls password for the particular user | |
| 10:42:06 | sean-k-mooney | then try and do a reconfigure later | |
| 10:42:13 | EugenMayer | yes, seems that this might be the option i should go for | |
| 10:43:16 | sean-k-mooney | i have not deployed kolla since wallaby by the way | |
| 10:43:27 | sean-k-mooney | so im not up to date on how they currenlty do things | |
| 10:44:28 | sean-k-mooney | lookign at the error | |
| 10:44:45 | sean-k-mooney | i wonder is this related to userids and group ids | |
| 10:44:55 | sean-k-mooney | libvirt is sayign the user does not exist | |
| 10:45:30 | sean-k-mooney | user | |
| 10:45:33 | sean-k-mooney | | not found: unable to canonify user and get auxprops | |
| 10:46:58 | sean-k-mooney | EugenMayer: kolla defiens the nova user id and group id https://github.com/openstack/kolla/blob/master/kolla/common/users.py#L148-L151= | |
| 10:47:08 | sean-k-mooney | but maybe that user is not create in the libvirt contaienr | |
| 10:47:28 | EugenMayer | the docs are wrong there anyway | |
| 10:47:56 | EugenMayer | generated with other passwords using and stored in ``passwords.yml``. | |
| 10:47:56 | EugenMayer | ``kolla``. The password is configured via ``libvirt_sasl_password``, and is | |
| 10:47:56 | EugenMayer | The username is configured via ``libvirt_sasl_authname``, and defaults to | |
| 10:48:14 | EugenMayer | which is wrong, the default user is nova | |
| 10:48:17 | EugenMayer | libvirt_sasl_authname: "nova" | |
| 10:48:17 | EugenMayer | # Username for libvirt SASL. | |
| 10:48:23 | sean-k-mooney | ack | |
| 10:48:24 | EugenMayer | and also in my auth.conf it is nova | |
| 10:48:26 | sean-k-mooney | https://github.com/openstack/kolla/blob/master/docker/nova/nova-libvirt/Dockerfile.j2#L10= | |
| 10:48:38 | sean-k-mooney | and nova should exist in the libvirt container | |
| 10:49:07 | EugenMayer | i did create it using saslpasswd2 -c -p -a libvirt nova | |
| 10:49:18 | EugenMayer | myself, just to ensure it has been provisioned | |
| 10:49:38 | sean-k-mooney | in the container | |
| 10:49:42 | sean-k-mooney | ok | |
| 10:50:04 | EugenMayer | you mean, there should be a linux user 'nova'? | |
| 10:50:12 | sean-k-mooney | yes | |
| 10:50:18 | sean-k-mooney | i think that is what the error is about | |
| 10:50:21 | EugenMayer | nova:x:42436:42436::/var/lib/nova:/usr/sbin/nologin | |
| 10:50:21 | EugenMayer | (nova-libvirt)[root@compute1 /]# cat /etc/passwd | grep nova | |
| 10:50:30 | sean-k-mooney | yep | |
| 10:50:33 | sean-k-mooney | that is what we expect | |
| 10:50:42 | EugenMayer | yes | |
| 10:51:28 | sean-k-mooney | what host os are you using | |
| 10:51:50 | sean-k-mooney | i assum this is not related to md5 beign diabled because of fips or similar | |
| 10:52:48 | EugenMayer | host os would not make any sense here | |
| 10:53:04 | sean-k-mooney | ack | |
| 10:53:08 | EugenMayer | sicne nova_compute and nova_libvirt run in dedicated docker containers, so the os is picked by kolla | |
| 10:53:20 | sean-k-mooney | not nessisarly | |
| 10:53:23 | sean-k-mooney | they share the kernel | |
| 10:53:41 | sean-k-mooney | and if md5 was disabeled by the kernel security policy it would not work | |
| 10:53:41 | EugenMayer | the container runs debian bullseye | |
| 10:53:58 | EugenMayer | i see, well the host is what is expected by kolla, some version of ubuntu | |
| 10:54:15 | sean-k-mooney | ya should not be an issue | |
| 10:54:20 | EugenMayer | 11.4 | |
| 10:54:56 | sean-k-mooney | i just asked becasue i know in fips enforcing mode md5 is not allowed by the kernel and its default to DIGEST_MD5 when libvirt_tls is not enabled | |
| 10:55:22 | sean-k-mooney | that woudl likely give you a diffent error anyway | |
| 10:56:02 | sean-k-mooney | https://github.com/openstack/kolla-ansible/blob/stable/xena/ansible/roles/nova-cell/handlers/main.yml#L118-L129= feels like a bit of a hack but form what i can see you appear to have set it correctly manually too | |
| 10:57:56 | sean-k-mooney | ah i see | |
| 10:57:59 | sean-k-mooney | https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-4b6ee2f357ff265811c14e74dd6144d494c4f2baf9e00a9871207114b1172a4bR58 | |
| 10:58:05 | sean-k-mooney | that is how they are making this work | |
| 10:58:40 | EugenMayer | yes | |
| 10:58:51 | sean-k-mooney | EugenMayer: so they are modifying the default session config for the nova user | |
| 10:59:07 | sean-k-mooney | that is distro specific normally | |
| 10:59:32 | sean-k-mooney | as in rhel adn ubuntu have different default for how virsh functions | |
| 10:59:54 | sean-k-mooney | but since its the same debian image | |
| 11:00:06 | sean-k-mooney | i woudl expect it to be the same with kolla | |
| 11:00:28 | EugenMayer | probably | |
| 11:00:39 | sean-k-mooney | EugenMayer: have you tried using virsh in the nova_compute container | |
| 11:00:45 | sean-k-mooney | and seeign if it can coonnect | |
| 11:01:12 | EugenMayer | i did not - running with sasl false right now (running reconfigure) | |
| 11:01:19 | EugenMayer | cannot test that rigtht now | |
| 11:01:24 | sean-k-mooney | no worries | |
| 11:01:25 | EugenMayer | important thing would be https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-daa292a9ef0f95cd3db9000779f7ae3caf56cc4628790fc37c361638d32e5861R12 | |
| 11:01:47 | EugenMayer | if that was not set, it would not matter that the auth.conf was there and the linux user had the proper sasl password | |
| 11:02:07 | sean-k-mooney | right | |
| 11:02:51 | sean-k-mooney | presumable this is tested in the kolla ci | |
| 11:03:12 | sean-k-mooney | if not perhaps a recnet libvirt release changed something | |
| 11:03:51 | EugenMayer | without sasl it works, its all up and runnign again | |
| 11:04:56 | sean-k-mooney | ack. | |
| 11:05:11 | EugenMayer | my best guess right now is https://github.com/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5#diff-daa292a9ef0f95cd3db9000779f7ae3caf56cc4628790fc37c361638d32e5861R12 as not set to sasl | |
| 11:05:25 | EugenMayer | right now it is none, but well i did not peak that file beforehand | |
| 11:05:37 | EugenMayer | just while i was applying the configuration with disabled sasl | |
| 11:05:51 | sean-k-mooney | kolla can generate teh config without pushing them right | |
| 11:05:52 | EugenMayer | i guess i will need to delay this, i did all this mess on prod | |
| 11:06:05 | EugenMayer | no it generates them on the spot AFAIU | |
| 11:06:05 | sean-k-mooney | oh ya | |
| 11:06:20 | sean-k-mooney | ok i tough tyou coudl run config without reconfigure | |
| 11:06:29 | sean-k-mooney | its been a while so i may be wrong | |
| 11:06:29 | EugenMayer | since it uses ansible, it will do all that remotely | |
| 11:06:42 | sean-k-mooney | in anycase i would play with this in a vm not prod | |
| 11:07:00 | EugenMayer | sure, but it is not easy to reproduce that | |
| 11:07:14 | sean-k-mooney | deploy Wallaby | |
| 11:07:19 | EugenMayer | i bet this is an upgrade issue and if a spin up a new openstack cluster with kolla it will work | |
| 11:07:20 | sean-k-mooney | then upgrade and it breaks no? | |
| 11:07:53 | sean-k-mooney | yep i woudl expect if you do singel node wallay in a vm | |
| 11:07:56 | EugenMayer | well that does not ensure we get into the same case. right now i upgraded in serias from xena x to xena >x, and this failed | |