| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-22 | |||
| 18:52:05 | sean-k-mooney | but ya ill take a look quickly and ssee if i can see teh actual failure | |
| 18:52:39 | melwitt | the first problem was on compute1 it would never find a ipv6 interface because the awk command or whatever was not able to work with debian. so I made a devstack dnm patch for that. then it got farther to the point where compute1 is trying to do a very basic GET to /identity and it just hangs | |
| 18:54:08 | melwitt | the second problem was compute1 tls cert was for the controller commonName so it did not work (the code simply copied controller's certs to subnode) so I added to my dnm patch to make the subnode actually make a cert for itself, for its commonName. then it got farther and then crapped out trying to GET /identity | |
| 18:54:40 | sean-k-mooney | so we have a role that does that copy | |
| 18:54:44 | melwitt | at that point I didn't know what to do | |
| 18:54:58 | sean-k-mooney | but we dont normally need a cert for the compute | |
| 18:55:03 | sean-k-mooney | we just need the ca | |
| 18:55:09 | sean-k-mooney | so it trust the contoler | |
| 18:55:33 | melwitt | I got a cert validation error when subnode tried to contact controller so that's why I did it. oh, ok | |
| 18:55:58 | melwitt | not sure why what was there originally didn't work. maybe it was a one-off | |
| 18:56:40 | sean-k-mooney | maybe or maybe its because of how the name/ip is included in the cert | |
| 18:57:05 | sean-k-mooney | like if we are connetiv via the ipv6 adress rather then the name we might need to do soemthign differnt | |
| 18:57:10 | melwitt | yeah the complaint mentioned the commonName not matching or something like that | |
| 18:57:29 | sean-k-mooney | oh https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-hosts-file | |
| 18:57:43 | sean-k-mooney | so that only adds the ipv4 adresses to /etc/hosts | |
| 18:58:24 | sean-k-mooney | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-hosts-file/tasks/main.yaml so maybe that a factor | |
| 18:58:34 | melwitt | hm ok | |
| 18:59:29 | sean-k-mooney | what that does is loop over all the hsot in the inventory adn update /etc/host so that contoller maps to the correct ip | |
| 18:59:57 | sean-k-mooney | that may or may not be related jsut noticed the doc says its only for ipv4 which the code confirms | |
| 19:01:09 | melwitt | I see | |
| 19:05:08 | sean-k-mooney | thisd does nto prove anyting but if you look at the zuul-info.* files https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/index.html | |
| 19:05:44 | sean-k-mooney | what i do notice is the compute seams to have much hihger latency then the contoler | |
| 19:07:17 | sean-k-mooney | they both have the saem default route | |
| 19:07:47 | sean-k-mooney | btu they took difent path back to open dev (ill ignore the fact the ipv4 trace rotes is actully v6 in both) | |
| 19:08:23 | sean-k-mooney | we also see fe80::ac5a:77ff:fe03:adb2 dev ens3 lladdr ae:5a:77:03:ad:b2 router DELAY vs fe80::3457:e8ff:fe88:9852 dev ens3 lladdr 36:57:e8:88:98:52 router REACHABLE | |
| 19:09:14 | sean-k-mooney | both those got a routabel ip in teh same ipv6 subnet 2001:41d0:302:1000::cbc/56 s | |
| 19:09:48 | sean-k-mooney | so they shoudl be able to comunicate fine | |
| 19:10:07 | melwitt | :/ | |
| 19:10:42 | sean-k-mooney | https://[2001:41d0:302:1000::1ab]/identity | |
| 19:10:53 | sean-k-mooney | so that what its trying to conenct too | |
| 19:11:05 | sean-k-mooney | but that is not the adress of the contoler | |
| 19:11:27 | sean-k-mooney | that is the adress of the compute | |
| 19:11:48 | sean-k-mooney | so the compute is trying to conenct to its own ipv6 adress to talk to keystone | |
| 19:11:58 | sean-k-mooney | and that is because SERVICE_HOST | |
| 19:12:01 | sean-k-mooney | is not set properly | |
| 19:13:00 | sean-k-mooney | your setting SERVICE_HOST: "" | |
| 19:13:05 | sean-k-mooney | that will work on singel node | |
| 19:13:36 | sean-k-mooney | but we actully need to set that to the ipv6 adress of the contoler | |
| 19:13:44 | melwitt | oh, bleh ok. | |
| 19:13:59 | sean-k-mooney | or the host naem in /etc/hosts need to resove to the ipv6 adres | |
| 19:14:01 | melwitt | I just copied it from the base ipv6 job out of desperation | |
| 19:14:05 | melwitt | ok. sorry about that | |
| 19:16:26 | sean-k-mooney | dont be sorry. but that why i didnt set service host in my orgianl patch | |
| 19:17:00 | sean-k-mooney | the orcstarte devstack role shoudl populate that for us | |
| 19:17:13 | melwitt | I wonder why does the base job do that then | |
| 19:17:15 | sean-k-mooney | but it may or may not work out of the box for ipv6 | |
| 19:17:39 | sean-k-mooney | melwitt: my gess it its a hack because it use the ipv4 adress or similar | |
| 19:17:48 | sean-k-mooney | but lets find out | |
| 19:19:47 | opendevreview | melanie witt proposed openstack/nova master: DNM test ceph ipv6 https://review.opendev.org/c/openstack/nova/+/984623 | |
| 19:20:40 | sean-k-mooney | so i tought it was configred via https://github.com/openstack/devstack/blob/03ece8f88040be9b0b14dd1cfe93076ad2419a80/playbooks/pre.yaml#L37 | |
| 19:20:47 | sean-k-mooney | but no... | |
| 19:22:09 | sean-k-mooney | ah https://github.com/openstack/devstack/blob/03ece8f88040be9b0b14dd1cfe93076ad2419a80/.zuul.yaml#L514 | |
| 19:22:33 | sean-k-mooney | so devstack-minimal is doing SERVICE_HOST: "{{ hostvars['controller']['nodepool']['private_ipv4'] }}" | |
| 19:22:53 | sean-k-mooney | melwitt: can you try changing that to SERVICE_HOST: "{{ hostvars['controller']['nodepool']['private_ipv6'] }}" im goign to doubel check that thats a thing | |
| 19:23:34 | melwitt | sure | |
| 19:24:18 | sean-k-mooney | ok that will not work but we can get the ip adress in a difennt place | |
| 19:24:21 | sean-k-mooney | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/inventory.yaml | |
| 19:24:31 | sean-k-mooney | is the inventory file and that has private_ipv6: null | |
| 19:25:24 | melwitt | ah | |
| 19:26:00 | sean-k-mooney | but if we look in teh ansibel fact https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/host-info.controller.yaml | |
| 19:26:12 | sean-k-mooney | we defnilly have access to the ip | |
| 19:26:44 | sean-k-mooney | i think this will work. one sec whil i type it out | |
| 19:28:15 | sean-k-mooney | SERVICE_HOST: ""{{hostvars['controller']['ansible_defult_ipv6']['address']}}" | |
| 19:28:41 | sean-k-mooney | well - the extra " | |
| 19:28:51 | sean-k-mooney | SERVICE_HOST: "{{hostvars['controller']['ansible_defult_ipv6']['address']}}" | |
| 19:28:53 | melwitt | yes I will edit the typos :) | |
| 19:29:45 | sean-k-mooney | you will also need to set it for the subvar | |
| 19:29:58 | melwitt | ack | |
| 19:30:18 | sean-k-mooney | we can proably fix this in the devstack defintion to be based on if you have selecte ipv4 or ipv6 for the serivce ip later | |
| 19:31:22 | sean-k-mooney | in the job defintion we are seting SERVICE_IP_VERSION: 6 so we have something to switch off | |
| 19:31:41 | sean-k-mooney | withthat said if i look at the job vriable for the compute | |
| 19:31:43 | sean-k-mooney | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/inventory.yaml | |
| 19:32:01 | sean-k-mooney | i see a lot of other ip v4 stuff there but we can revisti that if needed | |
| 19:32:43 | opendevreview | melanie witt proposed openstack/nova master: DNM test ceph ipv6 https://review.opendev.org/c/openstack/nova/+/984623 | |
| 19:33:22 | sean-k-mooney | melwitt: as an asside we started looking at this because we wanted to test a patch that was related to it | |
| 19:33:32 | sean-k-mooney | melwitt: do you recall if we you knwo merged that bug fix? | |
| 19:33:43 | sean-k-mooney | the libvirt migration one | |
| 19:34:06 | melwitt | right. no we did not merge it yet and I have been trying to make this work in order to verify it | |
| 19:34:13 | sean-k-mooney | ack | |
| 19:34:16 | melwitt | I'm not opposed to merging it without verification but it would be really great to be able to | |
| 19:34:43 | sean-k-mooney | oh ya i agree it would be good to have it | |
| 19:34:59 | sean-k-mooney | i just lost context on where we were at with the patch | |
| 19:35:06 | melwitt | after seeing that XML ipv6 brackets thing, I am feeling like I really want to see this work haha | |
| 19:35:16 | sean-k-mooney | yep | |
| 19:35:23 | sean-k-mooney | ok im going to wrap for today | |
| 19:36:09 | melwitt | it looks fine to me. just wanted to see if I could do this. if not, I'll just review it without | |
| 19:36:16 | sean-k-mooney | ill try and remember to check the resutls of tha trun but feel free to ping me to take another look | |
| 19:36:41 | melwitt | cool thanks for the help | |
| 19:37:19 | sean-k-mooney | melwitt: by the way i said this before but there is currently no ipv6 + ceph covage anywhere. i dont even think there is singel node | |
| 19:37:41 | sean-k-mooney | so defintly nice ot fix that at some point | |
| 19:37:57 | melwitt | ++ | |
| 19:39:16 | isaacvicente[m] | hi guys o/ i would like to start solving this bug: https://bugs.launchpad.net/nova/+bug/2035911 | |
| 19:39:16 | isaacvicente[m] | given the last time it was assigned to someone, its ok to assign to myself? i replied on the bug but got no reponse | |
| 19:40:38 | melwitt | isaacvicente[m]: yeah I would go ahead and assign it to yourself | |
| 19:48:10 | isaacvicente[m] | nice, i started looking into it, and i wonder where I would do the changes. Trying to understand better the bug and the code base itself I asked chatgpt-codex-5.3 where the changes would be. It said that drivers under nova/virt/libvirt/volumes has two functions: connect_volume and disconnect_volume. I would then add the os-brick context manager guard_connections on those functions. I'm not sure if this is the right | |
| 19:48:10 | isaacvicente[m] | direction. Does it make sense? | |
| 19:49:05 | isaacvicente[m] | * nice, i started looking into it, and i wonder where I would do the changes. Trying to understand better the bug and the code base itself I asked chatgpt-codex-5.3 where the changes would be. It said that drivers under nova/virt/libvirt/volumes has two functions: connect_volume and disconnect_volume. I would then add the os-brick context manager guard_connection on those functions. I'm not sure if this is the right | |
| 19:49:05 | isaacvicente[m] | direction. Does it make sense? | |
| #openstack-nova - 2026-04-23 | |||
| 00:58:55 | opendevreview | melanie witt proposed openstack/nova master: DNM test ceph ipv6 https://review.opendev.org/c/openstack/nova/+/984623 | |
| 02:09:31 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |