| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-22 | |||
| 18:33:53 | melwitt | this isn't super important obviously but just wanted to let you know what I did haha | |
| 18:34:06 | sean-k-mooney | we could jsut swap to ml2 ovs | |
| 18:34:12 | sean-k-mooney | like the alt config job | |
| 18:34:35 | melwitt | sometimes ip route list finds host_ip_iface=enX0 sometimes it finds host_ip_iface=ens3 | |
| 18:34:50 | sean-k-mooney | that depens on the disto/kernle | |
| 18:35:05 | sean-k-mooney | the systemd/bios dev name stuff | |
| 18:35:26 | melwitt | ah ok. yeah I ended up having to make DNM https://review.opendev.org/c/openstack/devstack/+/984605 to handle was it debian? I can't remember | |
| 18:36:02 | sean-k-mooney | yes we moved the ceph jobs ot debian 13 becuase of some qemu/ceph isues | |
| 18:36:16 | sean-k-mooney | basiclly to get newer versions with less bugs | |
| 18:36:24 | sean-k-mooney | so this is debian 13 | |
| 18:36:33 | melwitt | ok, so my next experiment can be configuring ml2 ovs in my dnm patch | |
| 18:36:38 | melwitt | that's helpful to know, thanks | |
| 18:36:58 | sean-k-mooney | you coudl do that or you could enable ceph in the nova alt config job | |
| 18:37:09 | sean-k-mooney | but eihter would worth a try | |
| 18:37:24 | melwitt | ah ok | |
| 18:37:38 | sean-k-mooney | the alt config job is already ml2 ovs i bleive | |
| 18:37:46 | sean-k-mooney | as is nova next | |
| 18:38:03 | sean-k-mooney | but one has legacy ip tables and hte other the ovs firewall driver | |
| 18:38:06 | melwitt | gotcha | |
| 18:41:31 | sean-k-mooney | melwitt: the other thin we may or many not need too do is modify the multi node bridge | |
| 18:42:07 | melwitt | that is uncharted territory for me but I will keep in mind | |
| 18:42:36 | sean-k-mooney | melwitt: if you have not looked at the multi node job in detail, it create linuxor ovs? briges on each of the hosts and and interconenct them with vxlan tunnels | |
| 18:42:54 | sean-k-mooney | so that we get a flat l2 network that we can shove vlans through | |
| 18:43:20 | sean-k-mooney | we might need to do somethign like assign ipv6 ips to that bridge and use those instead | |
| 18:43:31 | melwitt | is that something that's done in devstack? | |
| 18:43:35 | sean-k-mooney | yes | |
| 18:43:43 | sean-k-mooney | well no | |
| 18:43:47 | sean-k-mooney | in the devstack job | |
| 18:43:53 | melwitt | oh interesting | |
| 18:43:55 | sean-k-mooney | its done by a role in zuul-jobs | |
| 18:43:58 | melwitt | I'll go look at it | |
| 18:44:14 | melwitt | ahhh roles, that explains why I never saw anything | |
| 18:44:15 | sean-k-mooney | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge | |
| 18:44:45 | sean-k-mooney | ah its using ovs in the upstream verison i have a version of this that uses linux bridge | |
| 18:44:55 | melwitt | neat | |
| 18:45:29 | sean-k-mooney | if you have ever looked at the multi node nodeset you will see https://github.com/openstack/devstack/blob/master/.zuul.yaml#L133-L140 | |
| 18:45:47 | sean-k-mooney | the swtich is the contoler and the peers are the subnode | |
| 18:45:55 | sean-k-mooney | that use for cretating the tunnels and bridges | |
| 18:46:06 | sean-k-mooney | by https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge | |
| 18:46:32 | sean-k-mooney | so its not a full mesh but a set of point to point lings between the conotler node and the rest | |
| 18:46:55 | melwitt | I never knew what that did. that's cool | |
| 18:48:59 | sean-k-mooney | so i dont recall all the history but i knwo we use that to test vlan network sbut i belive its orgial use was to make sure we just had l2 connectivity betweeen each fo the subnodes as we coudl not alwasy depend on that | |
| 18:49:17 | sean-k-mooney | but like this role in some from has existied since we use jenkins | |
| 18:49:56 | sean-k-mooney | melwitt: so just looking at the job run https://zuul.opendev.org/t/openstack/build/dcc395a124c04f4fbc3368ddf97400d3/log/job-output.txt | |
| 18:50:08 | sean-k-mooney | what exactly was the issues the ovn setup ? | |
| 18:50:28 | sean-k-mooney | becasue the interfac echeted or soemthin else | |
| 18:51:11 | melwitt | tbh I don't really know other than the compute1 could not get to the /identity url on the controller. and I assumed that meant some aspect of networking was not connected somehow | |
| 18:51:49 | sean-k-mooney | :) as a compute person blamin gnetworkign and storage first alwasy means less work | |
| 18:52:05 | sean-k-mooney | but ya ill take a look quickly and ssee if i can see teh actual failure | |
| 18:52:39 | melwitt | the first problem was on compute1 it would never find a ipv6 interface because the awk command or whatever was not able to work with debian. so I made a devstack dnm patch for that. then it got farther to the point where compute1 is trying to do a very basic GET to /identity and it just hangs | |
| 18:54:08 | melwitt | the second problem was compute1 tls cert was for the controller commonName so it did not work (the code simply copied controller's certs to subnode) so I added to my dnm patch to make the subnode actually make a cert for itself, for its commonName. then it got farther and then crapped out trying to GET /identity | |
| 18:54:40 | sean-k-mooney | so we have a role that does that copy | |
| 18:54:44 | melwitt | at that point I didn't know what to do | |
| 18:54:58 | sean-k-mooney | but we dont normally need a cert for the compute | |
| 18:55:03 | sean-k-mooney | we just need the ca | |
| 18:55:09 | sean-k-mooney | so it trust the contoler | |
| 18:55:33 | melwitt | I got a cert validation error when subnode tried to contact controller so that's why I did it. oh, ok | |
| 18:55:58 | melwitt | not sure why what was there originally didn't work. maybe it was a one-off | |
| 18:56:40 | sean-k-mooney | maybe or maybe its because of how the name/ip is included in the cert | |
| 18:57:05 | sean-k-mooney | like if we are connetiv via the ipv6 adress rather then the name we might need to do soemthign differnt | |
| 18:57:10 | melwitt | yeah the complaint mentioned the commonName not matching or something like that | |
| 18:57:29 | sean-k-mooney | oh https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-hosts-file | |
| 18:57:43 | sean-k-mooney | so that only adds the ipv4 adresses to /etc/hosts | |
| 18:58:24 | sean-k-mooney | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-hosts-file/tasks/main.yaml so maybe that a factor | |
| 18:58:34 | melwitt | hm ok | |
| 18:59:29 | sean-k-mooney | what that does is loop over all the hsot in the inventory adn update /etc/host so that contoller maps to the correct ip | |
| 18:59:57 | sean-k-mooney | that may or may not be related jsut noticed the doc says its only for ipv4 which the code confirms | |
| 19:01:09 | melwitt | I see | |
| 19:05:08 | sean-k-mooney | thisd does nto prove anyting but if you look at the zuul-info.* files https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dcc/openstack/dcc395a124c04f4fbc3368ddf97400d3/zuul-info/index.html | |
| 19:05:44 | sean-k-mooney | what i do notice is the compute seams to have much hihger latency then the contoler | |
| 19:07:17 | sean-k-mooney | they both have the saem default route | |
| 19:07:47 | sean-k-mooney | btu they took difent path back to open dev (ill ignore the fact the ipv4 trace rotes is actully v6 in both) | |
| 19:08:23 | sean-k-mooney | we also see fe80::ac5a:77ff:fe03:adb2 dev ens3 lladdr ae:5a:77:03:ad:b2 router DELAY vs fe80::3457:e8ff:fe88:9852 dev ens3 lladdr 36:57:e8:88:98:52 router REACHABLE | |
| 19:09:14 | sean-k-mooney | both those got a routabel ip in teh same ipv6 subnet 2001:41d0:302:1000::cbc/56 s | |
| 19:09:48 | sean-k-mooney | so they shoudl be able to comunicate fine | |
| 19:10:07 | melwitt | :/ | |
| 19:10:42 | sean-k-mooney | https://[2001:41d0:302:1000::1ab]/identity | |
| 19:10:53 | sean-k-mooney | so that what its trying to conenct too | |
| 19:11:05 | sean-k-mooney | but that is not the adress of the contoler | |
| 19:11:27 | sean-k-mooney | that is the adress of the compute | |
| 19:11:48 | sean-k-mooney | so the compute is trying to conenct to its own ipv6 adress to talk to keystone | |
| 19:11:58 | sean-k-mooney | and that is because SERVICE_HOST | |
| 19:12:01 | sean-k-mooney | is not set properly | |
| 19:13:00 | sean-k-mooney | your setting SERVICE_HOST: "" | |
| 19:13:05 | sean-k-mooney | that will work on singel node | |
| 19:13:36 | sean-k-mooney | but we actully need to set that to the ipv6 adress of the contoler | |
| 19:13:44 | melwitt | oh, bleh ok. | |
| 19:13:59 | sean-k-mooney | or the host naem in /etc/hosts need to resove to the ipv6 adres | |
| 19:14:01 | melwitt | I just copied it from the base ipv6 job out of desperation | |
| 19:14:05 | melwitt | ok. sorry about that | |
| 19:16:26 | sean-k-mooney | dont be sorry. but that why i didnt set service host in my orgianl patch | |
| 19:17:00 | sean-k-mooney | the orcstarte devstack role shoudl populate that for us | |
| 19:17:13 | melwitt | I wonder why does the base job do that then | |
| 19:17:15 | sean-k-mooney | but it may or may not work out of the box for ipv6 | |
| 19:17:39 | sean-k-mooney | melwitt: my gess it its a hack because it use the ipv4 adress or similar | |
| 19:17:48 | sean-k-mooney | but lets find out | |
| 19:19:47 | opendevreview | melanie witt proposed openstack/nova master: DNM test ceph ipv6 https://review.opendev.org/c/openstack/nova/+/984623 | |
| 19:20:40 | sean-k-mooney | so i tought it was configred via https://github.com/openstack/devstack/blob/03ece8f88040be9b0b14dd1cfe93076ad2419a80/playbooks/pre.yaml#L37 | |
| 19:20:47 | sean-k-mooney | but no... | |
| 19:22:09 | sean-k-mooney | ah https://github.com/openstack/devstack/blob/03ece8f88040be9b0b14dd1cfe93076ad2419a80/.zuul.yaml#L514 | |