Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-06-23
20:16:21 sean-k-mooney erlon: lookign at ti brifly yes it looks more or less correct ill try and review it proeprly but probaly tomorrow
20:16:34 erlon nice, thanks
20:17:48 sean-k-mooney i see melwitt updated https://review.opendev.org/c/openstack/nova/+/984623/21 as well so we shoudl get som new reuslt of the ipv6 test in a few hours
20:19:35 melwitt yeah fingers crossed it doesn't run into another issue. this is the first time I try it with depends-on patches proposed to the proper repos. last fail was for database host using ipv4 address. I'll keep iterating on it
20:25:06 sean-k-mooney is there a reason you made the ipv6 connectivity opt in instead of opt out?
20:25:27 sean-k-mooney we often have ipv6 in the jobs
20:25:40 sean-k-mooney even if its not the ipv6 job
20:25:55 sean-k-mooney we just dont use those adress for the services by default
20:25:55 melwitt no reason other than to keep existing behavior the same
20:26:01 sean-k-mooney ack
20:26:17 sean-k-mooney im not sure which is the better default i just said i would ask
20:26:28 sean-k-mooney incasue it was because you hit a bug or something i was not aware of
20:26:59 melwitt oh yeah. not yet anyway haha
20:27:54 sean-k-mooney so if you respin you might want to mention ipv6 in the readme for that roles https://opendev.org/zuul/zuul-jobs/src/commit/6c0c643b8cffa1f61cf7af34f29d8a7e4d1bb114/roles/multi-node-bridge/README.rst
20:29:26 sean-k-mooney melwitt: they do not use molecule in that repo
20:29:32 sean-k-mooney but you may also want to exted https://opendev.org/zuul/zuul-jobs/src/commit/6c0c643b8cffa1f61cf7af34f29d8a7e4d1bb114/test-playbooks/multinode/multi-node-bridge.yaml
20:29:42 sean-k-mooney to test the ipv6 functionlatiy
20:30:02 sean-k-mooney but i dont really review code in that repo oftent so im not sure what the normal expecation are
20:30:06 melwitt good point, I didn't know about those
20:31:10 sean-k-mooney fungi and clarkb are inovled so i knwo there woudl be tests but i am not very familar with how they are setup but i guess this is just usign zuul to run the test playbooks directly
20:31:57 sean-k-mooney looks like tey are using tox and unittest to run them at a quick glance
20:32:26 sean-k-mooney although that may just be for the inline moduels
20:32:59 fungi yeah, clarkb can probably explain it better since he wrote a lot of it, but the idea initially was to have an abstract foundation for multi-node devstack jobs to build on
20:34:11 sean-k-mooney fungi: for the most part it works quite well. fungi context is melaine is adding supprot for multi node ipv6
20:34:23 fungi in particular, avoiding dependence on the network topology available to the test nodes themselves and setting up an encapsulated layer 2 network between them that would provide consistency
20:34:23 sean-k-mooney so we can test live mgiration with ipv6 ceph
20:35:13 fungi just to be clear, you want to tunnel v6 over the bridge (that shouldn't be a problem but maybe we baked in assumptions)
20:36:25 sean-k-mooney fungi: yep that the idea. melwitt has some devstack josb mostly workign with pre-playbooks in nova and is now tryign to do it properly in zuul-jobs and devstack
20:38:47 melwitt with barely any knowledge 😩 it just failed again. obviously I must have lost something while splitting out from the pre-playbooks into zuul-jobs. will have to figure this out
20:39:24 fungi right, afaik we didn't want that role to restrict what kind of traffic you can send over it, modulo obvious mtu adjustments
20:39:32 sean-k-mooney you may need to set bridge_configure_ipv6_address: true in the subnodes as well
20:39:50 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/984623/21/.zuul.yaml#416
20:39:52 fungi also i think there may be some limitations or at least limitations carrying broadcast and multicast traffic on it
20:40:07 sean-k-mooney fungi: its using vxlan
20:40:08 fungi er, or at least inefficiencies
20:40:10 sean-k-mooney so it shoud be fine
20:40:21 sean-k-mooney ya its a partial mesh using ovs
20:40:44 sean-k-mooney so all multicast becoms bradcast which si copied intrenlly t o all spokes in the mesh
20:40:51 fungi i think vxlan relies on multicast groups underneath to carry broadcasts though? and our cloud providers don't let the multicast traffic through
20:40:57 sean-k-mooney so its a littele inefficnet but it shoudl work
20:41:02 fungi ah okay
20:41:06 sean-k-mooney fungi: not with ovs
20:41:11 sean-k-mooney but with linux bridge yes
20:41:17 sean-k-mooney ovs only supprot unicast tunnels
20:41:28 fungi you are far more familiar with all of this than i am
20:41:46 sean-k-mooney but that actully makes it more portable since you do not rely on the cloud provdier supproting multicast grouping at the fabic level
20:42:21 fungi all i can really say is afaik we didn't intentionally omit support for sending ipv6 packets over the bridge, so if it's not working i'd consider that a bug worth fixing
20:42:25 sean-k-mooney it comes form addign ovs-dpdk suppreot neutron/nova and maintianing it for years:)
20:42:43 melwitt sean-k-mooney: oh yeah, that makes sense. in pre, I was doing it for hosts: all but didn't put under both
20:43:39 fungi `hosts: all` should run the same task on every test node (but not the zuul executor, which needs `hosts: localhost` instead, not really relevant to this playbook anyway)
20:45:57 sean-k-mooney fungi: i have also impletend my own version fo the role 2 before although the most recent version i got ai to do :)
20:46:28 sean-k-mooney fungi: https://github.com/SeanMooney/ard/tree/master/ansible/roles/ard_multinode_bridge does basically the same thign but it uses linux bridge and gre tunnels to avoid installign ovs
20:46:57 sean-k-mooney i used to just use the zuul jobs one
20:48:19 sean-k-mooney im not entrily sure im happy with how the new version works but i was tryign to make it compatiabe with our operator based isntaller and i dint want to fight with its installation and management of ovs
20:49:54 sean-k-mooney that fact that ovs survies reboot makes the ovs based approch we use upstream less complex in many ways
20:56:19 sean-k-mooney my the way the mtu caluation does have to be adjuscted for ipv6 but the role aslready subtratc 70 bytes which si enough to have ipv6 geneve trafic go over the bridge
20:56:23 sean-k-mooney melwitt: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge/tasks/common.yaml#L103-L139
20:58:00 melwitt hm ok
20:59:17 sean-k-mooney techinaly the 70 bytes of overad is the bare minium for geneve to work over the bridge
20:59:25 sean-k-mooney and it shoudl work for neutron in general
20:59:57 clarkb its not geneve its vxlan iirc
21:00:02 clarkb and that is why there isn't ipv6
21:00:04 sean-k-mooney i use a 100 byte safty buffer in my verions
21:00:07 fungi sean-k-mooney: yeah, we originally used gre i believe, but it isn't allowed in all out providers
21:00:22 clarkb there is/was work to convert it to geneve and drop the use of the ansible bridge module
21:00:24 sean-k-mooney clarkb: yes the overly is
21:00:32 fungi at least in some openstack clouds, the kernel conntrack plugin for gre isn't running
21:00:43 sean-k-mooney clarkb: actully that not the part the matters
21:00:59 sean-k-mooney we have other code in devstack playbooks to calulate teh neutron mtu config
21:01:13 sean-k-mooney that the one that need to accomidate geneve whwen ovn is used
21:01:15 clarkb https://review.opendev.org/c/zuul/zuul-jobs/+/959393
21:01:36 sean-k-mooney fungi: ya gre is not an ip protocol so its often blockded
21:01:51 clarkb yes the original implementation used gre and linux bridges
21:02:10 sean-k-mooney i depebated about gre or vxlan and for 8 bytpes of mtu vxlan is definly more portabl since its just udp form the network layer
21:02:15 clarkb that didn't work because we couldn't get gre to pass even with wide open security groups so we switched to vxlan as it is udp
21:02:39 sean-k-mooney clarkb: :) yep i remember
21:02:53 clarkb and now ansible has dropped supprot for much of the machinery that we relied on to setup vxlan and ovs so we're switching back to linux bridge and taking the opportunity to use geneve in that change
21:03:11 clarkb that work has clearly stalled out and probably should be picked up again. I think it is very close if not ready
21:04:33 clarkb and the reason we didn't support ipv6 is taht netiher vxlan nor gre could support it iirc
21:04:41 clarkb or maybe it was just vxlan that couldn't
21:04:47 sean-k-mooney both can
21:04:58 clarkb I think vxlan can now
21:05:00 sean-k-mooney at both the underlay and over the mesh
21:05:01 clarkb but it was a v4 only protocol
21:05:13 sean-k-mooney it dependied on the kernel version
21:05:19 sean-k-mooney and ovs version
21:05:24 clarkb in any case if you want to make changes like this getting 959393 in first is probably a prereq
21:05:36 sean-k-mooney vxlan suppoted it but we could supprot it on all distos we supproted if i recll
21:05:58 sean-k-mooney well it does not stricly ned the refacto to linux bridge
21:06:06 sean-k-mooney but that cerntly viable
21:06:09 clarkb it does not but I will nto review a chnge that creatse a merge conflict
21:06:17 clarkb maybe other people are willing to review it and land it
21:06:25 opendevreview melanie witt proposed openstack/nova master: Add non-voting IPv6 ceph live migration job https://review.opendev.org/c/openstack/nova/+/984623
21:06:28 sean-k-mooney that is one of my version of that https://github.com/SeanMooney/ard/tree/master/ansible/roles/ard_multinode_bridge
21:06:39 sean-k-mooney i have a much older version too somehwere
21:06:42 clarkb the current code only works because I vendored the entirety of the ansible module that is no longer supported into that role
21:07:05 sean-k-mooney ya i replaced all of tha twith a systemd unit file that calls a bashcript
21:07:05 clarkb we really need to stop using that stuff and leaving it as is is fine. But if you want to make changes then we should do the right thing to fix the underlying ansible stuff
21:07:13 sean-k-mooney https://github.com/SeanMooney/ard/blob/master/ansible/roles/ard_multinode_bridge/templates/ard-bridge-setup.sh.j2
21:07:24 clarkb and that work is basically done so there isn't any reason not to

Earlier   Later