| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-19 | |||
| 12:55:38 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 12:55:39 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | |
| 13:34:17 | opendevreview | Merged openstack/nova stable/2023.1: [ironic] Partition & use cache for list_instance* https://review.opendev.org/c/openstack/nova/+/922155 | |
| 14:03:15 | elodilles | bauzas: now that you are editing .zuul.yaml: there are some warnings regarding deprecated regex syntax. if that irritates you, here is a fix for that: https://review.opendev.org/c/openstack/nova/+/922212 o:) | |
| 14:05:10 | bauzas | elodilles: I'm just fighting with multinode zuul :) | |
| 14:07:14 | elodilles | yeah i saw that o:) | |
| 14:08:34 | bauzas | sean-k-mooney: I maybe need some help for debugging my mtty job https://59eb0550fb97fc327d84-51181049425e2f372c12ca87aad48047.ssl.cf2.rackcdn.com/922140/8/experimental/nova-mtty/8550c60/controller/logs/screen-br-ex-tcpdump.txt | |
| 14:08:45 | bauzas | the multinode job fails ^ | |
| 14:12:38 | bauzas | everytime you think you can understand OVN, then you find something you don't know | |
| 14:14:58 | bauzas | just saying, the VTEP failed : https://59eb0550fb97fc327d84-51181049425e2f372c12ca87aad48047.ssl.cf2.rackcdn.com/922140/8/experimental/nova-mtty/8550c60/controller/logs/ovn/ovn-controller-vtep_log.txt | |
| 14:15:30 | bauzas | maybe it's related if the public IPs are directly connected thru northbound OVN with br-ex | |
| 14:20:22 | sean-k-mooney | well you can trun off the tcpdump service | |
| 14:20:30 | sean-k-mooney | its just an extra dbuging thing for the ci | |
| 14:20:45 | sean-k-mooney | and its really only used by the neuron team | |
| 14:20:57 | sean-k-mooney | so for right now you could disabel it as a work around | |
| 14:21:25 | bauzas | k | |
| 14:21:42 | sean-k-mooney | ill quickly take a look at the devstack log and see if anything jumps out | |
| 14:23:46 | sean-k-mooney | ya so looking at the ovs-vswitchd log on the compute https://59eb0550fb97fc327d84-51181049425e2f372c12ca87aad48047.ssl.cf2.rackcdn.com/922140/8/experimental/nova-mtty/8550c60/compute1/logs/openvswitch/ovs-vswitchd_log.txt | |
| 14:23:50 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 14:23:50 | sean-k-mooney | there is not br-ex | |
| 14:23:50 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | |
| 14:23:52 | sean-k-mooney | which is normal | |
| 14:24:20 | bauzas | I just disabled the service, we'll see | |
| 14:24:32 | sean-k-mooney | so for ml2-ovs in DVR (distirbuted virtural router) mode we also create an instance of the l3 agent on all the compute and create a br-ex | |
| 14:24:47 | sean-k-mooney | but for ovn we do not use the l3 agent and the br-ex is only created on the contoler | |
| 14:25:00 | sean-k-mooney | so you basicaly shoudl not run that service on the subnode | |
| 14:25:11 | sean-k-mooney | you can just disable it on both if you want however | |
| 14:25:54 | sean-k-mooney | on the contoler the bride does exist https://59eb0550fb97fc327d84-51181049425e2f372c12ca87aad48047.ssl.cf2.rackcdn.com/922140/8/experimental/nova-mtty/8550c60/controller/logs/openvswitch/ovs-vswitchd_log.txt | |
| 14:26:44 | sean-k-mooney | and tcpdump was installed | |
| 14:30:32 | sean-k-mooney | bauzas: there is noting obvious in the journal or in the devstack log but my guess is eitehr there has been a command line change or something like apparmour is causing the comand to fail | |
| 14:31:04 | bauzas | ack thanks for digging | |
| 14:31:13 | sean-k-mooney | im not sure what 203/EXEC is evstack@br-ex-tcpdump.service: Main process exited, code=exited, status=203/EXEC | |
| 14:34:28 | bauzas_ | yeah, that's a weird error code | |
| 14:34:30 | bauzas_ | shit my bouncer restarted the conn | |
| 14:34:32 | bauzas_ | shit my bouncer restarted the conn | |
| 16:13:44 | bauzas | https://zuul.opendev.org/t/openstack/build/32d394d8265e4cc188fc3c5a50d33993 🎉 | |
| 16:20:57 | bauzas | dang, Tempest is creating new flavors and they don't ask for VGPUs | |
| 16:21:17 | bauzas | 2024-06-19 15:18:47.346 | ++ lib/tempest:configure_tempest:298 : openstack --os-cloud devstack-admin flavor create --id 42 --ram 128 --disk 1 --vcpus 1 --property hw_rng:allowed=True m1.nano | |
| 16:22:00 | bauzas | sean-k-mooney: do you think I could change my own ansible script in https://review.opendev.org/c/openstack/nova/+/922140/9/playbooks/nova-mtty/pre.yaml to do post.yaml for updating all the flavors ? | |
| 16:23:00 | bauzas | https://d2ba671e5a990e8f3d73-f7c3a0f9889b6932e080ae1513d0dd0b.ssl.cf1.rackcdn.com/922140/9/experimental/nova-mtty/32d394d/controller/logs/devstacklog.txt => here I see that my pre.yaml script updates all the pre-existing flavors with resources:VGPU=1 | |
| 16:23:05 | bauzas | 2024-06-19 15:17:53.434 | ++ opt/stack/devstack/local.sh:main:8 : openstack flavor set --property resources:VGPU=1 1 | |
| 16:23:13 | sean-k-mooney | you could but the flavors used by tempest by defautl have know ides | |
| 16:23:19 | sean-k-mooney | namely 42 an 84 | |
| 16:23:30 | sean-k-mooney | which are m1.nano and m1.micro | |
| 16:23:41 | sean-k-mooney | so you dont need to update all of them you can just update the two used by tempest | |
| 16:24:19 | sean-k-mooney | local.sh will be run after tempest has created all the flavors | |
| 16:26:03 | bauzas | sean-k-mooney: if I use post.yaml tbc, right? | |
| 16:26:10 | sean-k-mooney | no | |
| 16:26:30 | sean-k-mooney | local.sh is used by devstack and included after tempest is configured but before tempest test are run | |
| 16:26:31 | bauzas | because pre.yaml runs before tempest creates its own flavors | |
| 16:26:43 | sean-k-mooney | pre runs before the run playbook | |
| 16:26:48 | sean-k-mooney | so be fore devstack is run | |
| 16:27:41 | sean-k-mooney | local.sh is run here https://github.com/openstack/devstack/blob/master/stack.sh#L1485-L1492 | |
| 16:27:56 | bauzas | 2024-06-19 15:18:24.159 | ++ lib/tempest:install_tempest:815 : git_clone https://opendev.org/openstack/tempest.git /opt/stack/tempest master | |
| 16:28:09 | bauzas | compared to 2024-06-19 15:17:50.794 | + ./stack.sh:main:1489 : [[ -x /opt/stack/devstack/local.sh ]] | |
| 16:28:27 | bauzas | so pre-run is called before install_tempest | |
| 16:28:42 | sean-k-mooney | pre-run is called before the run playbook | |
| 16:28:54 | sean-k-mooney | so its run after all the parent jobs pre playbooks | |
| 16:29:04 | sean-k-mooney | and be fore the current jobs run playbook | |
| 16:29:14 | sean-k-mooney | so its before stack.sh is invoked | |
| 16:29:30 | sean-k-mooney | the proable i think is your using the wrong path | |
| 16:29:32 | sean-k-mooney | on sec | |
| 16:29:36 | sean-k-mooney | your using /opt/stack/devstack/local.sh | |
| 16:30:17 | bauzas | which is expected path, right? | |
| 16:30:35 | sean-k-mooney | not in ci | |
| 16:30:46 | sean-k-mooney | that is not where devstack will be cloned | |
| 16:32:03 | bauzas | https://github.com/openstack/devstack/blob/master/stack.sh#L1489 is where my /opt/stack/devstack/local.sh script is called | |
| 16:32:11 | bauzas | 2024-06-19 15:17:50.794 | + ./stack.sh:main:1489 : [[ -x /opt/stack/devstack/local.sh ]] | |
| 16:32:20 | sean-k-mooney | oh actully i gues its correct | |
| 16:32:22 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/679656/13/playbooks/nfv/nfv.yaml | |
| 16:32:30 | sean-k-mooney | oh right | |
| 16:32:40 | bauzas | but alas, install_tempest runs *after* stack.sh | |
| 16:32:57 | sean-k-mooney | it should not | |
| 16:33:11 | sean-k-mooney | we install tempet as part of devstack | |
| 16:34:04 | bauzas | yeah I'm confused | |
| 16:34:13 | sean-k-mooney | do you have a link to th job | |
| 16:34:18 | bauzas | yeah | |
| 16:34:29 | bauzas | https://zuul.opendev.org/t/openstack/build/32d394d8265e4cc188fc3c5a50d33993 | |
| 16:39:06 | sean-k-mooney | im about to finish for today but ill take a look beofre i drop | |
| 16:40:28 | bauzas | I think I found why | |
| 16:40:34 | bauzas | 2024-06-19 15:18:23.933 | + ./stack.sh:main:1517 : run_phase stack test-config | |
| 16:40:45 | bauzas | then runs 2024-06-19 15:18:23.963 | ++ functions-common:run_phase:1860 : basename /opt/stack/devstack/extras.d/80-tempest.sh | |
| 16:41:13 | bauzas | which then does some async_wait 2024-06-19 15:18:24.047 | ++ extras.d/80-tempest.sh:source:20 : async_wait install_tempest | |
| 16:41:49 | bauzas | which then installs tempest 2024-06-19 15:18:24.159 | ++ lib/tempest:install_tempest:815 : git_clone https://opendev.org/openstack/tempest.git /opt/stack/tempest master | |
| 16:49:34 | sean-k-mooney | that sounds like a regression in devstack | |
| 16:51:12 | bauzas | maybe I should create the flavors and then configure tempest to use them | |
| 16:52:22 | sean-k-mooney | i have doen that in the past too | |
| 16:52:25 | sean-k-mooney | it works | |
| 16:52:37 | bauzas | do you have some job using it ? | |
| 16:52:38 | sean-k-mooney | thats what i normally do https://review.opendev.org/c/openstack/nova/+/679656/13/playbooks/nfv/nfv.yaml | |
| 16:52:50 | sean-k-mooney | bauzas: not that we merged | |
| 16:53:08 | sean-k-mooney | but i created multi numa testing jobs a few times in the past | |
| 16:53:12 | sean-k-mooney | just to show it workign | |
| 16:53:35 | bauzas | cool | |
| 16:53:52 | bauzas | I already knew how to use another flavor for Tempest https://docs.openstack.org/tempest/latest/configuration.html#flavors | |
| 16:54:20 | bauzas | but I found how to modify the config by a CI job with your change : https://review.opendev.org/c/openstack/nova/+/679656/13/.zuul.yaml | |
| 16:54:22 | bauzas | thanks | |
| 16:54:47 | bauzas | I think it's better than just modifying all of the flavors | |
| 16:55:01 | bauzas | anyway, I'm done too | |
| 16:55:13 | bauzas | I'll change my stuff tomorrow morning | |