| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-01 | |||
| 08:45:59 | tonyb | sean-k-mooney[m]: thanks for all your help | |
| 08:46:16 | sean-k-mooney[m] | https://github.com/openstack/nova/blob/stable/victoria/nova/cmd/manage.py#L2695 | |
| 08:52:23 | tonyb | sean-k-mooney[m]++ | |
| 09:14:59 | gibi | elodilles, sean-k-mooney[m]: could you check these backports please https://review.opendev.org/q/topic:%22power-mgmt-fixups%22+branch:stable/2023.2 | |
| 09:15:09 | gibi | the master revert landed yesterday (finally) | |
| 09:15:40 | sean-k-mooney[m] | sure just reviewing one of mels patchs ill look at those next | |
| 09:19:10 | gibi | thanks | |
| 09:28:39 | bauzas | sean-k-mooney: gibi: got some devstack issue with ovn, because the package is missing | |
| 09:28:49 | bauzas | (RHEL9.3 here) | |
| 09:29:15 | sean-k-mooney[m] | its in a seperate repo call fast datapath on rhel | |
| 09:29:17 | bauzas | when looking at https://github.com/openstack/devstack/blob/master/tools/fixup_stuff.sh I see that centos9 installs centos-release-openstack-victoria package | |
| 09:29:21 | sean-k-mooney[m] | our you can just enabel the rdo repos | |
| 09:29:46 | sean-k-mooney[m] | i can give you the command for the rdo repos one sec | |
| 09:29:56 | bauzas | I can insdtall rdo-ovn | |
| 09:30:06 | bauzas | it says this is a wrapper for OVN | |
| 09:31:29 | sean-k-mooney[m] | https://github.com/openstack-k8s-operators/edpm-ansible/blob/28f80d2303497972a1d3b493760c4ff1557b973f/roles/edpm_nova/molecule/default/prepare.yml#L40 | |
| 09:31:41 | sean-k-mooney[m] | if you dont have the repos then i belive that will fix it for you | |
| 09:32:18 | sean-k-mooney[m] | change -b antelope to master | |
| 09:32:43 | bauzas | https://paste.opendev.org/show/b95bS56avBrLTeoma0QU/ | |
| 09:32:53 | bauzas | that's what I have | |
| 09:33:30 | sean-k-mooney[m] | ok then devstack already enabled rdo for you | |
| 09:33:38 | bauzas | correct | |
| 09:33:43 | sean-k-mooney[m] | how did it fail exactly | |
| 09:34:02 | sean-k-mooney[m] | you could just go back to ml2/ovs by the way | |
| 09:34:14 | sean-k-mooney[m] | but if you want to keep ovn | |
| 09:34:24 | bauzas | https://paste.opendev.org/show/bkHbTVtFyFTMA34VV1B7/ | |
| 09:34:30 | sean-k-mooney[m] | i can take a look at the error with you in a few minutes | |
| 09:34:46 | bauzas | yeah I think I'll change my local.conf and play with ml2/ovs | |
| 09:35:25 | bauzas | (my local.conf is pretty simple, nothing is told about the networking) | |
| 09:35:44 | sean-k-mooney[m] | oh its the metadata agent not ovn | |
| 09:36:49 | sean-k-mooney[m] | you can always just trun that off and use config drive | |
| 09:37:05 | bauzas | that's my local.conf https://paste.opendev.org/show/bwqKtvfeZhUT9aL1kFoZ/ | |
| 09:37:39 | sean-k-mooney[m] | https://github.com/openstack/nova/blob/master/.zuul.yaml#L192-L201 | |
| 09:37:49 | sean-k-mooney[m] | that is how to swap to ml2/ovs | |
| 09:38:57 | bauzas | q-ovn-metadata-agent.service that's the service which failed | |
| 09:39:12 | sean-k-mooney[m] | oh you need https://github.com/openstack/nova/blob/master/.zuul.yaml#L186-L189 as well | |
| 09:39:27 | bauzas | oh wait, I have ovn installed | |
| 09:39:33 | sean-k-mooney[m] | yes | |
| 09:39:43 | sean-k-mooney[m] | its the neturon metadata agent for ovn that failed | |
| 09:39:52 | sean-k-mooney[m] | not the ovn install | |
| 09:40:00 | bauzas | yeah | |
| 09:40:14 | bauzas | I can try to dig into the logs of that agent, if I'm able to find them | |
| 09:40:27 | sean-k-mooney[m] | they are in the journal like all the rest | |
| 09:40:41 | sean-k-mooney[m] | so just journalctl -u devstack@whatever | |
| 09:41:31 | sean-k-mooney[m] | im stepping away for 5-10 minutes to grab coffee but when im back if you want me to ssh in and take a look with you i can | |
| 09:41:50 | bauzas | yeah | |
| 09:41:55 | bauzas | and thanks | |
| 09:48:58 | bauzas | ahah, found the culprit https://paste.opendev.org/show/bFigNVupZWRpjlmuzXZ9/ | |
| 09:50:21 | sahid | o/ regarding live migrations, we are agree that we can live-migrate vm from compute node that are N-1 to N ? | |
| 09:50:53 | sahid | in a situation of upgrading version of openstack | |
| 09:53:49 | bauzas | sure, not the other way | |
| 09:58:10 | sahid | ack thanks | |
| 10:12:56 | sean-k-mooney | bauzas: ah the privsep helper | |
| 10:13:36 | sean-k-mooney | bauzas: so tht issue is that its not on your path | |
| 10:13:59 | sean-k-mooney | bauzas: i put a path fix in devstack to fix that at one point | |
| 10:14:41 | bauzas | /usr/bin/which: no privsep-helper in (/opt/stack/.local/bin:/opt/stack/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin) | |
| 10:14:41 | bauzas | [stack@lenovo-sr655-01 devstack]$ which privsep-helper | |
| 10:14:47 | bauzas | hmmmm | |
| 10:16:09 | sean-k-mooney | it will be in the venv bin dir | |
| 10:16:24 | sean-k-mooney | if you have it disabled | |
| 10:16:46 | bauzas | I disabled the GLOBAL venv indeed | |
| 10:16:50 | sean-k-mooney | then it should be in /usr/local/bin but apprently not | |
| 10:17:14 | bauzas | I have oslo.privsep | |
| 10:17:21 | bauzas | lemme see where it's landed | |
| 10:17:49 | sean-k-mooney | https://github.com/openstack/devstack/blob/5c1736b78256f5da86a91c4489f43f8ba1bce224/stack.sh#L838 this is the rwrokaround for usign the venv | |
| 10:17:59 | sean-k-mooney | so you just need to do the same with where ever it is | |
| 10:18:33 | bauzas | I suspect that privsep got installed in the venv as I first started to stack.sh with this flag | |
| 10:18:57 | sean-k-mooney | ah ya so there are a bunch of things you have to manually clean up | |
| 10:18:59 | sean-k-mooney | if you did that | |
| 10:19:14 | sean-k-mooney | and removing those symlinks form /usr/local/bin is one of them | |
| 10:19:18 | bauzas | and after that, even when asking devstack to reclone, I tihnk it got a problem | |
| 10:19:28 | sean-k-mooney | reclone is not enaouch | |
| 10:19:37 | sean-k-mooney | so you need to delete the venv dir | |
| 10:19:55 | sean-k-mooney | and then remove all the broken symlinks form /usr/local/bin | |
| 10:20:14 | bauzas | yeah | |
| 10:20:16 | bauzas | [stack@lenovo-sr655-01 devstack]$ ll /usr/local/bin/privsep-helper | |
| 10:20:18 | bauzas | lrwxrwxrwx. 1 root root 39 Jan 31 11:10 /usr/local/bin/privsep-helper -> /opt/stack/data/venv/bin/privsep-helper | |
| 10:20:30 | sean-k-mooney | yep | |
| 10:20:46 | sean-k-mooney | but neutron is proably not in the venv | |
| 10:20:48 | bauzas | okay, I'll reinstall the package | |
| 10:21:07 | bauzas | and I'll look at other symlinks in /usr/local/bin | |
| 10:21:20 | sean-k-mooney | as i said nuke the venv first then remove any broken links | |
| 10:22:28 | sean-k-mooney | i didnt document the process but it took me about an hour to do that on centos properly but its doable to make work | |
| 10:23:48 | bauzas | yeah | |
| 10:23:54 | bauzas | thanks for the help | |
| 10:24:21 | sean-k-mooney | note i had to remove some egg link files too | |
| 10:24:38 | sean-k-mooney | what i basically did ws unstack.sh; clean.sh | |
| 10:24:48 | bauzas | yup, https://github.com/openstack/devstack/blob/master/tools/fixup_stuff.sh says it | |
| 10:24:54 | sean-k-mooney | remove all openstack related thigns form /usr/local/bin | |
| 10:25:26 | sean-k-mooney | and remove all openstack packages via pip and or egglink files | |
| 10:26:26 | sean-k-mooney | after that i cloud jsut stack aagain without the venv and it was fine | |
| 10:27:03 | sean-k-mooney | the venv is a huge improvement but goign between each mode is a pain | |
| 10:27:22 | bauzas | yeah and clean.sh doesn't do all the cleanup :) | |
| 10:27:45 | bauzas | I recommend everyone stacking a new env every quarter :) | |
| 10:27:47 | sean-k-mooney | it appreanly does not install ovs and ovn which i coudl have bet money on it doing in the past | |
| 10:27:54 | sean-k-mooney | that bit me a few days ago | |
| 10:28:32 | sean-k-mooney | i almost added that to clean but i had other things to do | |
| 10:28:54 | bauzas | sean-k-mooney: I wonder, I guess we can't easily run the zuul jobs on our local machines ? | |
| 10:29:16 | sean-k-mooney | well... i can kind of | |
| 10:29:37 | bauzas | it would be simplier for a new developer to just hook their dev machines into some zuul registery so they would just run a specific zuul job for installing devstack and its knobs | |