| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-31 | |||
| 20:01:05 | sean-k-mooney | o/ | |
| 20:04:27 | melwitt | o/ | |
| #openstack-nova - 2024-02-01 | |||
| 00:31:13 | tonyb | Thanks to sean-k-mooney[m] I managed to reduce the list of stale/stuck/corrupt VMs from 28 to 12 (It took a while because I was *very* careful with my db manipulations) | |
| 00:32:37 | tonyb | Now I have 12 instances that are in the building/scheduling vm/task state. | |
| 00:33:23 | tonyb | Some have entries in nova_api.instace_mappings pointing at cell0 some at the real cell (which has a NULL name?) | |
| 00:34:41 | tonyb | AFAICT the instances don't have any allocations in placement | |
| 00:35:43 | tonyb | What is the "most correct"/"least horrible" DB update I can do to let nova know those instances are gone? | |
| 00:39:07 | tonyb | I was thinking of something like update instances set vm_state='ERROR', set task_state=???, set updated_at=now(), set deleted_at=now() where uuid in (....) ? | |
| 02:17:20 | clarkb | tonyb: I wonder if we just set vm state to active if a delete woudl work | |
| 03:16:51 | tonyb | I didn't try that. | |
| 05:10:45 | opendevreview | Merged openstack/nova stable/2023.2: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/906088 | |
| 06:40:38 | opendevreview | Merged openstack/nova stable/2023.2: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/906089 | |
| 08:37:23 | sean-k-mooney[m] | tonyb: to mark an instance as deleted in nova you set deleted=id | |
| 08:38:06 | tonyb | thanks that's basically what I did. | |
| 08:39:01 | sean-k-mooney[m] | so there is no way to take an instance from cell 0 and do anything other then delete it | |
| 08:39:32 | sean-k-mooney[m] | cell0 is basically a gravyard for instnce we were unable to boot | |
| 08:39:54 | sean-k-mooney[m] | and they should not have any allocaations in placement if they are in cell 0 | |
| 08:40:19 | sean-k-mooney[m] | for the ones in schduling they have not had a host selected so they willl not be in any cell db | |
| 08:40:33 | sean-k-mooney[m] | they will just have a build request in the api db | |
| 08:41:06 | sean-k-mooney[m] | the ones in the real cell in building likely failed because of the rabbit outage | |
| 08:41:40 | sean-k-mooney[m] | i assume you just want them gone so you can boot new vms with nodepool | |
| 08:42:05 | tonyb | yup that's the aim. I did get rid of them | |
| 08:42:58 | sean-k-mooney[m] | if they are in building and are in a really cell then they likely have allocations so you would mark them as deleted in the cell db then delete there allocations in placement | |
| 08:43:15 | sean-k-mooney[m] | ok in that case are they all now cleanned up? | |
| 08:43:57 | tonyb | yup. I did discover some other nodes that have allocations but are gone. | |
| 08:44:25 | tonyb | I'll figure that out tomorrow, but it should be easy to find them in placement | |
| 08:44:33 | sean-k-mooney[m] | we have two related command nova-manage heal allocatiosn and audit | |
| 08:44:48 | sean-k-mooney[m] | the heal allocation command creates alloctions that are missing | |
| 08:45:04 | sean-k-mooney[m] | the audit command removes allocatons that should nolonger exist | |
| 08:45:15 | tonyb | oh! cool. | |
| 08:45:17 | sean-k-mooney[m] | and has a dry run mode by default that just prints them | |
| 08:45:37 | tonyb | that's perfect for tomorrow! | |
| 08:45:59 | tonyb | sean-k-mooney[m]: thanks for all your help | |
| 08:46:16 | sean-k-mooney[m] | https://github.com/openstack/nova/blob/stable/victoria/nova/cmd/manage.py#L2695 | |
| 08:52:23 | tonyb | sean-k-mooney[m]++ | |
| 09:14:59 | gibi | elodilles, sean-k-mooney[m]: could you check these backports please https://review.opendev.org/q/topic:%22power-mgmt-fixups%22+branch:stable/2023.2 | |
| 09:15:09 | gibi | the master revert landed yesterday (finally) | |
| 09:15:40 | sean-k-mooney[m] | sure just reviewing one of mels patchs ill look at those next | |
| 09:19:10 | gibi | thanks | |
| 09:28:39 | bauzas | sean-k-mooney: gibi: got some devstack issue with ovn, because the package is missing | |
| 09:28:49 | bauzas | (RHEL9.3 here) | |
| 09:29:15 | sean-k-mooney[m] | its in a seperate repo call fast datapath on rhel | |
| 09:29:17 | bauzas | when looking at https://github.com/openstack/devstack/blob/master/tools/fixup_stuff.sh I see that centos9 installs centos-release-openstack-victoria package | |
| 09:29:21 | sean-k-mooney[m] | our you can just enabel the rdo repos | |
| 09:29:46 | sean-k-mooney[m] | i can give you the command for the rdo repos one sec | |
| 09:29:56 | bauzas | I can insdtall rdo-ovn | |
| 09:30:06 | bauzas | it says this is a wrapper for OVN | |
| 09:31:29 | sean-k-mooney[m] | https://github.com/openstack-k8s-operators/edpm-ansible/blob/28f80d2303497972a1d3b493760c4ff1557b973f/roles/edpm_nova/molecule/default/prepare.yml#L40 | |
| 09:31:41 | sean-k-mooney[m] | if you dont have the repos then i belive that will fix it for you | |
| 09:32:18 | sean-k-mooney[m] | change -b antelope to master | |
| 09:32:43 | bauzas | https://paste.opendev.org/show/b95bS56avBrLTeoma0QU/ | |
| 09:32:53 | bauzas | that's what I have | |
| 09:33:30 | sean-k-mooney[m] | ok then devstack already enabled rdo for you | |
| 09:33:38 | bauzas | correct | |
| 09:33:43 | sean-k-mooney[m] | how did it fail exactly | |
| 09:34:02 | sean-k-mooney[m] | you could just go back to ml2/ovs by the way | |
| 09:34:14 | sean-k-mooney[m] | but if you want to keep ovn | |
| 09:34:24 | bauzas | https://paste.opendev.org/show/bkHbTVtFyFTMA34VV1B7/ | |
| 09:34:30 | sean-k-mooney[m] | i can take a look at the error with you in a few minutes | |
| 09:34:46 | bauzas | yeah I think I'll change my local.conf and play with ml2/ovs | |
| 09:35:25 | bauzas | (my local.conf is pretty simple, nothing is told about the networking) | |
| 09:35:44 | sean-k-mooney[m] | oh its the metadata agent not ovn | |
| 09:36:49 | sean-k-mooney[m] | you can always just trun that off and use config drive | |
| 09:37:05 | bauzas | that's my local.conf https://paste.opendev.org/show/bwqKtvfeZhUT9aL1kFoZ/ | |
| 09:37:39 | sean-k-mooney[m] | https://github.com/openstack/nova/blob/master/.zuul.yaml#L192-L201 | |
| 09:37:49 | sean-k-mooney[m] | that is how to swap to ml2/ovs | |
| 09:38:57 | bauzas | q-ovn-metadata-agent.service that's the service which failed | |
| 09:39:12 | sean-k-mooney[m] | oh you need https://github.com/openstack/nova/blob/master/.zuul.yaml#L186-L189 as well | |
| 09:39:27 | bauzas | oh wait, I have ovn installed | |
| 09:39:33 | sean-k-mooney[m] | yes | |
| 09:39:43 | sean-k-mooney[m] | its the neturon metadata agent for ovn that failed | |
| 09:39:52 | sean-k-mooney[m] | not the ovn install | |
| 09:40:00 | bauzas | yeah | |
| 09:40:14 | bauzas | I can try to dig into the logs of that agent, if I'm able to find them | |
| 09:40:27 | sean-k-mooney[m] | they are in the journal like all the rest | |
| 09:40:41 | sean-k-mooney[m] | so just journalctl -u devstack@whatever | |
| 09:41:31 | sean-k-mooney[m] | im stepping away for 5-10 minutes to grab coffee but when im back if you want me to ssh in and take a look with you i can | |
| 09:41:50 | bauzas | yeah | |
| 09:41:55 | bauzas | and thanks | |
| 09:48:58 | bauzas | ahah, found the culprit https://paste.opendev.org/show/bFigNVupZWRpjlmuzXZ9/ | |
| 09:50:21 | sahid | o/ regarding live migrations, we are agree that we can live-migrate vm from compute node that are N-1 to N ? | |
| 09:50:53 | sahid | in a situation of upgrading version of openstack | |
| 09:53:49 | bauzas | sure, not the other way | |
| 09:58:10 | sahid | ack thanks | |
| 10:12:56 | sean-k-mooney | bauzas: ah the privsep helper | |
| 10:13:36 | sean-k-mooney | bauzas: so tht issue is that its not on your path | |
| 10:13:59 | sean-k-mooney | bauzas: i put a path fix in devstack to fix that at one point | |
| 10:14:41 | bauzas | /usr/bin/which: no privsep-helper in (/opt/stack/.local/bin:/opt/stack/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin) | |
| 10:14:41 | bauzas | [stack@lenovo-sr655-01 devstack]$ which privsep-helper | |
| 10:14:47 | bauzas | hmmmm | |
| 10:16:09 | sean-k-mooney | it will be in the venv bin dir | |
| 10:16:24 | sean-k-mooney | if you have it disabled | |
| 10:16:46 | bauzas | I disabled the GLOBAL venv indeed | |
| 10:16:50 | sean-k-mooney | then it should be in /usr/local/bin but apprently not | |
| 10:17:14 | bauzas | I have oslo.privsep | |
| 10:17:21 | bauzas | lemme see where it's landed | |
| 10:17:49 | sean-k-mooney | https://github.com/openstack/devstack/blob/5c1736b78256f5da86a91c4489f43f8ba1bce224/stack.sh#L838 this is the rwrokaround for usign the venv | |
| 10:17:59 | sean-k-mooney | so you just need to do the same with where ever it is | |
| 10:18:33 | bauzas | I suspect that privsep got installed in the venv as I first started to stack.sh with this flag | |
| 10:18:57 | sean-k-mooney | ah ya so there are a bunch of things you have to manually clean up | |