| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-31 | |||
| 19:17:47 | sean-k-mooney | i would have done it with the api gating approch and lifted it. | |
| 19:17:54 | sean-k-mooney | but given the code appears to be working | |
| 19:18:02 | sean-k-mooney | and we can test it with functional tests | |
| 19:18:11 | sean-k-mooney | im hesitent to actullly ask you to flip it | |
| 19:18:17 | melwitt | sean-k-mooney: ack. wanted to make sure I understood what you meant | |
| 19:18:47 | sean-k-mooney | so i woudl like to be able to test each "feature" patch either with tempest ro functional ideally | |
| 19:19:23 | sean-k-mooney | so if we can add in a few functional tests along the way then im fine with the current order | |
| 19:19:35 | melwitt | yeah, I see what you mean | |
| 19:20:09 | sean-k-mooney | i can be conviced to keep the functional tests at the end and keep the currrent order | |
| 19:20:34 | sean-k-mooney | i just want to actully make sure this works meaning testing it myself or havign enough automated coverage | |
| 19:20:48 | sean-k-mooney | i like unit tests but i dont trust unit test for complex things | |
| 19:20:59 | melwitt | for the next rev I'll not change the way to flag gets flipped but add all the func tests inline and see how that looks. if it's not convincing enough I could move to the API guarding pattern | |
| 19:21:40 | sean-k-mooney | ack lets start with that and ill update my devstack vm to yoru latest revision and do some testign form the tip of the series | |
| 19:22:01 | sean-k-mooney | i ment to ask bou about one of the patches that is WIP | |
| 19:22:26 | melwitt | yeah, I agree with you. I've been testing using the massive DNM patch at the end (though I broke something recently and am fixing it). just saying that's how I have any confidence in it :P that and my local devstack testing | |
| 19:22:29 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/905512/8 | |
| 19:22:54 | sean-k-mooney | is that WIP because it does not work? or because you need more coverage or something else | |
| 19:22:57 | melwitt | yeah, that needs a lot more test coverage (unit and func) so I left it WIP | |
| 19:23:06 | sean-k-mooney | ack ok | |
| 19:23:19 | sean-k-mooney | do you think you can complete that before FF | |
| 19:23:35 | melwitt | I think so | |
| 19:23:43 | sean-k-mooney | im wonderign if that should go to the end with an api check? or if your confidnet we can merge that | |
| 19:24:33 | sean-k-mooney | ok then lets try and minimise the code curn by keepign the order and addign functional tests either inline or as followups at the end where you feel it fits best | |
| 19:24:59 | melwitt | it could. it's pretty simple really, the thing migrations needed to work is to have the libvirt secrets available on the destination and until that patch it wasn't creating them on the dest (and deleting them on the source) | |
| 19:25:23 | sean-k-mooney | ah ok | |
| 19:25:40 | melwitt | I've been trying to cut things down as small as possible because this stuff is complicated imho | |
| 19:26:00 | sean-k-mooney | i ran out os steam this morning but my goal is ot have reviewd each patch at least once by the end of the week | |
| 19:27:08 | melwitt | ack. I'm trying to get the next rev pushed sometime today/tonight | |
| 19:30:00 | sean-k-mooney | ok if you dont its not the end of the world. much of the content will be the same so i can still review to some degree | |
| 19:30:59 | melwitt | ack | |
| 20:00:59 | sean-k-mooney | ok i think its time for me to finish for today | |
| 20:01:05 | sean-k-mooney | o/ | |
| 20:04:27 | melwitt | o/ | |
| #openstack-nova - 2024-02-01 | |||
| 00:31:13 | tonyb | Thanks to sean-k-mooney[m] I managed to reduce the list of stale/stuck/corrupt VMs from 28 to 12 (It took a while because I was *very* careful with my db manipulations) | |
| 00:32:37 | tonyb | Now I have 12 instances that are in the building/scheduling vm/task state. | |
| 00:33:23 | tonyb | Some have entries in nova_api.instace_mappings pointing at cell0 some at the real cell (which has a NULL name?) | |
| 00:34:41 | tonyb | AFAICT the instances don't have any allocations in placement | |
| 00:35:43 | tonyb | What is the "most correct"/"least horrible" DB update I can do to let nova know those instances are gone? | |
| 00:39:07 | tonyb | I was thinking of something like update instances set vm_state='ERROR', set task_state=???, set updated_at=now(), set deleted_at=now() where uuid in (....) ? | |
| 02:17:20 | clarkb | tonyb: I wonder if we just set vm state to active if a delete woudl work | |
| 03:16:51 | tonyb | I didn't try that. | |
| 05:10:45 | opendevreview | Merged openstack/nova stable/2023.2: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/906088 | |
| 06:40:38 | opendevreview | Merged openstack/nova stable/2023.2: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/906089 | |
| 08:37:23 | sean-k-mooney[m] | tonyb: to mark an instance as deleted in nova you set deleted=id | |
| 08:38:06 | tonyb | thanks that's basically what I did. | |
| 08:39:01 | sean-k-mooney[m] | so there is no way to take an instance from cell 0 and do anything other then delete it | |
| 08:39:32 | sean-k-mooney[m] | cell0 is basically a gravyard for instnce we were unable to boot | |
| 08:39:54 | sean-k-mooney[m] | and they should not have any allocaations in placement if they are in cell 0 | |
| 08:40:19 | sean-k-mooney[m] | for the ones in schduling they have not had a host selected so they willl not be in any cell db | |
| 08:40:33 | sean-k-mooney[m] | they will just have a build request in the api db | |
| 08:41:06 | sean-k-mooney[m] | the ones in the real cell in building likely failed because of the rabbit outage | |
| 08:41:40 | sean-k-mooney[m] | i assume you just want them gone so you can boot new vms with nodepool | |
| 08:42:05 | tonyb | yup that's the aim. I did get rid of them | |
| 08:42:58 | sean-k-mooney[m] | if they are in building and are in a really cell then they likely have allocations so you would mark them as deleted in the cell db then delete there allocations in placement | |
| 08:43:15 | sean-k-mooney[m] | ok in that case are they all now cleanned up? | |
| 08:43:57 | tonyb | yup. I did discover some other nodes that have allocations but are gone. | |
| 08:44:25 | tonyb | I'll figure that out tomorrow, but it should be easy to find them in placement | |
| 08:44:33 | sean-k-mooney[m] | we have two related command nova-manage heal allocatiosn and audit | |
| 08:44:48 | sean-k-mooney[m] | the heal allocation command creates alloctions that are missing | |
| 08:45:04 | sean-k-mooney[m] | the audit command removes allocatons that should nolonger exist | |
| 08:45:15 | tonyb | oh! cool. | |
| 08:45:17 | sean-k-mooney[m] | and has a dry run mode by default that just prints them | |
| 08:45:37 | tonyb | that's perfect for tomorrow! | |
| 08:45:59 | tonyb | sean-k-mooney[m]: thanks for all your help | |
| 08:46:16 | sean-k-mooney[m] | https://github.com/openstack/nova/blob/stable/victoria/nova/cmd/manage.py#L2695 | |
| 08:52:23 | tonyb | sean-k-mooney[m]++ | |
| 09:14:59 | gibi | elodilles, sean-k-mooney[m]: could you check these backports please https://review.opendev.org/q/topic:%22power-mgmt-fixups%22+branch:stable/2023.2 | |
| 09:15:09 | gibi | the master revert landed yesterday (finally) | |
| 09:15:40 | sean-k-mooney[m] | sure just reviewing one of mels patchs ill look at those next | |
| 09:19:10 | gibi | thanks | |
| 09:28:39 | bauzas | sean-k-mooney: gibi: got some devstack issue with ovn, because the package is missing | |
| 09:28:49 | bauzas | (RHEL9.3 here) | |
| 09:29:15 | sean-k-mooney[m] | its in a seperate repo call fast datapath on rhel | |
| 09:29:17 | bauzas | when looking at https://github.com/openstack/devstack/blob/master/tools/fixup_stuff.sh I see that centos9 installs centos-release-openstack-victoria package | |
| 09:29:21 | sean-k-mooney[m] | our you can just enabel the rdo repos | |
| 09:29:46 | sean-k-mooney[m] | i can give you the command for the rdo repos one sec | |
| 09:29:56 | bauzas | I can insdtall rdo-ovn | |
| 09:30:06 | bauzas | it says this is a wrapper for OVN | |
| 09:31:29 | sean-k-mooney[m] | https://github.com/openstack-k8s-operators/edpm-ansible/blob/28f80d2303497972a1d3b493760c4ff1557b973f/roles/edpm_nova/molecule/default/prepare.yml#L40 | |
| 09:31:41 | sean-k-mooney[m] | if you dont have the repos then i belive that will fix it for you | |
| 09:32:18 | sean-k-mooney[m] | change -b antelope to master | |
| 09:32:43 | bauzas | https://paste.opendev.org/show/b95bS56avBrLTeoma0QU/ | |
| 09:32:53 | bauzas | that's what I have | |
| 09:33:30 | sean-k-mooney[m] | ok then devstack already enabled rdo for you | |
| 09:33:38 | bauzas | correct | |
| 09:33:43 | sean-k-mooney[m] | how did it fail exactly | |
| 09:34:02 | sean-k-mooney[m] | you could just go back to ml2/ovs by the way | |
| 09:34:14 | sean-k-mooney[m] | but if you want to keep ovn | |
| 09:34:24 | bauzas | https://paste.opendev.org/show/bkHbTVtFyFTMA34VV1B7/ | |
| 09:34:30 | sean-k-mooney[m] | i can take a look at the error with you in a few minutes | |
| 09:34:46 | bauzas | yeah I think I'll change my local.conf and play with ml2/ovs | |
| 09:35:25 | bauzas | (my local.conf is pretty simple, nothing is told about the networking) | |
| 09:35:44 | sean-k-mooney[m] | oh its the metadata agent not ovn | |
| 09:36:49 | sean-k-mooney[m] | you can always just trun that off and use config drive | |
| 09:37:05 | bauzas | that's my local.conf https://paste.opendev.org/show/bwqKtvfeZhUT9aL1kFoZ/ | |
| 09:37:39 | sean-k-mooney[m] | https://github.com/openstack/nova/blob/master/.zuul.yaml#L192-L201 | |
| 09:37:49 | sean-k-mooney[m] | that is how to swap to ml2/ovs | |
| 09:38:57 | bauzas | q-ovn-metadata-agent.service that's the service which failed | |
| 09:39:12 | sean-k-mooney[m] | oh you need https://github.com/openstack/nova/blob/master/.zuul.yaml#L186-L189 as well | |
| 09:39:27 | bauzas | oh wait, I have ovn installed | |