Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-01-31
15:41:12 bauzas so I'm litterally hitting the same bumps without learning
16:23:01 melwitt sean-k-mooney: thanks for the reviews! I will be going through and addressing the comments. and I agree re: functional tests, I have been working on them and other issues
16:24:10 sean-k-mooney ack i was talking to stephen an he made a point that we coudl do the functional tests at the end fo the series
16:24:36 sean-k-mooney i was going to chat to you later and see how you felt about doing it in line or at the end
16:25:55 melwitt I thought about that too and I think ideally I would put them inline (so that one can look at the code alongside the func test showing it's working)
16:37:04 opendevreview Merged openstack/nova master: Revert "[pwmgmt]ignore missin governor when cpu_state used" https://review.opendev.org/c/openstack/nova/+/905671
19:16:59 melwitt sean-k-mooney: re: your comment about image backends reporting support ... currently I have each patch adding one action (trying to make review not a nightmare) but I'm not flipping the image backend flag to reporting support until the end. in real life it won't be runnable but in func tests it would. are you saying you would prefer to see the flag reporting support from the first patch and guarding all other actions to reject at the
19:17:01 melwitt API instead? and lift each gate with each subsequent patch?
19:17:47 sean-k-mooney i would have done it with the api gating approch and lifted it.
19:17:54 sean-k-mooney but given the code appears to be working
19:18:02 sean-k-mooney and we can test it with functional tests
19:18:11 sean-k-mooney im hesitent to actullly ask you to flip it
19:18:17 melwitt sean-k-mooney: ack. wanted to make sure I understood what you meant
19:18:47 sean-k-mooney so i woudl like to be able to test each "feature" patch either with tempest ro functional ideally
19:19:23 sean-k-mooney so if we can add in a few functional tests along the way then im fine with the current order
19:19:35 melwitt yeah, I see what you mean
19:20:09 sean-k-mooney i can be conviced to keep the functional tests at the end and keep the currrent order
19:20:34 sean-k-mooney i just want to actully make sure this works meaning testing it myself or havign enough automated coverage
19:20:48 sean-k-mooney i like unit tests but i dont trust unit test for complex things
19:20:59 melwitt for the next rev I'll not change the way to flag gets flipped but add all the func tests inline and see how that looks. if it's not convincing enough I could move to the API guarding pattern
19:21:40 sean-k-mooney ack lets start with that and ill update my devstack vm to yoru latest revision and do some testign form the tip of the series
19:22:01 sean-k-mooney i ment to ask bou about one of the patches that is WIP
19:22:26 melwitt yeah, I agree with you. I've been testing using the massive DNM patch at the end (though I broke something recently and am fixing it). just saying that's how I have any confidence in it :P that and my local devstack testing
19:22:29 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/905512/8
19:22:54 sean-k-mooney is that WIP because it does not work? or because you need more coverage or something else
19:22:57 melwitt yeah, that needs a lot more test coverage (unit and func) so I left it WIP
19:23:06 sean-k-mooney ack ok
19:23:19 sean-k-mooney do you think you can complete that before FF
19:23:35 melwitt I think so
19:23:43 sean-k-mooney im wonderign if that should go to the end with an api check? or if your confidnet we can merge that
19:24:33 sean-k-mooney ok then lets try and minimise the code curn by keepign the order and addign functional tests either inline or as followups at the end where you feel it fits best
19:24:59 melwitt it could. it's pretty simple really, the thing migrations needed to work is to have the libvirt secrets available on the destination and until that patch it wasn't creating them on the dest (and deleting them on the source)
19:25:23 sean-k-mooney ah ok
19:25:40 melwitt I've been trying to cut things down as small as possible because this stuff is complicated imho
19:26:00 sean-k-mooney i ran out os steam this morning but my goal is ot have reviewd each patch at least once by the end of the week
19:27:08 melwitt ack. I'm trying to get the next rev pushed sometime today/tonight
19:30:00 sean-k-mooney ok if you dont its not the end of the world. much of the content will be the same so i can still review to some degree
19:30:59 melwitt ack
20:00:59 sean-k-mooney ok i think its time for me to finish for today
20:01:05 sean-k-mooney o/
20:04:27 melwitt o/
#openstack-nova - 2024-02-01
00:31:13 tonyb Thanks to sean-k-mooney[m] I managed to reduce the list of stale/stuck/corrupt VMs from 28 to 12 (It took a while because I was *very* careful with my db manipulations)
00:32:37 tonyb Now I have 12 instances that are in the building/scheduling vm/task state.
00:33:23 tonyb Some have entries in nova_api.instace_mappings pointing at cell0 some at the real cell (which has a NULL name?)
00:34:41 tonyb AFAICT the instances don't have any allocations in placement
00:35:43 tonyb What is the "most correct"/"least horrible" DB update I can do to let nova know those instances are gone?
00:39:07 tonyb I was thinking of something like update instances set vm_state='ERROR', set task_state=???, set updated_at=now(), set deleted_at=now() where uuid in (....) ?
02:17:20 clarkb tonyb: I wonder if we just set vm state to active if a delete woudl work
03:16:51 tonyb I didn't try that.
05:10:45 opendevreview Merged openstack/nova stable/2023.2: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/906088
06:40:38 opendevreview Merged openstack/nova stable/2023.2: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/906089
08:37:23 sean-k-mooney[m] tonyb: to mark an instance as deleted in nova you set deleted=id
08:38:06 tonyb thanks that's basically what I did.
08:39:01 sean-k-mooney[m] so there is no way to take an instance from cell 0 and do anything other then delete it
08:39:32 sean-k-mooney[m] cell0 is basically a gravyard for instnce we were unable to boot
08:39:54 sean-k-mooney[m] and they should not have any allocaations in placement if they are in cell 0
08:40:19 sean-k-mooney[m] for the ones in schduling they have not had a host selected so they willl not be in any cell db
08:40:33 sean-k-mooney[m] they will just have a build request in the api db
08:41:06 sean-k-mooney[m] the ones in the real cell in building likely failed because of the rabbit outage
08:41:40 sean-k-mooney[m] i assume you just want them gone so you can boot new vms with nodepool
08:42:05 tonyb yup that's the aim. I did get rid of them
08:42:58 sean-k-mooney[m] if they are in building and are in a really cell then they likely have allocations so you would mark them as deleted in the cell db then delete there allocations in placement
08:43:15 sean-k-mooney[m] ok in that case are they all now cleanned up?
08:43:57 tonyb yup. I did discover some other nodes that have allocations but are gone.
08:44:25 tonyb I'll figure that out tomorrow, but it should be easy to find them in placement
08:44:33 sean-k-mooney[m] we have two related command nova-manage heal allocatiosn and audit
08:44:48 sean-k-mooney[m] the heal allocation command creates alloctions that are missing
08:45:04 sean-k-mooney[m] the audit command removes allocatons that should nolonger exist
08:45:15 tonyb oh! cool.
08:45:17 sean-k-mooney[m] and has a dry run mode by default that just prints them
08:45:37 tonyb that's perfect for tomorrow!
08:45:59 tonyb sean-k-mooney[m]: thanks for all your help
08:46:16 sean-k-mooney[m] https://github.com/openstack/nova/blob/stable/victoria/nova/cmd/manage.py#L2695
08:52:23 tonyb sean-k-mooney[m]++
09:14:59 gibi elodilles, sean-k-mooney[m]: could you check these backports please https://review.opendev.org/q/topic:%22power-mgmt-fixups%22+branch:stable/2023.2
09:15:09 gibi the master revert landed yesterday (finally)
09:15:40 sean-k-mooney[m] sure just reviewing one of mels patchs ill look at those next
09:19:10 gibi thanks
09:28:39 bauzas sean-k-mooney: gibi: got some devstack issue with ovn, because the package is missing
09:28:49 bauzas (RHEL9.3 here)
09:29:15 sean-k-mooney[m] its in a seperate repo call fast datapath on rhel
09:29:17 bauzas when looking at https://github.com/openstack/devstack/blob/master/tools/fixup_stuff.sh I see that centos9 installs centos-release-openstack-victoria package
09:29:21 sean-k-mooney[m] our you can just enabel the rdo repos
09:29:46 sean-k-mooney[m] i can give you the command for the rdo repos one sec
09:29:56 bauzas I can insdtall rdo-ovn
09:30:06 bauzas it says this is a wrapper for OVN
09:31:29 sean-k-mooney[m] https://github.com/openstack-k8s-operators/edpm-ansible/blob/28f80d2303497972a1d3b493760c4ff1557b973f/roles/edpm_nova/molecule/default/prepare.yml#L40
09:31:41 sean-k-mooney[m] if you dont have the repos then i belive that will fix it for you
09:32:18 sean-k-mooney[m] change -b antelope to master
09:32:43 bauzas https://paste.opendev.org/show/b95bS56avBrLTeoma0QU/
09:32:53 bauzas that's what I have
09:33:30 sean-k-mooney[m] ok then devstack already enabled rdo for you
09:33:38 bauzas correct
09:33:43 sean-k-mooney[m] how did it fail exactly
09:34:02 sean-k-mooney[m] you could just go back to ml2/ovs by the way
09:34:14 sean-k-mooney[m] but if you want to keep ovn
09:34:24 bauzas https://paste.opendev.org/show/bkHbTVtFyFTMA34VV1B7/
09:34:30 sean-k-mooney[m] i can take a look at the error with you in a few minutes
09:34:46 bauzas yeah I think I'll change my local.conf and play with ml2/ovs
09:35:25 bauzas (my local.conf is pretty simple, nothing is told about the networking)

Earlier   Later