| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-31 | |||
| 14:21:25 | bauzas | DONTs and the DOES | |
| 14:21:50 | sean-k-mooney | but lets loop back to this after Uggla has updated things | |
| 14:21:53 | sean-k-mooney | and we have more time | |
| 14:22:05 | bauzas | I think we said we should never try to fail on an instance delete | |
| 14:22:15 | bauzas | as there are also some capacity and billing implications | |
| 14:22:25 | sean-k-mooney | we can fail on isntance delete for other reasons im pretty sure | |
| 14:22:45 | sean-k-mooney | but you are correct on the capsity and billing implciations if we do too | |
| 14:22:55 | gibi | it is a trade off if we let the VM delete to happen but leak resources in the meantime you still get billing issues :D | |
| 14:23:12 | bauzas | as an operator, I'd be more angry if my instance would continue to exist because of a cheap open file not deleted | |
| 14:23:37 | bauzas | but I could be wrong | |
| 14:23:50 | gibi | bauzas: e.g. we delete your VM happily but fail to unlock the share in manila. So you are billed for a share you cannot use :) | |
| 14:24:03 | bauzas | hence the periodic :) | |
| 14:24:27 | gibi | which can be turned off : | |
| 14:24:28 | gibi | :) | |
| 14:24:34 | bauzas | I'm not saying we should leave the pipe leaking | |
| 14:24:53 | bauzas | what I'm saying is that there are bigger things to do than fixing the pipe | |
| 14:24:54 | sean-k-mooney | bauzas: would that same operator be ok if custoemr data was leaked via presitent memroy, or if vgpus were never made avaiable again or pci devices | |
| 14:24:58 | gibi | at least if my VM is in ERROR and I cannot delete it I will notice and raise a ticket | |
| 14:25:18 | sean-k-mooney | so there is a way to fix this | |
| 14:25:40 | sean-k-mooney | we coudl ask manilla to aloow the share delteion if the nova instnace nolonger exists | |
| 14:25:56 | sean-k-mooney | that coudl be a followup enhacnment on there side | |
| 14:26:24 | sean-k-mooney | so if its locked by nova but nova says the isntnace is gone aloow the user to delelete it even with the lock | |
| 14:26:31 | bauzas | gibi: sean-k-mooney: standing as a user for a sec, | |
| 14:26:42 | bauzas | if I delete my instance and it goes into ERROR | |
| 14:26:53 | bauzas | the second after, I'm gonna try to delete again | |
| 14:27:01 | bauzas | so I could be okay | |
| 14:27:01 | sean-k-mooney | yep which might work | |
| 14:27:04 | gibi | :) | |
| 14:27:21 | bauzas | provided we ensure that the instance deletion path is idempotent | |
| 14:27:38 | sean-k-mooney | yes but we already need to do that | |
| 14:28:05 | sean-k-mooney | i belive one path where instance delete ends with an instance in error | |
| 14:28:10 | sean-k-mooney | is if the connection to libvirt is down | |
| 14:29:23 | sean-k-mooney | or ironic/hyperv/vmware api is not accable | |
| 14:29:45 | sean-k-mooney | assuming the comptue agent is fine it will have to put it in error when it cant talk to the hypervior | |
| 14:30:00 | sean-k-mooney | well that or leave the instnace in active | |
| 14:30:05 | sean-k-mooney | and put the instnace action in errror | |
| 14:30:21 | sean-k-mooney | ok meetings starting | |
| 15:11:52 | bauzas | every time I'm trying to pull a devstack on a RHEL machine, I'm getting funny stories | |
| 15:24:58 | bauzas | Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Service Unavailable (HTTP 503) | |
| 15:24:58 | bauzas | Failed to discover available identity versions when contacting http://10.73.116.68/identity. Attempting to parse version from URL. | |
| 15:25:15 | bauzas | but when looking at keystone logs in the journal, lgtm | |
| 15:25:31 | bauzas | wtf... | |
| 15:27:04 | bauzas | I did set GLOBAL_VENV to False fwiw | |
| 15:29:56 | sean-k-mooney | well its technially not a supported operating system | |
| 15:30:10 | sean-k-mooney | i can likely take a look in a while if you are still having issues | |
| 15:35:18 | bauzas | SELinux is preventing /usr/sbin/httpd from write access on the sock_file keystone-wsgi-public.socket | |
| 15:35:21 | bauzas | doh | |
| 15:39:19 | sean-k-mooney | devstack disabels it on centos and fedora | |
| 15:39:28 | sean-k-mooney | so it proably is missing a check for rhel | |
| 15:39:43 | sean-k-mooney | that is proably a simple fix in devstack but also you can do it locally | |
| 15:40:21 | bauzas | I think I need to do some kind of install notes for kashyap and others wanting to test vGPUs with devstack | |
| 15:40:40 | bauzas | that's always good to reinstall your devstack | |
| 15:40:59 | bauzas | and I think I got the same problem 6 months ago but I forgot | |
| 15:41:12 | bauzas | so I'm litterally hitting the same bumps without learning | |
| 16:23:01 | melwitt | sean-k-mooney: thanks for the reviews! I will be going through and addressing the comments. and I agree re: functional tests, I have been working on them and other issues | |
| 16:24:10 | sean-k-mooney | ack i was talking to stephen an he made a point that we coudl do the functional tests at the end fo the series | |
| 16:24:36 | sean-k-mooney | i was going to chat to you later and see how you felt about doing it in line or at the end | |
| 16:25:55 | melwitt | I thought about that too and I think ideally I would put them inline (so that one can look at the code alongside the func test showing it's working) | |
| 16:37:04 | opendevreview | Merged openstack/nova master: Revert "[pwmgmt]ignore missin governor when cpu_state used" https://review.opendev.org/c/openstack/nova/+/905671 | |
| 19:16:59 | melwitt | sean-k-mooney: re: your comment about image backends reporting support ... currently I have each patch adding one action (trying to make review not a nightmare) but I'm not flipping the image backend flag to reporting support until the end. in real life it won't be runnable but in func tests it would. are you saying you would prefer to see the flag reporting support from the first patch and guarding all other actions to reject at the | |
| 19:17:01 | melwitt | API instead? and lift each gate with each subsequent patch? | |
| 19:17:47 | sean-k-mooney | i would have done it with the api gating approch and lifted it. | |
| 19:17:54 | sean-k-mooney | but given the code appears to be working | |
| 19:18:02 | sean-k-mooney | and we can test it with functional tests | |
| 19:18:11 | sean-k-mooney | im hesitent to actullly ask you to flip it | |
| 19:18:17 | melwitt | sean-k-mooney: ack. wanted to make sure I understood what you meant | |
| 19:18:47 | sean-k-mooney | so i woudl like to be able to test each "feature" patch either with tempest ro functional ideally | |
| 19:19:23 | sean-k-mooney | so if we can add in a few functional tests along the way then im fine with the current order | |
| 19:19:35 | melwitt | yeah, I see what you mean | |
| 19:20:09 | sean-k-mooney | i can be conviced to keep the functional tests at the end and keep the currrent order | |
| 19:20:34 | sean-k-mooney | i just want to actully make sure this works meaning testing it myself or havign enough automated coverage | |
| 19:20:48 | sean-k-mooney | i like unit tests but i dont trust unit test for complex things | |
| 19:20:59 | melwitt | for the next rev I'll not change the way to flag gets flipped but add all the func tests inline and see how that looks. if it's not convincing enough I could move to the API guarding pattern | |
| 19:21:40 | sean-k-mooney | ack lets start with that and ill update my devstack vm to yoru latest revision and do some testign form the tip of the series | |
| 19:22:01 | sean-k-mooney | i ment to ask bou about one of the patches that is WIP | |
| 19:22:26 | melwitt | yeah, I agree with you. I've been testing using the massive DNM patch at the end (though I broke something recently and am fixing it). just saying that's how I have any confidence in it :P that and my local devstack testing | |
| 19:22:29 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/905512/8 | |
| 19:22:54 | sean-k-mooney | is that WIP because it does not work? or because you need more coverage or something else | |
| 19:22:57 | melwitt | yeah, that needs a lot more test coverage (unit and func) so I left it WIP | |
| 19:23:06 | sean-k-mooney | ack ok | |
| 19:23:19 | sean-k-mooney | do you think you can complete that before FF | |
| 19:23:35 | melwitt | I think so | |
| 19:23:43 | sean-k-mooney | im wonderign if that should go to the end with an api check? or if your confidnet we can merge that | |
| 19:24:33 | sean-k-mooney | ok then lets try and minimise the code curn by keepign the order and addign functional tests either inline or as followups at the end where you feel it fits best | |
| 19:24:59 | melwitt | it could. it's pretty simple really, the thing migrations needed to work is to have the libvirt secrets available on the destination and until that patch it wasn't creating them on the dest (and deleting them on the source) | |
| 19:25:23 | sean-k-mooney | ah ok | |
| 19:25:40 | melwitt | I've been trying to cut things down as small as possible because this stuff is complicated imho | |
| 19:26:00 | sean-k-mooney | i ran out os steam this morning but my goal is ot have reviewd each patch at least once by the end of the week | |
| 19:27:08 | melwitt | ack. I'm trying to get the next rev pushed sometime today/tonight | |
| 19:30:00 | sean-k-mooney | ok if you dont its not the end of the world. much of the content will be the same so i can still review to some degree | |
| 19:30:59 | melwitt | ack | |
| 20:00:59 | sean-k-mooney | ok i think its time for me to finish for today | |
| 20:01:05 | sean-k-mooney | o/ | |
| 20:04:27 | melwitt | o/ | |
| #openstack-nova - 2024-02-01 | |||
| 00:31:13 | tonyb | Thanks to sean-k-mooney[m] I managed to reduce the list of stale/stuck/corrupt VMs from 28 to 12 (It took a while because I was *very* careful with my db manipulations) | |
| 00:32:37 | tonyb | Now I have 12 instances that are in the building/scheduling vm/task state. | |
| 00:33:23 | tonyb | Some have entries in nova_api.instace_mappings pointing at cell0 some at the real cell (which has a NULL name?) | |
| 00:34:41 | tonyb | AFAICT the instances don't have any allocations in placement | |
| 00:35:43 | tonyb | What is the "most correct"/"least horrible" DB update I can do to let nova know those instances are gone? | |
| 00:39:07 | tonyb | I was thinking of something like update instances set vm_state='ERROR', set task_state=???, set updated_at=now(), set deleted_at=now() where uuid in (....) ? | |