| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-05 | |||
| 19:27:35 | dansmith | which is actually a problem because of how we recreate compute nodes and potentially can have conflicts with the provider name in placement | |
| 19:27:37 | sean-k-mooney | it will mostly come back form the dead | |
| 19:27:45 | dansmith | because the name will be the same, but the uuid will be different (currently) | |
| 19:27:57 | sean-k-mooney | but not fully | |
| 19:28:18 | sean-k-mooney | right the uuid will be differnt and naythign like pci claims will not be recreated | |
| 19:28:31 | sean-k-mooney | so it will come back in a broken state | |
| 19:29:07 | sean-k-mooney | unfortuntly if our customer have shown us anything its posible to run in that broken state for an extended period of time without noticing | |
| 19:29:14 | melwitt | yeah. I mean like regression coverage that deleting the service and restarting nova-compute with the new undelete will remain working | |
| 19:29:17 | dansmith | heh yeah | |
| 19:29:47 | sean-k-mooney | melwitt: well it will actully work better then it does today | |
| 19:30:00 | melwitt | like is this test the only place we test this or is it covered somewhere else already and this test isn't providing anything new other than uuid checking | |
| 19:30:14 | sean-k-mooney | but that does not mean we technially supprot it today or sould support it going forward | |
| 19:30:37 | dansmith | it sounds like melwitt wants a more generic test to validate that the de-zombification works today, even though it shouldn't be expected to, and that this series will not make it worse | |
| 19:30:50 | dansmith | yeah, that's my only complaint about writing that test, but perhaps I should just do it | |
| 19:31:00 | melwitt | so you're saying we do *not* support deleting a service and restarting nova-compute and having stuff still wowrk? | |
| 19:31:02 | melwitt | *work? | |
| 19:31:30 | sean-k-mooney | melwitt: thats what im saying as an operator you should not expect that to work | |
| 19:31:43 | dansmith | agree, not sure if we're explicit about it though | |
| 19:32:00 | sean-k-mooney | if you do not use any pci/numa stuff or have not vms on it at the time it will work | |
| 19:32:04 | dansmith | also not defending that as a good thing :) | |
| 19:32:08 | melwitt | sean-k-mooney: that seems so unexpected to me. sorry, I just had no idea. I thought they're supposed to be able to do that if the hostname stays the same | |
| 19:32:33 | sean-k-mooney | there is no expection that the compute node uuid would remain the same | |
| 19:32:39 | dansmith | melwitt: the reality is different I think | |
| 19:32:43 | melwitt | so if someone messes up and deletes a service and then says oops that was a mistake, then all those instances are expected not to work? | |
| 19:32:45 | sean-k-mooney | its a uuid4 and not based on the hostname/hypervior_hostname | |
| 19:32:52 | melwitt | dang | |
| 19:32:53 | dansmith | you can't delete a service with instances on it | |
| 19:33:10 | melwitt | ok, so that saves it I guess? ok | |
| 19:33:23 | sean-k-mooney | dansmith: are you sure | |
| 19:33:23 | dansmith | saves it from the single-click-mega-fail, but.. :) | |
| 19:33:27 | dansmith | pretty sure | |
| 19:33:52 | sean-k-mooney | ok cause i know we have code to loop over the allocation in placment and delete them before we delete the placment rp when teh compute serivce is deleted | |
| 19:34:05 | melwitt | just seems so harsh lol (if it were possible to delete the service while instances are on it) | |
| 19:34:07 | dansmith | yup | |
| 19:34:20 | sean-k-mooney | i guess that is just to prevent leaked allocation blocking the placment cleanup | |
| 19:35:37 | sean-k-mooney | ah https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/services.py#L269-L282 | |
| 19:35:43 | sean-k-mooney | we special case the nova-compute | |
| 19:36:02 | sean-k-mooney | so ya you cant delete it if it has instance | |
| 19:36:18 | melwitt | ok, well, if that's the case then I understand why and agree the test can be removed entirely. just seems so harsh, if what I was thinking were possible (and it is not possible bc we don't let you delete a service with instances mapped to it) | |
| 19:36:19 | sean-k-mooney | in which case provide the placment clean up happens properly it does not really matter if the uuid changes in that case | |
| 19:36:28 | sean-k-mooney | or if we undelete | |
| 19:37:07 | sean-k-mooney | https://github.com/openstack/nova/commit/42f62f1ed2ad76829eb9d40a8b9646a523f6381f | |
| 19:37:25 | sean-k-mooney | melwitt: it was only blokced in rocky it looks like | |
| 19:38:03 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1763183 | |
| 19:38:13 | melwitt | I think we (maybe I) backported it downstream | |
| 19:38:36 | sean-k-mooney | well it was backported upstream to pike | |
| 19:38:39 | melwitt | I just was not thinking about it or remembering it | |
| 19:38:47 | melwitt | ah ok | |
| 19:39:56 | sean-k-mooney | i rememebr being able to delete compute serivce with instance at one point but i feel like that is just because i mess up my local devstack not because i planed to do it | |
| 19:40:08 | dansmith | melwitt: here are the most service-delete-y tests we have in functional/ https://github.com/openstack/nova/blob/master/nova/tests/functional/wsgi/test_services.py#L119 | |
| 19:40:11 | melwitt | yeah you used to be able to | |
| 19:40:18 | dansmith | none of them ensure we can start an instance on the resurrected service, | |
| 19:40:28 | dansmith | although they do restart the compute to make sure it comes back up | |
| 19:40:43 | dansmith | which is the thing sean-k-mooney and laugh at outside a fake environment :P | |
| 19:41:08 | melwitt | I see, ok. thanks | |
| 19:41:22 | dansmith | melwitt: so your demand is me adding a test that a resurrected compute can fake boot a fake instance un a fake environment, and then I can delete this regression test, right? | |
| 19:41:26 | dansmith | (snarky on purpose, but serious) | |
| 19:41:45 | melwitt | sorry for the longer convo. I was very confused by the test and then I was erroneously thinking of an accidental service delete scenario | |
| 19:42:04 | dansmith | don't apologize | |
| 19:42:20 | melwitt | yeah, I said earlier I understand now and agree the test can be removed without loss of anything | |
| 19:42:24 | dansmith | the stuff I'm having to do in this set to make such a simple thing work is ridiculously incestuous | |
| 19:42:59 | dansmith | melwitt: well, I think adding a "and can boot something" thing to those ^ would make that a defensible position for me :) | |
| 19:43:01 | melwitt | I bet :\ | |
| 19:44:12 | melwitt | thanks for that 😂 | |
| 19:44:12 | melwitt | thanks for that 😂 | |
| 19:49:54 | sean-k-mooney | dansmith: alot of that likel come form how the fixture make restarting compute service work in the past | |
| 19:50:07 | dansmith | yes, I'm well aware | |
| 19:50:33 | melwitt | dansmith: I agree adding a "and can boot something" to those existing tests is a nice thing to cover. but I don't expect it to have to be part of your series, to be clear | |
| 19:51:02 | sean-k-mooney | with the stable uuid serise i am assuming you will have a functional test that start with an empty db and starts a comptue service with the uuid specifed in a file | |
| 19:51:33 | sean-k-mooney | you have a seperte test that delete it form teh db and starts it again if you wanted | |
| 19:52:51 | sean-k-mooney | but ya i think we agreed on nuke the thing and move on with your seriese | |
| 19:53:36 | melwitt | yes | |
| 19:55:05 | dansmith | well, I figure I need to add the other when I drop the regression test | |
| 19:55:21 | dansmith | there's something weird though about not seeing the provider get recreated after restarting the old compute, | |
| 19:55:27 | dansmith | although I see it happen in the logs | |
| 19:56:37 | sean-k-mooney | that happens after teh perodic task runs although it also happens i think in init host | |
| 19:57:06 | dansmith | I see it created before I look for it | |
| 19:57:29 | dansmith | https://pastebin.com/isqJXnfW | |
| 19:57:43 | dansmith | first line is it being created in our db, then placement, then the last one is looking for it, but it's missing | |
| 19:59:08 | dansmith | I kinda wonder if there's a bug causing us to find the old deleted compute node before the new one, and then return nothing because it's deleted | |
| 20:02:09 | dansmith | hah | |
| 20:02:10 | dansmith | 2023-01-05 12:01:58,067 INFO [nova.api.openstack.compute.hypervisors] Unable to find service for compute node host1. The service may be deleted and compute nodes need to be manually cleaned up. | |
| 20:02:37 | dansmith | that's what happens when I try to list hypervisors with the old name after re-starting the service | |
| 20:02:48 | dansmith | the service object should be undeleted, a new compute node was created, | |
| 20:03:01 | dansmith | yet listing doesn't include *either* because of that ^ | |
| 20:03:18 | dansmith | melwitt: see what we mean now? :) | |
| 20:03:56 | melwitt | 😵💫 | |
| 20:04:45 | sean-k-mooney | could this be related to the cell mappings | |
| 20:04:51 | sean-k-mooney | in the api db | |
| 20:05:25 | sean-k-mooney | as in does discover host need to be run | |
| 20:06:10 | dansmith | god I hope not | |
| 20:06:24 | sean-k-mooney | dansmith: by the way i do know that if the resouce tracker is broken the compute service can show up in the comptue service list but the compute node will not show up in the hypervior list | |
| 20:06:48 | sean-k-mooney | so if you run the test with OS_DEBUG maybe there is somethign breaking in the restart | |
| 20:07:36 | sean-k-mooney | i only see info logs in the output you pasted so fi this is from a functional test then you might need OS_DEBUG=1 | |
| 20:08:14 | dansmith | sure enough: Host 'host1' is not mapped to any cell | |
| 20:08:17 | sean-k-mooney | although if it was broken that way i woudl expect to see some trace backs or Error logs so debug should not be required | |
| 20:09:53 | dansmith | OS_DEBUG changed lately btw | |
| 20:10:03 | dansmith | I used to set OS_DEBUG=y but that doesn't work anymore | |
| 20:10:12 | dansmith | is =1 the new magic? | |
| 20:11:06 | sean-k-mooney | i have always used 1 but not sure if/when that changed | |