| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-24 | |||
| 09:01:11 | noonedeadpunk | As iirc 66 is 2023.1 | |
| 09:01:14 | noonedeadpunk | yeah... | |
| 09:01:44 | bauzas | are you sure you restarted all your conductors after the upgrade ? :) | |
| 09:01:54 | bauzas | sorry for that silly question but I had to | |
| 09:02:02 | noonedeadpunk | They won't report 66 otherwise? | |
| 09:02:06 | bauzas | good call | |
| 09:02:17 | noonedeadpunk | I was actually trying to find a way to check what I could forgot to restart :) | |
| 09:02:31 | noonedeadpunk | But I'm checking now.... | |
| 09:02:51 | bauzas | I guess you can find the error in the conductor log, isn't it ? | |
| 09:03:02 | bauzas | if so, identify the conductor service in cause | |
| 09:10:04 | noonedeadpunk | Well, I've checked ps on all conductor hosts - all running from same venv and venv have same nova version installed... | |
| 09:10:21 | noonedeadpunk | And logs are clean as conductor doesn't obviously see any issues | |
| 09:10:40 | noonedeadpunk | let me check if code does contain these patches you've sent me | |
| 09:14:26 | noonedeadpunk | yeah, it's obviously there for all conductors... | |
| 09:21:10 | noonedeadpunk | Well, upgrade check throws `hw_machine_type unset` warning. Will try to sort it out - not sure if that's related though | |
| 09:23:51 | bauzas | no, this isn't | |
| 09:25:19 | bauzas | noonedeadpunk: since it's a venv, can you force to add some log in the conditional ? | |
| 09:25:52 | noonedeadpunk | sure | |
| 09:26:18 | bauzas | oh waity | |
| 09:26:27 | bauzas | this is a rebuild revert, right? | |
| 09:26:39 | bauzas | uh no | |
| 09:27:00 | bauzas | we're just calling the decorator in case rebuild_instance raises an exception | |
| 09:27:40 | bauzas | speaking out loud, | |
| 09:27:55 | bauzas | the RPC check on conductor considers that compute is older | |
| 09:28:38 | bauzas | but this is also not passing the target_state parameter or we would have an exception | |
| 09:29:31 | noonedeadpunk | no, it's not revert, it's bassically just `openstack server rebuild <server_uuid> --image <image_uuid>` | |
| 09:29:48 | bauzas | yeah yeah | |
| 09:30:13 | bauzas | it's popping up in the deco just because this is wrapping the call itself | |
| 09:30:21 | noonedeadpunk | But yes, looking at code you've shown I tend to agreee that issue is in rpc version, but it's very confusing, given that in DB it's reported as 66... And I pretty sure that before upgrade it was 61 or smth | |
| 09:30:40 | bauzas | but that means the RPC call doesn't contain the expected target_state arg | |
| 09:30:56 | bauzas | so something happens in the negociation | |
| 09:31:17 | bauzas | compute says "give me target_state" but conductor doesn't offer it | |
| 09:31:33 | noonedeadpunk | let me try to restart compute maybe, just in case.... | |
| 09:32:09 | bauzas | I'd say the other way, a conductor restart | |
| 09:32:22 | bauzas | compute is new enough to require target_state | |
| 09:32:52 | noonedeadpunk | yeah, makes sense... | |
| 09:33:30 | bauzas | and https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/manager.py#L621 exposes 6.2 I just checked | |
| 09:34:19 | bauzas | so yeah, I'd add a few debugging logs in the compute rpcapi method | |
| 09:34:31 | bauzas | (well, if this was my env, I'd pdb :D) | |
| 09:34:48 | noonedeadpunk | `Oct 24 09:34:35 control01-nova-api-container-13281850 nova-conductor[4131712]: 2023-10-24 09:34:35.835 4131712 INFO nova.compute.rpcapi [None req-28a1e0a6-3f71-4238-825d-b508d33ac758 - - - - - -] Automatically selected compute RPC version 6.1 from minimum service version 64` | |
| 09:34:51 | noonedeadpunk | o_O | |
| 09:34:59 | bauzas | but logging the object states before the RPC call seems a way forward | |
| 09:35:32 | bauzas | ahaha ! | |
| 09:35:36 | noonedeadpunk | Hm... Maybe that\s a compute which is down making things weird | |
| 09:35:41 | bauzas | no | |
| 09:36:10 | bauzas | I have to think about it, sec | |
| 09:36:25 | noonedeadpunk | Well. Compute that is down still having version 64 | |
| 09:36:42 | noonedeadpunk | So it can bring negotiation to that version? | |
| 09:37:29 | noonedeadpunk | I pretty much can bet it's the case... | |
| 09:37:46 | bauzas | oh, wait a sec | |
| 09:38:15 | noonedeadpunk | the compute was down during the upgrade | |
| 09:38:18 | bauzas | I need to remember my rpc compat skilles | |
| 09:38:23 | bauzas | skills* | |
| 09:39:07 | bauzas | yeah, so the thing is,https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L504-L505 gives you 64 | |
| 09:39:34 | bauzas | but then, the manager shouldn't yell | |
| 09:39:34 | bauzas | so we're capping to the minimum version | |
| 09:39:52 | bauzas | I'm missing something obvious but I'm rusty | |
| 09:40:23 | noonedeadpunk | I kinda wonder if `get_minimum_version` should account only for ones that are up... | |
| 09:41:16 | noonedeadpunk | But yeah, also if that minimum version is fine, then computes shouldn't fail because version is too low either I guess... | |
| 09:41:19 | bauzas | no no | |
| 09:41:23 | bauzas | this is a bug | |
| 09:41:27 | bauzas | I just checked | |
| 09:41:39 | bauzas | we broke rolling-upgrade compat, that's it | |
| 09:41:57 | noonedeadpunk | Should I submit one then? | |
| 09:42:03 | bauzas | the manager should default target_state=None | |
| 09:42:26 | bauzas | because we automatically pin our RPC calls to the min version | |
| 09:42:47 | bauzas | I'll doublecheck that with dansmith but I bet this is the problem | |
| 09:43:25 | noonedeadpunk | fwiw, dropping that old compute from services made negitoation pass | |
| 09:43:33 | noonedeadpunk | `Automatically selected compute RPC version 6.2 from minimum service version 66` | |
| 09:43:37 | noonedeadpunk | and rebuild worked out | |
| 09:43:37 | bauzas | and https://github.com/openstack/nova/commit/30aab9c234035b49c7e2cdc940f624a63eeffc1b#diff-47eb12598e353b9e0689707d7b477353200d0aa3ed13045ffd3d017ee7d9e753R3709 has the same problem | |
| 09:45:29 | bauzas | yeah, 64 is zed | |
| 09:45:51 | noonedeadpunk | Ok, I can submit a bug report if you think it's a bug, and I have a way around the issue as well :) | |
| 09:45:58 | bauzas | but basically, rebuild is broken on a rolling-upgrade from Zed to Antelope | |
| 09:46:10 | bauzas | noonedeadpunk: sure, file it please | |
| 09:46:48 | bauzas | and I think rebuild is also broken from Yoga to Zed btw. :) | |
| 09:47:26 | noonedeadpunk | Yes, another region we were upgrading from Yoga was also broken... | |
| 09:47:40 | noonedeadpunk | But it's unofficially supported upgrade, so I took better example :p | |
| 09:47:40 | bauzas | on rebuild ? | |
| 09:47:48 | noonedeadpunk | Yeah, same error | |
| 09:47:57 | bauzas | but not the same argument I guess | |
| 09:48:01 | noonedeadpunk | The only that has rebuild working is the one that has no computes down | |
| 09:48:23 | bauzas | from yoga to zed, it should yell at reimage_boot_volume missing | |
| 09:48:30 | noonedeadpunk | I actually wasn't digging there yet to be frank, as reported by users issues were the same from their side | |
| 09:49:09 | bauzas | I have to doublecheck but I think we're missing to test rebuild with grenade | |
| 09:50:32 | bauzas | anyway, I need to disappear, I have a few toxins to sweat before we start the PTG | |
| 09:53:48 | noonedeadpunk | sure, thanks a lot for the help, report is on it's way | |
| 09:58:06 | noonedeadpunk | eventually, since I found out about `hw_machine_type` what you would suggest adding? `q35` for all images basically? | |
| 10:15:42 | noonedeadpunk | bug: https://bugs.launchpad.net/nova/+bug/2040264 | |
| 11:39:00 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/2023.1: Translate VF network capabilities to port binding https://review.opendev.org/c/openstack/nova/+/898945 | |
| 12:09:20 | sean-k-mooney | noonedeadpunk: i have not read back fully but one of the main issues with detachign a boot drive is it leave the instance in a fundementally broken state | |
| 12:10:22 | sean-k-mooney | if we were every to supprot that i htink we woudl need to have a new state for the vm to model that. and or allow away to say use a seperate disk as the boot device. | |
| 12:10:48 | noonedeadpunk | Well, shelved_offloaded should work kinda? | |
| 12:10:55 | sean-k-mooney | it also kind of messes with the resouce allcoatins if you dont follwo our guidance and use flavor with 0 disk_gb for bfv guests | |
| 12:11:11 | noonedeadpunk | But I guess challange there was also to record what the drive is detached to re-attach it as root lately | |
| 12:11:31 | sean-k-mooney | noonedeadpunk: shelve_offloaded does not work because we cant unshelve if you have detached the volume and attached it to somethign else | |
| 12:11:54 | noonedeadpunk | but it would be a valid failure that VM can't be offloaded | |
| 12:12:22 | sean-k-mooney | ya so it really comes down to the usecase | |
| 12:12:23 | noonedeadpunk | sean-k-mooney: sorry, I didn't get part about "use flavor with 0 disk_gb for bfv guests" - can you elaborate a bit on this part? | |
| 12:12:44 | noonedeadpunk | As I thought that for bfv you're kinda good at having 0 disk_gb? | |