| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-24 | |||
| 08:16:48 | bauzas | fwiw, I'm more than happy to welcome volunteers wanting to work on https://blueprints.launchpad.net/nova/+spec/detach-boot-volume in particular if they want this is important for their users | |
| 08:17:19 | bauzas | I can offer nova onboarding and knowledge transfer if that helps | |
| 08:17:48 | bauzas | I'm paid for leading a community, but I need this community to enact by itself :) | |
| 08:41:29 | bauzas | simondodsley: fwiw, we have an PTG operator hour that's proposed today at 1500UTC https://ptg.opendev.org/ptg.html | |
| 08:41:51 | bauzas | simondodsley: if you feel you'd like to discuss your feature request, feel free to join | |
| 08:42:27 | noonedeadpunk | hey folks. I get weird error on HEAD of stable/2023.1 when trying to rebuild the instance: https://paste.openstack.org/show/biUIcOzMCx0YlsFob2KK/ | |
| 08:42:29 | bauzas | and like I said, if I can help someone volunteering, I'll surely offer onboarding | |
| 08:43:02 | bauzas | noonedeadpunk: eek | |
| 08:43:14 | noonedeadpunk | I decided to ask if it's smth known already so that I won't spend time on looking deeper (as for me it's not very obvious from the trace) | |
| 08:46:32 | noonedeadpunk | yeah, detach-boot-volume is really interesting thing, though I can recall it being not as trivial as it might look at the beginning | |
| 08:46:38 | bauzas | noonedeadpunk: this is a RPC call | |
| 08:46:56 | noonedeadpunk | ok, so I should check on.. conductor I guess | |
| 08:47:19 | bauzas | this sounds a compat issue between conductor and compute | |
| 08:47:36 | bauzas | basically your compute awaits some argument | |
| 08:47:42 | bauzas | that conductor isn't passing | |
| 08:47:52 | bauzas | I guess you upgraded ? | |
| 08:48:02 | bauzas | do you have a rolling upgrade in lieu ? | |
| 08:50:04 | bauzas | noonedeadpunk: that's the client RPC code that does the RPC negociation https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L1106-L1110 | |
| 08:50:30 | bauzas | this client code is run by the nova-conductor service | |
| 08:50:42 | bauzas | or by a compute service if this is compute-to-compute calls | |
| 08:50:45 | noonedeadpunk | Yeah, I am. And I see version 66 for every service in `services` table of cell db | |
| 08:51:35 | noonedeadpunk | oh, wait, couple of computes are still 60.... Hm | |
| 08:51:55 | noonedeadpunk | Not the affected one though, but still smth I need to fix | |
| 08:52:46 | bauzas | do you have some rpc pins ? | |
| 08:54:01 | noonedeadpunk | nah, I don't | |
| 08:54:13 | noonedeadpunk | Or well, should not have - never set them | |
| 08:54:15 | bauzas | https://github.com/openstack/nova/blob/stable/2023.1/nova/conductor/manager.py#L1395-L1414 is where conductor calls compute for rebuilding | |
| 08:55:05 | noonedeadpunk | Ok, computes that have eariler version are down right now... | |
| 08:55:09 | bauzas | the conductor manager is then invoking compute.rpcapi's rebuild_instance() which does RPC negociation | |
| 08:55:33 | bauzas | and then remove target_state if computes are older than rpc 6.2 | |
| 08:57:58 | noonedeadpunk | bauzas: is there any easy way to see reported prc version? becauce I guess in nova.services it's a different version rather then RPC. | |
| 08:58:00 | bauzas | noonedeadpunk: https://github.com/openstack/nova/commit/8c2e76598995f0d417653c60a63ea342baf4e880 is the commit that adds RPC 6.2 | |
| 08:58:24 | bauzas | noonedeadpunk: good question | |
| 08:58:58 | bauzas | I'd say internal RPC versions aren't meant to be exposed, but we provide aliases | |
| 08:59:20 | bauzas | https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L428 | |
| 08:59:45 | bauzas | zed computes are exposing 6.1, antelope ones are 6.2 | |
| 09:00:00 | noonedeadpunk | I guess I was more willing to check what computes/conductor reports as versions | |
| 09:00:08 | bauzas | if both your conductor *and* computes are 6.2, then try out to pin the rpc version | |
| 09:00:17 | bauzas | yeah, got it | |
| 09:00:28 | bauzas | the service version should tho give you that info | |
| 09:00:46 | noonedeadpunk | so if all is 66 - it should kinda work... | |
| 09:00:54 | bauzas | https://github.com/openstack/nova/commit/8c2e76598995f0d417653c60a63ea342baf4e880#diff-c0b6a5928be3ac40200a2078b084341bb9187a12b1f959ad862e0038c9029193 | |
| 09:01:09 | bauzas | 66 is the object version that matches RPC 6.2 | |
| 09:01:11 | noonedeadpunk | As iirc 66 is 2023.1 | |
| 09:01:14 | noonedeadpunk | yeah... | |
| 09:01:44 | bauzas | are you sure you restarted all your conductors after the upgrade ? :) | |
| 09:01:54 | bauzas | sorry for that silly question but I had to | |
| 09:02:02 | noonedeadpunk | They won't report 66 otherwise? | |
| 09:02:06 | bauzas | good call | |
| 09:02:17 | noonedeadpunk | I was actually trying to find a way to check what I could forgot to restart :) | |
| 09:02:31 | noonedeadpunk | But I'm checking now.... | |
| 09:02:51 | bauzas | I guess you can find the error in the conductor log, isn't it ? | |
| 09:03:02 | bauzas | if so, identify the conductor service in cause | |
| 09:10:04 | noonedeadpunk | Well, I've checked ps on all conductor hosts - all running from same venv and venv have same nova version installed... | |
| 09:10:21 | noonedeadpunk | And logs are clean as conductor doesn't obviously see any issues | |
| 09:10:40 | noonedeadpunk | let me check if code does contain these patches you've sent me | |
| 09:14:26 | noonedeadpunk | yeah, it's obviously there for all conductors... | |
| 09:21:10 | noonedeadpunk | Well, upgrade check throws `hw_machine_type unset` warning. Will try to sort it out - not sure if that's related though | |
| 09:23:51 | bauzas | no, this isn't | |
| 09:25:19 | bauzas | noonedeadpunk: since it's a venv, can you force to add some log in the conditional ? | |
| 09:25:52 | noonedeadpunk | sure | |
| 09:26:18 | bauzas | oh waity | |
| 09:26:27 | bauzas | this is a rebuild revert, right? | |
| 09:26:39 | bauzas | uh no | |
| 09:27:00 | bauzas | we're just calling the decorator in case rebuild_instance raises an exception | |
| 09:27:40 | bauzas | speaking out loud, | |
| 09:27:55 | bauzas | the RPC check on conductor considers that compute is older | |
| 09:28:38 | bauzas | but this is also not passing the target_state parameter or we would have an exception | |
| 09:29:31 | noonedeadpunk | no, it's not revert, it's bassically just `openstack server rebuild <server_uuid> --image <image_uuid>` | |
| 09:29:48 | bauzas | yeah yeah | |
| 09:30:13 | bauzas | it's popping up in the deco just because this is wrapping the call itself | |
| 09:30:21 | noonedeadpunk | But yes, looking at code you've shown I tend to agreee that issue is in rpc version, but it's very confusing, given that in DB it's reported as 66... And I pretty sure that before upgrade it was 61 or smth | |
| 09:30:40 | bauzas | but that means the RPC call doesn't contain the expected target_state arg | |
| 09:30:56 | bauzas | so something happens in the negociation | |
| 09:31:17 | bauzas | compute says "give me target_state" but conductor doesn't offer it | |
| 09:31:33 | noonedeadpunk | let me try to restart compute maybe, just in case.... | |
| 09:32:09 | bauzas | I'd say the other way, a conductor restart | |
| 09:32:22 | bauzas | compute is new enough to require target_state | |
| 09:32:52 | noonedeadpunk | yeah, makes sense... | |
| 09:33:30 | bauzas | and https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/manager.py#L621 exposes 6.2 I just checked | |
| 09:34:19 | bauzas | so yeah, I'd add a few debugging logs in the compute rpcapi method | |
| 09:34:31 | bauzas | (well, if this was my env, I'd pdb :D) | |
| 09:34:48 | noonedeadpunk | `Oct 24 09:34:35 control01-nova-api-container-13281850 nova-conductor[4131712]: 2023-10-24 09:34:35.835 4131712 INFO nova.compute.rpcapi [None req-28a1e0a6-3f71-4238-825d-b508d33ac758 - - - - - -] Automatically selected compute RPC version 6.1 from minimum service version 64` | |
| 09:34:51 | noonedeadpunk | o_O | |
| 09:34:59 | bauzas | but logging the object states before the RPC call seems a way forward | |
| 09:35:32 | bauzas | ahaha ! | |
| 09:35:36 | noonedeadpunk | Hm... Maybe that\s a compute which is down making things weird | |
| 09:35:41 | bauzas | no | |
| 09:36:10 | bauzas | I have to think about it, sec | |
| 09:36:25 | noonedeadpunk | Well. Compute that is down still having version 64 | |
| 09:36:42 | noonedeadpunk | So it can bring negotiation to that version? | |
| 09:37:29 | noonedeadpunk | I pretty much can bet it's the case... | |
| 09:37:46 | bauzas | oh, wait a sec | |
| 09:38:15 | noonedeadpunk | the compute was down during the upgrade | |
| 09:38:18 | bauzas | I need to remember my rpc compat skilles | |
| 09:38:23 | bauzas | skills* | |
| 09:39:07 | bauzas | yeah, so the thing is,https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L504-L505 gives you 64 | |
| 09:39:34 | bauzas | but then, the manager shouldn't yell | |
| 09:39:34 | bauzas | so we're capping to the minimum version | |
| 09:39:52 | bauzas | I'm missing something obvious but I'm rusty | |