| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-23 | |||
| 21:04:58 | clarkb | I'm not saying its the main implementation/ I'm just giving my experience as a user. boot from volume is more problematic than the value we get out of it so we try to use volumes more explicitly for where the data lives rather than our operating systems which are actually quite ephemeral | |
| 21:05:11 | clarkb | which is similar to how containers deal with this stuff too | |
| 21:05:57 | clarkb | For example rescuing boot from volume instances is still a bit of an ordeal | |
| 21:08:11 | clarkb | I shouldtest that again actually | |
| 21:13:34 | simondodsley | using boot from volume can be highly advantagous to users as external arrays can both deduplicate and compress volumes on the array and so 100s of VMs can boot from snapshot clones of the boot image, thereby actually using only the size of the source volume on the array. | |
| 21:13:52 | simondodsley | this is why we need features like changing the boot volume to be available. | |
| 21:14:27 | simondodsley | yes, cinder can do restore from snapshot, but that is limited as well. | |
| #openstack-nova - 2023-10-24 | |||
| 01:10:28 | sean-k-mooney[m] | simondodsley: one thing to keep in mind is openstack and nova in particular is not an enterprise virtualisation platform and vmware is not a compator, aws or azure is a better comparison | |
| 01:12:50 | sean-k-mooney[m] | ignoreing the fact ware are likely to remove the vmware driver shortly, vmware is one of the backend hyperviors you can use with nova. but nova is a cloud compute project so feature parity with enterpirse virtualisation software is not a goal for nova | |
| 07:10:23 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova stable/2023.2: Install lxml before we need it in post-run https://review.opendev.org/c/openstack/nova/+/899142 | |
| 07:10:33 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova stable/2023.2: Revert "Add upgrade check for compute-object-ids linkage" https://review.opendev.org/c/openstack/nova/+/898953 | |
| 08:16:48 | bauzas | fwiw, I'm more than happy to welcome volunteers wanting to work on https://blueprints.launchpad.net/nova/+spec/detach-boot-volume in particular if they want this is important for their users | |
| 08:17:19 | bauzas | I can offer nova onboarding and knowledge transfer if that helps | |
| 08:17:48 | bauzas | I'm paid for leading a community, but I need this community to enact by itself :) | |
| 08:41:29 | bauzas | simondodsley: fwiw, we have an PTG operator hour that's proposed today at 1500UTC https://ptg.opendev.org/ptg.html | |
| 08:41:51 | bauzas | simondodsley: if you feel you'd like to discuss your feature request, feel free to join | |
| 08:42:27 | noonedeadpunk | hey folks. I get weird error on HEAD of stable/2023.1 when trying to rebuild the instance: https://paste.openstack.org/show/biUIcOzMCx0YlsFob2KK/ | |
| 08:42:29 | bauzas | and like I said, if I can help someone volunteering, I'll surely offer onboarding | |
| 08:43:02 | bauzas | noonedeadpunk: eek | |
| 08:43:14 | noonedeadpunk | I decided to ask if it's smth known already so that I won't spend time on looking deeper (as for me it's not very obvious from the trace) | |
| 08:46:32 | noonedeadpunk | yeah, detach-boot-volume is really interesting thing, though I can recall it being not as trivial as it might look at the beginning | |
| 08:46:38 | bauzas | noonedeadpunk: this is a RPC call | |
| 08:46:56 | noonedeadpunk | ok, so I should check on.. conductor I guess | |
| 08:47:19 | bauzas | this sounds a compat issue between conductor and compute | |
| 08:47:36 | bauzas | basically your compute awaits some argument | |
| 08:47:42 | bauzas | that conductor isn't passing | |
| 08:47:52 | bauzas | I guess you upgraded ? | |
| 08:48:02 | bauzas | do you have a rolling upgrade in lieu ? | |
| 08:50:04 | bauzas | noonedeadpunk: that's the client RPC code that does the RPC negociation https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L1106-L1110 | |
| 08:50:30 | bauzas | this client code is run by the nova-conductor service | |
| 08:50:42 | bauzas | or by a compute service if this is compute-to-compute calls | |
| 08:50:45 | noonedeadpunk | Yeah, I am. And I see version 66 for every service in `services` table of cell db | |
| 08:51:35 | noonedeadpunk | oh, wait, couple of computes are still 60.... Hm | |
| 08:51:55 | noonedeadpunk | Not the affected one though, but still smth I need to fix | |
| 08:52:46 | bauzas | do you have some rpc pins ? | |
| 08:54:01 | noonedeadpunk | nah, I don't | |
| 08:54:13 | noonedeadpunk | Or well, should not have - never set them | |
| 08:54:15 | bauzas | https://github.com/openstack/nova/blob/stable/2023.1/nova/conductor/manager.py#L1395-L1414 is where conductor calls compute for rebuilding | |
| 08:55:05 | noonedeadpunk | Ok, computes that have eariler version are down right now... | |
| 08:55:09 | bauzas | the conductor manager is then invoking compute.rpcapi's rebuild_instance() which does RPC negociation | |
| 08:55:33 | bauzas | and then remove target_state if computes are older than rpc 6.2 | |
| 08:57:58 | noonedeadpunk | bauzas: is there any easy way to see reported prc version? becauce I guess in nova.services it's a different version rather then RPC. | |
| 08:58:00 | bauzas | noonedeadpunk: https://github.com/openstack/nova/commit/8c2e76598995f0d417653c60a63ea342baf4e880 is the commit that adds RPC 6.2 | |
| 08:58:24 | bauzas | noonedeadpunk: good question | |
| 08:58:58 | bauzas | I'd say internal RPC versions aren't meant to be exposed, but we provide aliases | |
| 08:59:20 | bauzas | https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L428 | |
| 08:59:45 | bauzas | zed computes are exposing 6.1, antelope ones are 6.2 | |
| 09:00:00 | noonedeadpunk | I guess I was more willing to check what computes/conductor reports as versions | |
| 09:00:08 | bauzas | if both your conductor *and* computes are 6.2, then try out to pin the rpc version | |
| 09:00:17 | bauzas | yeah, got it | |
| 09:00:28 | bauzas | the service version should tho give you that info | |
| 09:00:46 | noonedeadpunk | so if all is 66 - it should kinda work... | |
| 09:00:54 | bauzas | https://github.com/openstack/nova/commit/8c2e76598995f0d417653c60a63ea342baf4e880#diff-c0b6a5928be3ac40200a2078b084341bb9187a12b1f959ad862e0038c9029193 | |
| 09:01:09 | bauzas | 66 is the object version that matches RPC 6.2 | |
| 09:01:11 | noonedeadpunk | As iirc 66 is 2023.1 | |
| 09:01:14 | noonedeadpunk | yeah... | |
| 09:01:44 | bauzas | are you sure you restarted all your conductors after the upgrade ? :) | |
| 09:01:54 | bauzas | sorry for that silly question but I had to | |
| 09:02:02 | noonedeadpunk | They won't report 66 otherwise? | |
| 09:02:06 | bauzas | good call | |
| 09:02:17 | noonedeadpunk | I was actually trying to find a way to check what I could forgot to restart :) | |
| 09:02:31 | noonedeadpunk | But I'm checking now.... | |
| 09:02:51 | bauzas | I guess you can find the error in the conductor log, isn't it ? | |
| 09:03:02 | bauzas | if so, identify the conductor service in cause | |
| 09:10:04 | noonedeadpunk | Well, I've checked ps on all conductor hosts - all running from same venv and venv have same nova version installed... | |
| 09:10:21 | noonedeadpunk | And logs are clean as conductor doesn't obviously see any issues | |
| 09:10:40 | noonedeadpunk | let me check if code does contain these patches you've sent me | |
| 09:14:26 | noonedeadpunk | yeah, it's obviously there for all conductors... | |
| 09:21:10 | noonedeadpunk | Well, upgrade check throws `hw_machine_type unset` warning. Will try to sort it out - not sure if that's related though | |
| 09:23:51 | bauzas | no, this isn't | |
| 09:25:19 | bauzas | noonedeadpunk: since it's a venv, can you force to add some log in the conditional ? | |
| 09:25:52 | noonedeadpunk | sure | |
| 09:26:18 | bauzas | oh waity | |
| 09:26:27 | bauzas | this is a rebuild revert, right? | |
| 09:26:39 | bauzas | uh no | |
| 09:27:00 | bauzas | we're just calling the decorator in case rebuild_instance raises an exception | |
| 09:27:40 | bauzas | speaking out loud, | |
| 09:27:55 | bauzas | the RPC check on conductor considers that compute is older | |
| 09:28:38 | bauzas | but this is also not passing the target_state parameter or we would have an exception | |
| 09:29:31 | noonedeadpunk | no, it's not revert, it's bassically just `openstack server rebuild <server_uuid> --image <image_uuid>` | |
| 09:29:48 | bauzas | yeah yeah | |
| 09:30:13 | bauzas | it's popping up in the deco just because this is wrapping the call itself | |
| 09:30:21 | noonedeadpunk | But yes, looking at code you've shown I tend to agreee that issue is in rpc version, but it's very confusing, given that in DB it's reported as 66... And I pretty sure that before upgrade it was 61 or smth | |
| 09:30:40 | bauzas | but that means the RPC call doesn't contain the expected target_state arg | |
| 09:30:56 | bauzas | so something happens in the negociation | |
| 09:31:17 | bauzas | compute says "give me target_state" but conductor doesn't offer it | |
| 09:31:33 | noonedeadpunk | let me try to restart compute maybe, just in case.... | |
| 09:32:09 | bauzas | I'd say the other way, a conductor restart | |
| 09:32:22 | bauzas | compute is new enough to require target_state | |
| 09:32:52 | noonedeadpunk | yeah, makes sense... | |
| 09:33:30 | bauzas | and https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/manager.py#L621 exposes 6.2 I just checked | |
| 09:34:19 | bauzas | so yeah, I'd add a few debugging logs in the compute rpcapi method | |
| 09:34:31 | bauzas | (well, if this was my env, I'd pdb :D) | |
| 09:34:48 | noonedeadpunk | `Oct 24 09:34:35 control01-nova-api-container-13281850 nova-conductor[4131712]: 2023-10-24 09:34:35.835 4131712 INFO nova.compute.rpcapi [None req-28a1e0a6-3f71-4238-825d-b508d33ac758 - - - - - -] Automatically selected compute RPC version 6.1 from minimum service version 64` | |
| 09:34:51 | noonedeadpunk | o_O | |
| 09:34:59 | bauzas | but logging the object states before the RPC call seems a way forward | |
| 09:35:32 | bauzas | ahaha ! | |
| 09:35:36 | noonedeadpunk | Hm... Maybe that\s a compute which is down making things weird | |
| 09:35:41 | bauzas | no | |
| 09:36:10 | bauzas | I have to think about it, sec | |