| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-24 | |||
| 09:34:31 | bauzas | (well, if this was my env, I'd pdb :D) | |
| 09:34:48 | noonedeadpunk | `Oct 24 09:34:35 control01-nova-api-container-13281850 nova-conductor[4131712]: 2023-10-24 09:34:35.835 4131712 INFO nova.compute.rpcapi [None req-28a1e0a6-3f71-4238-825d-b508d33ac758 - - - - - -] Automatically selected compute RPC version 6.1 from minimum service version 64` | |
| 09:34:51 | noonedeadpunk | o_O | |
| 09:34:59 | bauzas | but logging the object states before the RPC call seems a way forward | |
| 09:35:32 | bauzas | ahaha ! | |
| 09:35:36 | noonedeadpunk | Hm... Maybe that\s a compute which is down making things weird | |
| 09:35:41 | bauzas | no | |
| 09:36:10 | bauzas | I have to think about it, sec | |
| 09:36:25 | noonedeadpunk | Well. Compute that is down still having version 64 | |
| 09:36:42 | noonedeadpunk | So it can bring negotiation to that version? | |
| 09:37:29 | noonedeadpunk | I pretty much can bet it's the case... | |
| 09:37:46 | bauzas | oh, wait a sec | |
| 09:38:15 | noonedeadpunk | the compute was down during the upgrade | |
| 09:38:18 | bauzas | I need to remember my rpc compat skilles | |
| 09:38:23 | bauzas | skills* | |
| 09:39:07 | bauzas | yeah, so the thing is,https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L504-L505 gives you 64 | |
| 09:39:34 | bauzas | so we're capping to the minimum version | |
| 09:39:34 | bauzas | but then, the manager shouldn't yell | |
| 09:39:52 | bauzas | I'm missing something obvious but I'm rusty | |
| 09:40:23 | noonedeadpunk | I kinda wonder if `get_minimum_version` should account only for ones that are up... | |
| 09:41:16 | noonedeadpunk | But yeah, also if that minimum version is fine, then computes shouldn't fail because version is too low either I guess... | |
| 09:41:19 | bauzas | no no | |
| 09:41:23 | bauzas | this is a bug | |
| 09:41:27 | bauzas | I just checked | |
| 09:41:39 | bauzas | we broke rolling-upgrade compat, that's it | |
| 09:41:57 | noonedeadpunk | Should I submit one then? | |
| 09:42:03 | bauzas | the manager should default target_state=None | |
| 09:42:26 | bauzas | because we automatically pin our RPC calls to the min version | |
| 09:42:47 | bauzas | I'll doublecheck that with dansmith but I bet this is the problem | |
| 09:43:25 | noonedeadpunk | fwiw, dropping that old compute from services made negitoation pass | |
| 09:43:33 | noonedeadpunk | `Automatically selected compute RPC version 6.2 from minimum service version 66` | |
| 09:43:37 | bauzas | and https://github.com/openstack/nova/commit/30aab9c234035b49c7e2cdc940f624a63eeffc1b#diff-47eb12598e353b9e0689707d7b477353200d0aa3ed13045ffd3d017ee7d9e753R3709 has the same problem | |
| 09:43:37 | noonedeadpunk | and rebuild worked out | |
| 09:45:29 | bauzas | yeah, 64 is zed | |
| 09:45:51 | noonedeadpunk | Ok, I can submit a bug report if you think it's a bug, and I have a way around the issue as well :) | |
| 09:45:58 | bauzas | but basically, rebuild is broken on a rolling-upgrade from Zed to Antelope | |
| 09:46:10 | bauzas | noonedeadpunk: sure, file it please | |
| 09:46:48 | bauzas | and I think rebuild is also broken from Yoga to Zed btw. :) | |
| 09:47:26 | noonedeadpunk | Yes, another region we were upgrading from Yoga was also broken... | |
| 09:47:40 | bauzas | on rebuild ? | |
| 09:47:40 | noonedeadpunk | But it's unofficially supported upgrade, so I took better example :p | |
| 09:47:48 | noonedeadpunk | Yeah, same error | |
| 09:47:57 | bauzas | but not the same argument I guess | |
| 09:48:01 | noonedeadpunk | The only that has rebuild working is the one that has no computes down | |
| 09:48:23 | bauzas | from yoga to zed, it should yell at reimage_boot_volume missing | |
| 09:48:30 | noonedeadpunk | I actually wasn't digging there yet to be frank, as reported by users issues were the same from their side | |
| 09:49:09 | bauzas | I have to doublecheck but I think we're missing to test rebuild with grenade | |
| 09:50:32 | bauzas | anyway, I need to disappear, I have a few toxins to sweat before we start the PTG | |
| 09:53:48 | noonedeadpunk | sure, thanks a lot for the help, report is on it's way | |
| 09:58:06 | noonedeadpunk | eventually, since I found out about `hw_machine_type` what you would suggest adding? `q35` for all images basically? | |
| 10:15:42 | noonedeadpunk | bug: https://bugs.launchpad.net/nova/+bug/2040264 | |
| 11:39:00 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/2023.1: Translate VF network capabilities to port binding https://review.opendev.org/c/openstack/nova/+/898945 | |
| 12:09:20 | sean-k-mooney | noonedeadpunk: i have not read back fully but one of the main issues with detachign a boot drive is it leave the instance in a fundementally broken state | |
| 12:10:22 | sean-k-mooney | if we were every to supprot that i htink we woudl need to have a new state for the vm to model that. and or allow away to say use a seperate disk as the boot device. | |
| 12:10:48 | noonedeadpunk | Well, shelved_offloaded should work kinda? | |
| 12:10:55 | sean-k-mooney | it also kind of messes with the resouce allcoatins if you dont follwo our guidance and use flavor with 0 disk_gb for bfv guests | |
| 12:11:11 | noonedeadpunk | But I guess challange there was also to record what the drive is detached to re-attach it as root lately | |
| 12:11:31 | sean-k-mooney | noonedeadpunk: shelve_offloaded does not work because we cant unshelve if you have detached the volume and attached it to somethign else | |
| 12:11:54 | noonedeadpunk | but it would be a valid failure that VM can't be offloaded | |
| 12:12:22 | sean-k-mooney | ya so it really comes down to the usecase | |
| 12:12:23 | noonedeadpunk | sean-k-mooney: sorry, I didn't get part about "use flavor with 0 disk_gb for bfv guests" - can you elaborate a bit on this part? | |
| 12:12:44 | noonedeadpunk | As I thought that for bfv you're kinda good at having 0 disk_gb? | |
| 12:12:56 | sean-k-mooney | yes that is the recomendation | |
| 12:13:09 | sean-k-mooney | but some peopel dont do that and use a normal flavor | |
| 12:13:13 | bauzas | sean-k-mooney: the fact is, we need a new design discussin | |
| 12:13:22 | bauzas | but if someone wants to work on it, sure I can hel | |
| 12:13:24 | noonedeadpunk | ah, ok, as I read it as "you don't follow recommendation AND use 0 disk_gb" | |
| 12:13:25 | bauzas | help | |
| 12:13:29 | bauzas | that was my point | |
| 12:13:39 | bauzas | noonedeadpunk: ++ I'll also discuss it on the PTG | |
| 12:14:14 | sean-k-mooney | bauzas: well right now i think its a hard cell to add to nova unless we put some restricitons in place | |
| 12:14:23 | sean-k-mooney | ... sell | |
| 12:14:42 | bauzas | sure, see my above comments :) | |
| 12:14:49 | bauzas | anyway | |
| 12:15:03 | bauzas | noonedeadpunk: I'll provide a bugfix hopefully next week | |
| 12:15:13 | sean-k-mooney | noonedeadpunk: i would have less issue with saying "you can only detach the root voluem flavor with 0 disk_gb" | |
| 12:15:15 | bauzas | or tomorrow morning if I have time | |
| 12:15:27 | sean-k-mooney | and then putting the vm in to an "unbootable state" | |
| 12:15:40 | sean-k-mooney | or only supporting this for shleve_offloaded instances | |
| 12:16:22 | sean-k-mooney | noonedeadpunk: the real concern was not with the detach but with reattaching potentially a diffent root volume with image properties that were invlaid for the current host | |
| 12:16:53 | sean-k-mooney | but if its shelve_offloaded after the reattachment then that not a problem as when we unshleve it will go throughthe schduler | |
| 12:17:28 | noonedeadpunk | Yeah, that's why I thought that shelved should be quite appropriate state where you potentially can do that | |
| 12:17:46 | noonedeadpunk | And I would say that having 0 disk_gb for flavor would be also fair requirements | |
| 12:18:03 | noonedeadpunk | for the feature to work | |
| 12:18:27 | sean-k-mooney | the 0 disk_gb thing is partly a personal prefence. we do not allow non bfv guest to use 0 disk flavors | |
| 12:18:39 | sean-k-mooney | but we dont require them too either and i really would prefer to require that | |
| 12:19:37 | sean-k-mooney | we dont actully need to use the flavor size to knwo if its bfv in the api | |
| 12:19:47 | sean-k-mooney | so we could make it work without that restirction | |
| 12:20:06 | sean-k-mooney | noonedeadpunk: what i really want to aovid is needing to hit the schduler on every volume attach | |
| 12:20:16 | noonedeadpunk | well. requiring that might complicate some things actually... | |
| 12:20:34 | sean-k-mooney | just in case its happens to be a root volume | |
| 12:20:47 | noonedeadpunk | I mean prohibit to do bfv with non-0 flavor | |
| 12:21:01 | sean-k-mooney | we didnt do that in the past due to upgrade concerns | |
| 12:21:19 | sean-k-mooney | what we did instead id if its BFV we nolonger include the disk_gb in the allcoation request | |
| 12:21:24 | noonedeadpunk | nah, I think that it indeed should be some offloaded state imo... Or indeed introduce new one, so that resources won't be wasted on offload per sat | |
| 12:21:30 | sean-k-mooney | as of some releivly recent openstack version | |
| 12:21:59 | noonedeadpunk | but which will still go throguh scheduler on boot | |
| 12:22:18 | noonedeadpunk | but yeah, eventually it's still offloaded... | |
| 12:22:55 | sean-k-mooney | noonedeadpunk: we will talk about this later in the ptg but im also wondering about the orginal usecase that is of most interest to people | |
| 12:23:17 | sean-k-mooney | i.e. what fraction of those that want this are using a version of nova that does not supprot rescue or rebuild for bfv guests | |