| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-08-30 | |||
| 19:56:45 | gibi | I cannot argue that this can be done differently | |
| 19:57:08 | gibi | and dansmith you are right that the system metadata dependency makes it at least a grey interface | |
| 19:57:42 | gibi | it is sad that we figured out this issue late in the cycle | |
| 19:58:12 | dansmith | gibi: I don't understand what you mean by #1, but yeah I guess you're right on the trait.. that's preeety thin though :) | |
| 19:59:02 | gibi | #1 is probably just a missunderstanding from | |
| 19:59:03 | gibi | 20:58 < dansmith> rebuild is basically growing a new feature, and we need to pass it a flag, | |
| 20:00:03 | sean-k-mooney[m] | the trait wont be reported on a non upgraded compute yes | |
| 20:00:14 | sean-k-mooney[m] | which might help for the upgrade chase specifically | |
| 20:00:26 | dansmith | gibi: oh yeah I meant reboot there sorry | |
| 20:00:27 | gibi | on the flag itself. The RPC flag vs the persisted field has some semantic difference. If we update the user_data in the DB in the API layer then pass a flag to regenerate the config drive via the RPC then a lost RPC means that the DB data and the config data got out of sync | |
| 20:01:24 | dansmith | gibi: but we can and should revert if we're reporting failure to the user | |
| 20:01:30 | sean-k-mooney[m] | so im not sure if we should revert | |
| 20:01:32 | dansmith | because if this happens because of the lack of a trait or version, | |
| 20:01:34 | gibi | the reboot RPC is a cast | |
| 20:01:45 | gibi | so if the RPC lost the API wont notice | |
| 20:01:46 | sean-k-mooney[m] | the reason for that is todya id we update instnace metadata | |
| 20:01:48 | dansmith | then it will pop into being in six months and be very confusing | |
| 20:01:52 | sean-k-mooney[m] | we dont update the config drive | |
| 20:02:04 | sean-k-mooney[m] | unless you do a cross cell migration | |
| 20:02:23 | sean-k-mooney[m] | so you can have a delta between the metadta api and the config drive today | |
| 20:02:31 | melwitt | ugh, my irc client had "froze" not receiving new messages for only this network and I didn't realize it until now. had to close and reopen the client to receive and send messages | |
| 20:03:02 | dansmith | gibi: ack, not for a version conflict, but for an actual lost RPC we'd get out of sync.. I'm not sure if that's better or worse than queuing an update for six months later on a different version of the software, but fair point | |
| 20:03:35 | gibi | yeah, both case seems problematic | |
| 20:03:39 | dansmith | indeed | |
| 20:03:55 | melwitt | I just skimmed through yalls review comments from today a little while ago and don't have a handle yet on what's going on. I will read further and add a comment once I understand it | |
| 20:04:23 | dansmith | I guess the trait eliminates the acute concern of this being actually broken, but I'm still concerned about setting the precedent for shadow RPC interfaces in metadata, even if protected by a flag like that | |
| 20:04:24 | gibi | could we do both the DB update and the config driver regeneration from the nova-compute service? | |
| 20:04:33 | sean-k-mooney[m] | do we consider the user_data to be higher imporantce to be updated then other info in the config drive | |
| 20:04:36 | dansmith | it's what we have versioning for and how we do math about "can we do this now or not" | |
| 20:04:38 | sean-k-mooney[m] | that we dont regenerate today | |
| 20:05:06 | dansmith | gibi: well my first thought was not a flag, but pass the user data to the reboot call, and let it update it | |
| 20:05:14 | dansmith | gibi: that would be much better all around | |
| 20:05:21 | dansmith | I need to look at the potential size limit though | |
| 20:05:29 | sean-k-mooney[m] | like if you attch a volume/interface or update server metadata that wont get updated in the config drive today | |
| 20:05:31 | gibi | ahh yeah, it is a blob | |
| 20:05:34 | dansmith | if you can pass a MiB that would be bad... | |
| 20:06:11 | sean-k-mooney[m] | its 64k i think | |
| 20:06:36 | dansmith | sean-k-mooney[m]: is it? | |
| 20:06:52 | sean-k-mooney[m] | its large yes | |
| 20:06:57 | dansmith | we might want to make that bigger though at some point, so expecting to put that into an rpc message might be a bad idea long-term | |
| 20:07:44 | melwitt | it was a bit of a coincidence :) I saw sean's comment on the userdata review come in email and they said "I talked to dan" but I didn't see any talking to dan in the channel. that's when I realized my client was messed up | |
| 20:07:56 | dansmith | aha | |
| 20:08:26 | gibi | I need to drop for the night. I'm fine pulling user_data out of the release while we design it better. I just whish we can somehow avoid in the future push contributors into a desgin dead end and then pulling the rug out at FF. | |
| 20:08:51 | dansmith | I know the feeling because I was arguing that we not do that for bfv rebuild either | |
| 20:09:01 | gibi | yeah | |
| 20:09:04 | dansmith | and I noted in my comment that (a) I know the implication and (b) I'm willing to scramble on the work | |
| 20:09:09 | melwitt | so I disconnected and reconnected the network and I saw yalls comments rolling in. but when I sent messages there was no acknowledgement, so I checked the irc logs and my messages weren't there. so I had to escalate to a full quit/start of my client. and now it's working 🙄 | |
| 20:09:29 | gibi | I can look at the patch / comments tomorrow morning. But now I drop. See you tomorrow | |
| 20:09:39 | dansmith | alright | |
| 20:09:45 | gibi | o/ | |
| 20:09:56 | dansmith | sean-k-mooney[m]: how about this: | |
| 20:10:28 | dansmith | sean-k-mooney[m]: how about we let this land as it is because theoretically the trait should catch it, and we convert to an RPC interface after BFV set and before the release | |
| 20:10:36 | dansmith | that won't be a behavioral change since it *should* be catching it now | |
| 20:10:58 | dansmith | if someone is deploying on master within a two week window they could have some sysmeta cruft, but highly unlikely | |
| 20:11:31 | sean-k-mooney[m] | ack we can likely get that working by the end of the week | |
| 20:11:43 | sean-k-mooney[m] | as part of the follow up patch once bfv is landed | |
| 20:11:46 | dansmith | it's mostly what I just wrote, but 6.2 | |
| 20:11:56 | sean-k-mooney[m] | yep | |
| 20:12:09 | dansmith | but I also think that this is missing a lot of testing it should have | |
| 20:12:23 | sean-k-mooney[m] | looking at it again you are right | |
| 20:12:26 | dansmith | has anyone other than the author tried this on a real devstack? with configdrive? | |
| 20:12:50 | sean-k-mooney[m] | no i was going to see if i could do that tonight but i might just do that torrow at this point | |
| 20:12:57 | dansmith | also, in my defense, gibi *did* ask me to review this :) | |
| 20:13:30 | dansmith | and I *did* try to punt to melwitt | |
| 20:13:46 | dansmith | and melwitt *did* sabotage her irc client so she "didn't see that" | |
| 20:13:52 | sean-k-mooney[m] | i was going to see if i could create a tempest test for this althogh im not sure how to force the vm too boot on the un upgraded node for grenade | |
| 20:14:12 | dansmith | yeah you can't really, so you have to boot two and hit both I think | |
| 20:14:29 | dansmith | but at least it would non-deterministically fail | |
| 20:14:32 | sean-k-mooney[m] | oh with the anti affintiy filter | |
| 20:15:09 | dansmith | melwitt: are you caught up yet, enough to grok that ^ plan? | |
| 20:16:12 | melwitt | dansmith: yeah I think so | |
| 20:16:30 | dansmith | and what say ye? | |
| 20:17:21 | melwitt | the plan sounds like a good compromise | |
| 20:18:44 | sean-k-mooney[m] | we can likely sync with the autour tomrrow but i can set this up and test it tomorrow in anycase and perhaps look at more testing | |
| 20:19:08 | sean-k-mooney[m] | so see if we can harden this and unblock bfv | |
| 20:20:05 | dansmith | I just un-1'd it with a writeup | |
| 20:20:24 | dansmith | sean-k-mooney[m]: are you saying that because you want to test it before it merges, given it has no real testing now? | |
| 20:20:33 | dansmith | or because you expect some change to this again? | |
| 20:21:10 | sean-k-mooney[m] | they were going to work on a followup patch to adress some nits | |
| 20:21:24 | sean-k-mooney[m] | and i want to test it and figure out what else they should add tests for in that | |
| 20:21:46 | dansmith | okay but we told them to do those as a follow-up right? | |
| 20:21:56 | sean-k-mooney[m] | yep | |
| 20:22:35 | dansmith | okay, well, tomorrow our runway is even shorter | |
| 20:22:40 | dansmith | so hopefully you can do that in the morning :) | |
| 20:23:06 | melwitt | sean-k-mooney[m]: something I was confused on was whether the admin password (if there is one) would be preserved across a configdrive recreate. it seemed like yes? based on userdata update during rebuild impl, but I couldn't find definitively how that works | |
| 20:23:20 | dansmith | I guess I thought you all were more confident in this with all the +2s it had | |
| 20:24:16 | sean-k-mooney[m] | i was priortising it more then i would otherwise becuase of the bfv series on top | |
| 20:25:37 | dansmith | yeah the ordering was unfortunate | |
| 20:26:05 | sean-k-mooney[m] | i should have reviewed it more closly sorry. thanks for reviewing though even if the timing is not idea its better to get this right | |
| 20:27:24 | dansmith | yep, so maybe if you test in the morning and +W it we can get the others in the queue and I'll start on the proper RPC stuff based on my pastebin above when I'm around | |
| 21:23:35 | opendevreview | Rajat Dhasmana proposed openstack/python-novaclient master: Add support to rebuild boot volume https://review.opendev.org/c/openstack/python-novaclient/+/827163 | |
| 21:31:59 | opendevreview | Rajat Dhasmana proposed openstack/python-novaclient master: Add support to rebuild boot volume 2.94 https://review.opendev.org/c/openstack/python-novaclient/+/827163 | |
| 21:33:02 | whoami-rajat | ^ rebased on top of 2.93 to avoid conflicts later | |
| 21:34:47 | whoami-rajat | sean-k-mooney[m], jfyi, you've a -2 on client patch of 2.93 https://review.opendev.org/c/openstack/python-novaclient/+/816158 | |
| 21:41:37 | sean-k-mooney[m] | oh i didnt clear that after you spun | |
| 21:41:50 | sean-k-mooney[m] | i proably wont review that tonight but ill drop the -2 sorry about that | |
| 21:43:01 | sean-k-mooney[m] | oh thats the user data one | |
| 21:44:25 | whoami-rajat | yep, you already dropped the -2 from mine some time back but without the user data one, my change can't get in :) | |
| 21:48:05 | sean-k-mooney[m] | i ment to do that when the spec was approved | |
| 21:50:12 | sean-k-mooney[m] | there is no reason your bfv sereise would not work with the lvm driver right | |