| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-17 | |||
| 16:31:34 | bauzas | elodilles: heard any other projects besides Cinder pursuing the idea to drop those releases ? | |
| 16:32:10 | elodilles | bauzas: well, some projects have eol'd their xena already | |
| 16:32:19 | elodilles | bauzas: let me check quickly | |
| 16:32:25 | bauzas | see, the ship has sailed then | |
| 16:32:35 | bauzas | should be a very quick discussion at the PTG then | |
| 16:32:55 | elodilles | kolla & magnum, otherwise i think projects are quiet about it yet | |
| 16:33:36 | elodilles | so no other projects yet | |
| 16:33:59 | bauzas | okay, we'll see at the PTG | |
| 16:34:03 | elodilles | +1 | |
| 16:34:09 | bauzas | thanks | |
| 16:34:19 | elodilles | ++ | |
| 16:34:20 | bauzas | #topic Open discussion | |
| 16:34:33 | bauzas | I have nothing in the wikipage | |
| 16:34:39 | bauzas | anything anyone ? | |
| 16:35:06 | bauzas | looks not | |
| 16:35:18 | bauzas | have a good day everyone and thanks all | |
| 16:35:22 | bauzas | #endmeeting | |
| 16:35:22 | opendevmeet | Meeting ended Tue Oct 17 16:35:22 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:35:22 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-17-16.01.html | |
| 16:35:22 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-17-16.01.txt | |
| 16:35:22 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-17-16.01.log.html | |
| 16:35:33 | elodilles | thanks bauzas o/ | |
| 16:38:05 | sean-k-mooney | bauzas: dansmith here is a short url to the greande failures https://tinyurl.com/2a63sy7e | |
| 17:08:52 | dansmith | sean-k-mooney: the fix patch passed grenade (or that phase of grenade) this time, so clearly not a hard fail | |
| 17:21:58 | sean-k-mooney | ack | |
| 17:22:25 | sean-k-mooney | it looks like it started aroudn october 5th | |
| 17:23:29 | auniyal | from this log https://3594ebcd65d47df3e70b-6ec9504d1ecc47a9ef6950d383ea355d.ssl.cf1.rackcdn.com/898435/1/check/nova-grenade-multinode/a3b1d83/controller/logs/grenade.sh_log.txt | |
| 17:23:29 | auniyal | I think, this https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L245 i.e. connecting to test_vm via ssh to verify verify.txt has test string, because, somewhat create failed at `2023-10-17 13:59:13.930` | |
| 18:02:43 | noonedeadpunk | Hey folks. I'm not really getting how to satisfy this check: https://opendev.org/openstack/nova/commit/27f384b7ac4f19ffaf884d77484814a220b2d51d | |
| 18:03:59 | noonedeadpunk | As eventually, you have a query for _compute_node_select where filter is `{"service_id": None}` | |
| 18:04:30 | noonedeadpunk | (as it's always being passed None as service_id here https://opendev.org/openstack/nova/src/branch/master/nova/db/main/api.py#L674) | |
| 18:05:29 | noonedeadpunk | And that eventually does not work at all for upgrades in our CI and quite reproducible https://paste.openstack.org/show/bWODASkcA0PkKHUHf3xG/ | |
| 18:07:15 | noonedeadpunk | And that's on current HEAD of 2023.2 | |
| 18:07:37 | noonedeadpunk | dansmith: I know you're on a tc meeting now, but maybe you have some thoughts on that | |
| 18:08:48 | dansmith | noonedeadpunk: have you done the upgrade before you run that check? | |
| 18:10:04 | dansmith | noonedeadpunk: perhaps you've done the upgrade but haven't started all the computes yet so they haven't fixed their records? | |
| 18:10:17 | dansmith | (as noted in the error message) | |
| 18:11:16 | noonedeadpunk | Yes, online data migrations was run on N-1 as well according to the logs | |
| 18:11:50 | noonedeadpunk | and that' the first thing I've checked - that all is running https://paste.openstack.org/show/b8ZgQRT6N2qmGRRkdEZz/ | |
| 18:12:16 | dansmith | the computes do their own migrations of this, so running online_data_migrations won't change it | |
| 18:12:43 | dansmith | hang on | |
| 18:13:08 | noonedeadpunk | well, talking about that - computes were not restarted yet after the upgrade of API | |
| 18:13:37 | noonedeadpunk | (I guess) | |
| 18:14:13 | dansmith | well, that'd be why then | |
| 18:15:49 | noonedeadpunk | but um... I guess then I'm confused at what point nova-status upgrade check should run? | |
| 18:16:17 | noonedeadpunk | As that is to ensure that things are ready for upgrade? and like - cancel upgrade if they're not? | |
| 18:16:41 | noonedeadpunk | So, in case of upgrade, you should now upgrade computes first and then only api/conductor/scheduler? | |
| 18:18:09 | noonedeadpunk | https://docs.openstack.org/nova/latest/cli/nova-status.html#upgrade `Performs a release-specific readiness check before restarting services with new code` | |
| 18:18:31 | noonedeadpunk | So exactly like I did - run check, it failed, services were not restarted with the new code... | |
| 18:18:41 | dansmith | noonedeadpunk: okay maybe that should have been a warning I guess, idk | |
| 18:18:51 | sean-k-mooney | noonedeadpunk: nova-status is ment to be run before you do any upgrades | |
| 18:19:19 | noonedeadpunk | ahaa..... that what can be wrong | |
| 18:19:34 | noonedeadpunk | as I'm running it with 2023.1 code before restart | |
| 18:19:39 | noonedeadpunk | sorry | |
| 18:19:43 | noonedeadpunk | 2023.2 code | |
| 18:20:12 | dansmith | sean-k-mooney: running the new nova-status before you upgrade right? | |
| 18:20:17 | noonedeadpunk | so 1. upgrade code. 2. run check 3. restart services if it passes rollback if it's not | |
| 18:20:29 | dansmith | so we shouldn't have made that an error because without having run computes, you can't have had the uuids added | |
| 18:20:31 | noonedeadpunk | (it's current flow) | |
| 18:20:41 | sean-k-mooney | so to upgrade to 2023.2 form 2023.1 you would run the 2023.2 version of nova-status before doign the upgrde | |
| 18:21:13 | noonedeadpunk | yeah, I think that what I do actually | |
| 18:21:36 | noonedeadpunk | Like code is upgraded but services still running old code | |
| 18:22:30 | sean-k-mooney | dansmith: am i right about runnign the 2023.2 nova status when upgradeign to 2023.2 or should it be the 2023.1 version | |
| 18:22:35 | sean-k-mooney | i feel like im wrong about htat | |
| 18:23:12 | sean-k-mooney | we are ment to deprecate thign at least one release early so we normally put the status check in with the deprecation as a warning | |
| 18:23:17 | dansmith | honestly, I think it's supposed to be "am I done with my upgrade yet" | |
| 18:23:21 | noonedeadpunk | but how with old code you can be aware if it's safe to upgrade to the new one? | |
| 18:23:28 | dansmith | which before you upgrade will be "no" | |
| 18:23:45 | dansmith | but maybe error should only be raised if you have to do something before upgrading, idk | |
| 18:24:03 | dansmith | I wrote that at your request (IIRC) and don't recall any review comments about it :) | |
| 18:24:16 | sean-k-mooney | :) | |
| 18:25:31 | sean-k-mooney | so the currnt check woudl only pass after all compute are fully upgraded but it also should only be an error if th min compute service version is above the version where we stated doing that | |
| 18:26:19 | sean-k-mooney | noonedeadpunk: is this curretly blocking you form upgrading by the way | |
| 18:26:31 | sean-k-mooney | or were you asking how to make the check happy | |
| 18:26:47 | sean-k-mooney | if the later then you need to start the computes with the bobcat code | |
| 18:26:54 | sean-k-mooney | and it should pass after they have all started | |
| 18:26:55 | noonedeadpunk | Well. It makes our upgrade code fail in osa | |
| 18:27:17 | noonedeadpunk | And I'm looking how to make our upgrade jobs happy as well as users that would perform upgrade | |
| 18:27:43 | sean-k-mooney | i would need to check our docs for which version of nova status we expect to be run and where | |
| 18:28:15 | noonedeadpunk | It says nothing after Xena.... | |
| 18:28:28 | noonedeadpunk | if you're about https://docs.openstack.org/nova/latest/cli/nova-status.html#upgrade | |
| 18:28:44 | sean-k-mooney | nova-status on -1 tells you about the thigns that are deprecated that we know might brake you in the future | |
| 18:28:57 | sean-k-mooney | so before upgrade you shoudl fix those with the n-1 version | |
| 18:29:09 | sean-k-mooney | then do the upgrade. | |
| 18:29:57 | noonedeadpunk | mhm. Ok, I see, so it should run not on the upgraded code | |
| 18:29:58 | sean-k-mooney | ok so reading the doc text | |
| 18:30:12 | sean-k-mooney | its say "Performs a release-specific readiness check before restarting services with new code." | |
| 18:30:39 | sean-k-mooney | that would implcy that the new check should be a warning in 2023.2 and an error in 2024.1 i think | |
| 18:30:44 | noonedeadpunk | Well, I read it in a way - upgrade code, run test, restart services | |
| 18:30:52 | dansmith | sean-k-mooney: yep | |
| 18:31:06 | sean-k-mooney | although with slurp it shold be a warnignin 2024.1 as well? | |
| 18:31:15 | noonedeadpunk | I'm not sure it should be a warning even.... | |
| 18:31:28 | noonedeadpunk | and yeah, error only in 2024.2 | |
| 18:31:55 | noonedeadpunk | As that is really expected, that you won't have that at this stage, so what to warn about... | |
| 18:32:21 | sean-k-mooney | well it may mean you have old non upgraded compute service records | |
| 18:32:37 | sean-k-mooney | that is pretty common where peope scale in or remove say ironic | |
| 18:32:44 | noonedeadpunk | but you 100% have them until you restart computes, that will happen only afterwards? | |
| 18:32:44 | sean-k-mooney | and forget to remove the compute service records | |
| 18:32:53 | dansmith | noonedeadpunk: it needs to go from some state to "all good" after you're done with the upgade | |
| 18:32:59 | sean-k-mooney | that hsoudl be caust by the "Check: Older than N-1 computes " check | |