| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-17 | |||
| 16:22:44 | bauzas | shall we run again this marvelous and succesful experience that is the operator-hour ? | |
| 16:23:23 | bauzas | I mean, I can just unbook some nova slot and officially pretend this is an operator hour | |
| 16:23:32 | dansmith | I think we should try yeah | |
| 16:23:57 | bauzas | and if nobody steps up, which would be sad and unfortunately expected, we could just consume the regular nova etherpad | |
| 16:24:20 | bauzas | okay, then I'll do the flip for tuesday 4pm | |
| 16:24:45 | bauzas | 4pm UTC allows us to get EU and US east-coast ops | |
| 16:25:15 | bauzas | and we could continue to discuss at 5pm if we really have audience and hot topics | |
| 16:25:53 | bauzas | #action bauzas to set up some operator hour, preferably Tuesday around 4pm UTC | |
| 16:26:09 | bauzas | #topic Review priorities | |
| 16:26:14 | bauzas | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2) | |
| 16:26:19 | bauzas | #info As a reminder, people eager to review changes can +1 to indicate their interest, +2 for asking cores to also review | |
| 16:26:43 | bauzas | yet again, taking the action to propose a Gerrit dash, once I'll have 5 mins of my time for doing this easy peasy | |
| 16:26:53 | bauzas | #topic Stable Branches | |
| 16:26:58 | bauzas | elodilles: ? | |
| 16:27:01 | elodilles | yepp | |
| 16:27:15 | elodilles | i'm not aware of any stable gate issues | |
| 16:27:31 | bauzas | the universe is smiling at me then | |
| 16:27:40 | elodilles | though nova-ceph-multistore is suspicious on stable/victoria | |
| 16:27:55 | elodilles | but need more check | |
| 16:28:02 | elodilles | otherwise gates should be OK | |
| 16:28:50 | elodilles | also, some bug fixes landed already on stable/2023.1 (antelope), so i'll propose a release patch, if that's OK for people | |
| 16:28:57 | bauzas | elodilles: thanks | |
| 16:29:10 | bauzas | elodilles: and yeah, sounds cool to me | |
| 16:29:16 | elodilles | ++ | |
| 16:29:18 | elodilles | and the usual: | |
| 16:29:21 | elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:29:35 | elodilles | add stable gate issues there ^^^ | |
| 16:29:38 | bauzas | fwiw, we'll discuss the State of Wallaby and Ussuri | |
| 16:29:41 | elodilles | if you encounter any | |
| 16:29:42 | bauzas | at the PTG* | |
| 16:29:51 | elodilles | bauzas: ACK | |
| 16:30:14 | elodilles | (wallaby, victoria, ussuri, you mean?) | |
| 16:30:38 | elodilles | anyway, we'll discuss at PTG :) | |
| 16:30:49 | bauzas | this is like the thanos blip | |
| 16:31:04 | bauzas | I pretend victoria never existed | |
| 16:31:34 | bauzas | elodilles: heard any other projects besides Cinder pursuing the idea to drop those releases ? | |
| 16:32:10 | elodilles | bauzas: well, some projects have eol'd their xena already | |
| 16:32:19 | elodilles | bauzas: let me check quickly | |
| 16:32:25 | bauzas | see, the ship has sailed then | |
| 16:32:35 | bauzas | should be a very quick discussion at the PTG then | |
| 16:32:55 | elodilles | kolla & magnum, otherwise i think projects are quiet about it yet | |
| 16:33:36 | elodilles | so no other projects yet | |
| 16:33:59 | bauzas | okay, we'll see at the PTG | |
| 16:34:03 | elodilles | +1 | |
| 16:34:09 | bauzas | thanks | |
| 16:34:19 | elodilles | ++ | |
| 16:34:20 | bauzas | #topic Open discussion | |
| 16:34:33 | bauzas | I have nothing in the wikipage | |
| 16:34:39 | bauzas | anything anyone ? | |
| 16:35:06 | bauzas | looks not | |
| 16:35:18 | bauzas | have a good day everyone and thanks all | |
| 16:35:22 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-17-16.01.log.html | |
| 16:35:22 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-17-16.01.txt | |
| 16:35:22 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-17-16.01.html | |
| 16:35:22 | opendevmeet | Meeting ended Tue Oct 17 16:35:22 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:35:22 | bauzas | #endmeeting | |
| 16:35:33 | elodilles | thanks bauzas o/ | |
| 16:38:05 | sean-k-mooney | bauzas: dansmith here is a short url to the greande failures https://tinyurl.com/2a63sy7e | |
| 17:08:52 | dansmith | sean-k-mooney: the fix patch passed grenade (or that phase of grenade) this time, so clearly not a hard fail | |
| 17:21:58 | sean-k-mooney | ack | |
| 17:22:25 | sean-k-mooney | it looks like it started aroudn october 5th | |
| 17:23:29 | auniyal | I think, this https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L245 i.e. connecting to test_vm via ssh to verify verify.txt has test string, because, somewhat create failed at `2023-10-17 13:59:13.930` | |
| 17:23:29 | auniyal | from this log https://3594ebcd65d47df3e70b-6ec9504d1ecc47a9ef6950d383ea355d.ssl.cf1.rackcdn.com/898435/1/check/nova-grenade-multinode/a3b1d83/controller/logs/grenade.sh_log.txt | |
| 18:02:43 | noonedeadpunk | Hey folks. I'm not really getting how to satisfy this check: https://opendev.org/openstack/nova/commit/27f384b7ac4f19ffaf884d77484814a220b2d51d | |
| 18:03:59 | noonedeadpunk | As eventually, you have a query for _compute_node_select where filter is `{"service_id": None}` | |
| 18:04:30 | noonedeadpunk | (as it's always being passed None as service_id here https://opendev.org/openstack/nova/src/branch/master/nova/db/main/api.py#L674) | |
| 18:05:29 | noonedeadpunk | And that eventually does not work at all for upgrades in our CI and quite reproducible https://paste.openstack.org/show/bWODASkcA0PkKHUHf3xG/ | |
| 18:07:15 | noonedeadpunk | And that's on current HEAD of 2023.2 | |
| 18:07:37 | noonedeadpunk | dansmith: I know you're on a tc meeting now, but maybe you have some thoughts on that | |
| 18:08:48 | dansmith | noonedeadpunk: have you done the upgrade before you run that check? | |
| 18:10:04 | dansmith | noonedeadpunk: perhaps you've done the upgrade but haven't started all the computes yet so they haven't fixed their records? | |
| 18:10:17 | dansmith | (as noted in the error message) | |
| 18:11:16 | noonedeadpunk | Yes, online data migrations was run on N-1 as well according to the logs | |
| 18:11:50 | noonedeadpunk | and that' the first thing I've checked - that all is running https://paste.openstack.org/show/b8ZgQRT6N2qmGRRkdEZz/ | |
| 18:12:16 | dansmith | the computes do their own migrations of this, so running online_data_migrations won't change it | |
| 18:12:43 | dansmith | hang on | |
| 18:13:08 | noonedeadpunk | well, talking about that - computes were not restarted yet after the upgrade of API | |
| 18:13:37 | noonedeadpunk | (I guess) | |
| 18:14:13 | dansmith | well, that'd be why then | |
| 18:15:49 | noonedeadpunk | but um... I guess then I'm confused at what point nova-status upgrade check should run? | |
| 18:16:17 | noonedeadpunk | As that is to ensure that things are ready for upgrade? and like - cancel upgrade if they're not? | |
| 18:16:41 | noonedeadpunk | So, in case of upgrade, you should now upgrade computes first and then only api/conductor/scheduler? | |
| 18:18:09 | noonedeadpunk | https://docs.openstack.org/nova/latest/cli/nova-status.html#upgrade `Performs a release-specific readiness check before restarting services with new code` | |
| 18:18:31 | noonedeadpunk | So exactly like I did - run check, it failed, services were not restarted with the new code... | |
| 18:18:41 | dansmith | noonedeadpunk: okay maybe that should have been a warning I guess, idk | |
| 18:18:51 | sean-k-mooney | noonedeadpunk: nova-status is ment to be run before you do any upgrades | |
| 18:19:19 | noonedeadpunk | ahaa..... that what can be wrong | |
| 18:19:34 | noonedeadpunk | as I'm running it with 2023.1 code before restart | |
| 18:19:39 | noonedeadpunk | sorry | |
| 18:19:43 | noonedeadpunk | 2023.2 code | |
| 18:20:12 | dansmith | sean-k-mooney: running the new nova-status before you upgrade right? | |
| 18:20:17 | noonedeadpunk | so 1. upgrade code. 2. run check 3. restart services if it passes rollback if it's not | |
| 18:20:29 | dansmith | so we shouldn't have made that an error because without having run computes, you can't have had the uuids added | |
| 18:20:31 | noonedeadpunk | (it's current flow) | |
| 18:20:41 | sean-k-mooney | so to upgrade to 2023.2 form 2023.1 you would run the 2023.2 version of nova-status before doign the upgrde | |
| 18:21:13 | noonedeadpunk | yeah, I think that what I do actually | |
| 18:21:36 | noonedeadpunk | Like code is upgraded but services still running old code | |
| 18:22:30 | sean-k-mooney | dansmith: am i right about runnign the 2023.2 nova status when upgradeign to 2023.2 or should it be the 2023.1 version | |
| 18:22:35 | sean-k-mooney | i feel like im wrong about htat | |
| 18:23:12 | sean-k-mooney | we are ment to deprecate thign at least one release early so we normally put the status check in with the deprecation as a warning | |