| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-02 | |||
| 17:40:07 | dansmith | bauzas: is this because you can't merge the log revert? | |
| 17:40:23 | bauzas | that + the prelude + the service object bump | |
| 17:40:28 | bauzas | 3 patches on fly | |
| 17:40:35 | dansmith | oh right, those, gotcha | |
| 17:40:55 | bauzas | we should end up releasing RC1 on the weekend | |
| 17:40:55 | dansmith | we can certainly merge the prelude right? | |
| 17:41:04 | bauzas | dansmith: the prelude is still in flight | |
| 17:41:22 | bauzas | oh my bad, not | |
| 17:41:27 | dansmith | bauzas: in flight meaning not reviewed by enough people right? not because you can't land due to tests | |
| 17:41:37 | bauzas | dansmith: send it to the gate, folks that care should have voted earlier | |
| 17:41:52 | dansmith | done | |
| 17:42:00 | bauzas | yeah, my bad I was paying attention to the two others | |
| 17:42:17 | bauzas | the revert logging may be merged later and backported | |
| 17:42:24 | dansmith | yeah the revert is not a huge deal | |
| 17:42:26 | bauzas | this is just a silly logging | |
| 17:42:30 | dansmith | would be nice, but not critical | |
| 17:42:40 | dansmith | I'll watch the service version one | |
| 17:42:43 | bauzas | but the prelude has to land, ditto the service check | |
| 17:42:46 | dansmith | yeah | |
| 17:43:00 | bauzas | thanks for caring this | |
| 17:43:08 | bauzas | I'll look at those later today | |
| 17:43:29 | dansmith | maybe if I can grind these two you can post the release tomorrow morning | |
| 17:44:01 | bauzas | maybe | |
| 17:44:08 | bauzas | gerrit will be restarted later today | |
| 17:44:24 | bauzas | that was an ask from the infra team | |
| 17:44:35 | bauzas | zuul will only be recycled this weekend | |
| 17:44:35 | dansmith | oye | |
| 17:46:03 | clarkb | note zuul is zero downtime updates | |
| 17:46:26 | bauzas | cool | |
| 17:46:27 | clarkb | maybe that wasn't clear in my comments in the release channel. Its fully rolling and starts at 00:00 UTC saturday and usually takes about a day | |
| 17:53:04 | dansmith | ah I assumed that meant some queue resets | |
| 18:18:51 | opendevreview | melanie witt proposed openstack/nova master: testing: Reset affinity support global variables https://review.opendev.org/c/openstack/nova/+/875991 | |
| 18:19:07 | gmann | bauzas: +1 on prelude | |
| 18:35:21 | clarkb | dansmith: we've been doing fully automatic weekly rolling zuul upgrades since like july? | |
| 18:35:56 | dansmith | clarkb: cool, didn't know (which I guess means it's working) | |
| 18:35:56 | clarkb | and before that we were doing them manually. Its been about a year since we had to intentionally take a zuul downtime for normal updates (they do still occasionally happen to apply bug fixes or security updates more quickly) | |
| 20:41:44 | opendevreview | Merged openstack/nova stable/yoga: add repoducer test for bug 1890244 https://review.opendev.org/c/openstack/nova/+/872663 | |
| 20:42:04 | sean-k-mooney1 | bauzas: https://bugs.launchpad.net/nova/+bug/2008883 | |
| 20:42:35 | sean-k-mooney1 | bauzas: did you ever test 2 mdev types with MIG on the same card concurrently | |
| 20:43:24 | sean-k-mooney1 | bauzas: i think this would need an a100 to test and we normally only used t4's so im not sure if we have ever tested that | |
| 22:14:57 | opendevreview | Merged openstack/nova master: Add the 2023.1 Antelope prelude section https://review.opendev.org/c/openstack/nova/+/875380 | |
| #openstack-nova - 2023-03-03 | |||
| 06:15:48 | opendevreview | Merged openstack/nova master: Add service version for Antelope https://review.opendev.org/c/openstack/nova/+/874932 | |
| 08:01:24 | opendevreview | Jorge San Emeterio proposed openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127 | |
| 08:41:20 | bauzas | looks like I shouldn't gamble with poker | |
| 08:42:08 | bauzas | all my rechecks were failing, while the new ones this night (thanks dansmith) got eventually merged | |
| 09:30:45 | opendevreview | Amit Uniyal proposed openstack/nova master: Allow swap resize from non-zero to zero https://review.opendev.org/c/openstack/nova/+/857339 | |
| 09:42:01 | admin1 | hi all .. what exactly does this mean ( which came during yoga -> zed upgrade) and now nova is down -- https://gist.githubusercontent.com/a1git/6f25cfb53feb2cb3b6d122da5664b462/raw/5bb0563f46c05bcea47fbc2a04f607f1f045f4ab/gistfile1.txt | |
| 09:45:16 | admin1 | Details: Current Nova version does not support computes older than |", "| Yoga but the minimum compute service level in your system |", "| is 60 and the oldest supported service level is 61" | |
| 10:08:10 | bauzas | admin1: are you sure *all* your computes are supporting at least Yoga ? | |
| 10:08:25 | admin1 | yes | |
| 10:08:35 | admin1 | those were upgraded 3 days prior and i had a canary deployed in all of them | |
| 10:08:43 | bauzas | something detected a compute having a 60 service version | |
| 10:08:45 | admin1 | now yoga -> zed, ( using openstack ansible) it just failed | |
| 10:09:18 | bauzas | admin1: can you please create a bug report ? | |
| 10:09:25 | bauzas | I'll try to look at it | |
| 10:09:32 | admin1 | bauzas, is it possible to somehow bypass or disable this check for a bit ? | |
| 10:09:42 | bauzas | indeed, sec | |
| 10:11:56 | bauzas | admin1: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_compute_service_check_for_ffu | |
| 10:12:37 | bauzas | but you'll need to find which compute is still using the older service version | |
| 10:12:45 | bauzas | maybe some of them wasn't restarted | |
| 10:12:52 | bauzas | (after upgrading) | |
| 10:14:30 | admin1 | output of select host,version from nova.services => https://gist.githubusercontent.com/a1git/13ceb2e181dab9532a5b229b2915b478/raw/eee560226905d21159808f36a038c9201d8e2d23/gistfile1.txt | |
| 10:14:37 | admin1 | i think some are affected | |
| 10:18:53 | bauzas | admin1: what's strange is that 60 is an interim service version | |
| 10:19:11 | bauzas | did you use a milestone or something for some computes ? | |
| 10:19:43 | bauzas | https://github.com/openstack/nova/blob/master/nova/objects/service.py#L260-L261 Xena is 57 and Yoga is 61 | |
| 10:20:12 | bauzas | https://github.com/openstack/nova/blob/master/nova/objects/service.py#L212-L215 and that's what was changed by the RPC version for the 60 service version | |
| 10:20:57 | bauzas | unless you deploy with master, of course | |
| 10:25:18 | admin1 | bauzas, so the servers that are 60 were not online .. | |
| 10:25:33 | admin1 | they were off temporarily | |
| 10:25:40 | admin1 | but this blocked the whole upgrade process | |
| 10:26:17 | bauzas | the compute state isn't and shouldn't be checked for safety reasons | |
| 10:26:56 | admin1 | it did .. temporarily what i did was update nova.services set version=61 where version=60 and trying to run the playbook again | |
| 10:27:12 | admin1 | if it works, then i am all good .. else i have to report it here again | |
| 10:27:27 | admin1 | if this works, then i can open a bug report saying unavailable compute node blocked upgrade | |
| 10:27:50 | bauzas | https://docs.openstack.org/nova/latest/cli/nova-status.html#nova-status-checks helps to test your upgrade | |
| 10:28:27 | bauzas | admin1: again, that's by design that we don't allow non-upgraded compute to be left registered | |
| 10:28:44 | bauzas | admin1: and that's why we have the workaround option for that intent | |
| 10:29:54 | bauzas | https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L251 | |
| 10:30:39 | bauzas | which exactly tests the support contract *before* you upgrade https://github.com/openstack/nova/blob/59f7a524fd4ded3c17b10abcedb0baff769c3a8a/nova/utils.py#L1052 | |
| 11:07:51 | sean-k-mooney | admin1: havign the server be off is expected to block the upgrade process | |
| 11:08:04 | sean-k-mooney | that would not be a bug | |
| 11:08:40 | sean-k-mooney | because if the service verion is 60 it means they never started with the offical yoga release (61) | |
| 11:09:08 | opendevreview | Amit Uniyal proposed openstack/nova master: Allow swap resize from non-zero to zero https://review.opendev.org/c/openstack/nova/+/857339 | |
| 11:09:11 | sean-k-mooney | if you had started them with yoga and then they were stop it would not cause this issue | |
| 11:09:43 | admin1 | i think they were off 2 days before when i did wallaby -> yoga | |
| 11:09:54 | admin1 | but wallaby -> yoga did not complained of this .. was this check added in zed ? | |
| 11:10:36 | sean-k-mooney | this was alwasy a requirement and we decied to start enforcining it in yoga becasue of operators violating the upgrade contract | |
| 11:10:41 | sean-k-mooney | and filing bugs :) | |
| 11:10:44 | admin1 | :D | |
| 11:11:37 | sean-k-mooney | nova before 2023.1/2024.1 only allows n to n+1 upgrades | |
| 11:11:54 | sean-k-mooney | we put the workaround option in place ans an escape hatch | |
| 11:12:42 | sean-k-mooney | so if you want to run nova in an unsupproted state you can but it should never be requried if you are upgrading withing the upgrade contract | |
| 11:13:02 | admin1 | i understand now .. will make sure no computes are down next time we upgrade | |
| 11:15:08 | sean-k-mooney | provided we do not do an rpc bump its generally possibel for > n->n+1 to function but the first time we tested that was yoga to antelope(2023.1) as a dry run for 2023.1->2024.1 | |
| 11:16:57 | sean-k-mooney | we will be offically testing that going forward in case you are not aware of this chagne https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adjustment.html | |
| 11:19:32 | opendevreview | Amit Uniyal proposed openstack/nova master: Allow swap resize from non-zero to zero https://review.opendev.org/c/openstack/nova/+/857339 | |
| 14:24:11 | dansmith | bauzas: \o/ | |
| 14:33:58 | dansmith | bauzas: are you cooking up the rc1 patch? | |
| 14:34:15 | bauzas | I was waiting for the revert to arrive | |