Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-03-02
15:20:12 dansmith I don't know why that needs its own job, so I hope we can roll that in,
15:20:19 dansmith but we have no coverage for that currently, unfortunately :/
15:20:49 bauzas today i haven't seen (yet) any volume-related issue
15:21:03 bauzas so far this was timeouts and a few unresponsive calls
15:21:16 bauzas none of them being cinder-related
15:21:36 dansmith well, that's progress IMHO :)
15:21:39 dansmith but yeah, timeouts
15:21:55 bauzas I have a rule of thumb
15:22:06 bauzas I had problems during the week of milestone-3
15:22:23 bauzas milestone-3 went and the weekend after that, all patches in queue merged without problems
15:22:29 bauzas now we're on RC1 period
15:22:53 bauzas the gate is mostly busted and the check pipeline is horribly on delay
15:23:05 bauzas I just think we should be able to branch RC1 this weekend, yay
15:23:48 bauzas anyway, me needs to disappear for a dozen of minutes for parenting responsibilities
15:24:01 bauzas bbias
15:32:53 dosaboy hi folks, if anyone has time to land these yoga backports it would be greatly appreciated - https://review.opendev.org/q/topic:bug%252F1890244
15:33:19 dosaboy they've got a +2 already
15:34:39 sean-k-mooney1 i fixed that... hum does that mean i need to drink more or less coffee if i dont remember it
15:35:31 dosaboy more coffee always, that's my M.O. at least
15:57:20 bauzas dosaboy: I'll give them a swing
16:00:03 bauzas dosaboy: aaaand done
16:35:36 dosaboy bauzas: many thanks!
16:35:52 bauzas np
16:40:40 bauzas I'm saying it here too, but we're busted to get the RC1 patch being merged today
17:38:30 bauzas folks, I end my day on a sad note : since we can't deliver RC1 today, we'll try again tomorrow
17:39:40 bauzas that's a bummer, but looks like nobody really cares about it
17:40:07 dansmith bauzas: is this because you can't merge the log revert?
17:40:23 bauzas that + the prelude + the service object bump
17:40:28 bauzas 3 patches on fly
17:40:35 dansmith oh right, those, gotcha
17:40:55 bauzas we should end up releasing RC1 on the weekend
17:40:55 dansmith we can certainly merge the prelude right?
17:41:04 bauzas dansmith: the prelude is still in flight
17:41:22 bauzas oh my bad, not
17:41:27 dansmith bauzas: in flight meaning not reviewed by enough people right? not because you can't land due to tests
17:41:37 bauzas dansmith: send it to the gate, folks that care should have voted earlier
17:41:52 dansmith done
17:42:00 bauzas yeah, my bad I was paying attention to the two others
17:42:17 bauzas the revert logging may be merged later and backported
17:42:24 dansmith yeah the revert is not a huge deal
17:42:26 bauzas this is just a silly logging
17:42:30 dansmith would be nice, but not critical
17:42:40 dansmith I'll watch the service version one
17:42:43 bauzas but the prelude has to land, ditto the service check
17:42:46 dansmith yeah
17:43:00 bauzas thanks for caring this
17:43:08 bauzas I'll look at those later today
17:43:29 dansmith maybe if I can grind these two you can post the release tomorrow morning
17:44:01 bauzas maybe
17:44:08 bauzas gerrit will be restarted later today
17:44:24 bauzas that was an ask from the infra team
17:44:35 bauzas zuul will only be recycled this weekend
17:44:35 dansmith oye
17:46:03 clarkb note zuul is zero downtime updates
17:46:26 bauzas cool
17:46:27 clarkb maybe that wasn't clear in my comments in the release channel. Its fully rolling and starts at 00:00 UTC saturday and usually takes about a day
17:53:04 dansmith ah I assumed that meant some queue resets
18:18:51 opendevreview melanie witt proposed openstack/nova master: testing: Reset affinity support global variables https://review.opendev.org/c/openstack/nova/+/875991
18:19:07 gmann bauzas: +1 on prelude
18:35:21 clarkb dansmith: we've been doing fully automatic weekly rolling zuul upgrades since like july?
18:35:56 dansmith clarkb: cool, didn't know (which I guess means it's working)
18:35:56 clarkb and before that we were doing them manually. Its been about a year since we had to intentionally take a zuul downtime for normal updates (they do still occasionally happen to apply bug fixes or security updates more quickly)
20:41:44 opendevreview Merged openstack/nova stable/yoga: add repoducer test for bug 1890244 https://review.opendev.org/c/openstack/nova/+/872663
20:42:04 sean-k-mooney1 bauzas: https://bugs.launchpad.net/nova/+bug/2008883
20:42:35 sean-k-mooney1 bauzas: did you ever test 2 mdev types with MIG on the same card concurrently
20:43:24 sean-k-mooney1 bauzas: i think this would need an a100 to test and we normally only used t4's so im not sure if we have ever tested that
22:14:57 opendevreview Merged openstack/nova master: Add the 2023.1 Antelope prelude section https://review.opendev.org/c/openstack/nova/+/875380
#openstack-nova - 2023-03-03
06:15:48 opendevreview Merged openstack/nova master: Add service version for Antelope https://review.opendev.org/c/openstack/nova/+/874932
08:01:24 opendevreview Jorge San Emeterio proposed openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127
08:41:20 bauzas looks like I shouldn't gamble with poker
08:42:08 bauzas all my rechecks were failing, while the new ones this night (thanks dansmith) got eventually merged
09:30:45 opendevreview Amit Uniyal proposed openstack/nova master: Allow swap resize from non-zero to zero https://review.opendev.org/c/openstack/nova/+/857339
09:42:01 admin1 hi all .. what exactly does this mean ( which came during yoga -> zed upgrade) and now nova is down -- https://gist.githubusercontent.com/a1git/6f25cfb53feb2cb3b6d122da5664b462/raw/5bb0563f46c05bcea47fbc2a04f607f1f045f4ab/gistfile1.txt
09:45:16 admin1 Details: Current Nova version does not support computes older than |", "| Yoga but the minimum compute service level in your system |", "| is 60 and the oldest supported service level is 61"
10:08:10 bauzas admin1: are you sure *all* your computes are supporting at least Yoga ?
10:08:25 admin1 yes
10:08:35 admin1 those were upgraded 3 days prior and i had a canary deployed in all of them
10:08:43 bauzas something detected a compute having a 60 service version
10:08:45 admin1 now yoga -> zed, ( using openstack ansible) it just failed
10:09:18 bauzas admin1: can you please create a bug report ?
10:09:25 bauzas I'll try to look at it
10:09:32 admin1 bauzas, is it possible to somehow bypass or disable this check for a bit ?
10:09:42 bauzas indeed, sec
10:11:56 bauzas admin1: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_compute_service_check_for_ffu
10:12:37 bauzas but you'll need to find which compute is still using the older service version
10:12:45 bauzas maybe some of them wasn't restarted
10:12:52 bauzas (after upgrading)
10:14:30 admin1 output of select host,version from nova.services => https://gist.githubusercontent.com/a1git/13ceb2e181dab9532a5b229b2915b478/raw/eee560226905d21159808f36a038c9201d8e2d23/gistfile1.txt
10:14:37 admin1 i think some are affected
10:18:53 bauzas admin1: what's strange is that 60 is an interim service version
10:19:11 bauzas did you use a milestone or something for some computes ?
10:19:43 bauzas https://github.com/openstack/nova/blob/master/nova/objects/service.py#L260-L261 Xena is 57 and Yoga is 61
10:20:12 bauzas https://github.com/openstack/nova/blob/master/nova/objects/service.py#L212-L215 and that's what was changed by the RPC version for the 60 service version
10:20:57 bauzas unless you deploy with master, of course
10:25:18 admin1 bauzas, so the servers that are 60 were not online ..
10:25:33 admin1 they were off temporarily
10:25:40 admin1 but this blocked the whole upgrade process
10:26:17 bauzas the compute state isn't and shouldn't be checked for safety reasons
10:26:56 admin1 it did .. temporarily what i did was update nova.services set version=61 where version=60 and trying to run the playbook again
10:27:12 admin1 if it works, then i am all good .. else i have to report it here again

Earlier   Later