| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-27 | |||
| 22:58:01 | melwitt | I noticed someone mentioned in #openstack-qa earlier today that they were able to hit the same error locally https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2021-09-27.log.html#t2021-09-27T19:54:38 | |
| 22:58:12 | johnsom | I am hitting it local | |
| 22:58:24 | melwitt | so hopefully one of us will figure out what's wrong | |
| #openstack-nova - 2021-09-28 | |||
| 00:00:49 | gmann | I am checking with sync mode also in case async tasks are creating issue here | |
| 00:01:31 | gmann | and some logging for list all endpoints though endpoints seems fine in this case but will be helpful for other issues https://review.opendev.org/c/openstack/devstack/+/811246 | |
| 00:15:24 | sean-k-mooney[m] | melwitt if i remember correctly its pulled in by cloud-init | |
| 00:15:49 | sean-k-mooney[m] | just uninstall python-simplejson | |
| 00:16:13 | sean-k-mooney[m] | you have had to do that on fedora in the past and python-yaml too form time to time | |
| 01:01:28 | gmann | seeing same error with async disable too | |
| 06:35:56 | gibi | good morning | |
| 06:36:09 | gibi | I see you had a busy day while I was asleep | |
| 07:32:20 | bauzas | good morning Nova | |
| 07:32:55 | gibi | bauzas: o/ | |
| 07:33:11 | bauzas | yeah I saw this | |
| 07:33:58 | bauzas | I'm glad Ian found the root cause | |
| 07:35:45 | gibi | yepp | |
| 09:27:33 | opendevreview | Merged openstack/nova master: tools: Ignore bot-generated branch creation patches https://review.opendev.org/c/openstack/nova/+/810285 | |
| 10:09:16 | gibi | bauzas: I broke the old qos min bw support in Xena when I extended it with qos min pps support: https://bugs.launchpad.net/nova/+bug/1945310 So this bug is probably a Xena regression, but I will only be certain that it wasn't broken after I backported the repro tests to wallaby to see if the fault is not there | |
| 10:09:35 | bauzas | ah zut | |
| 10:11:48 | gibi | I have the fixs locally so I think we can still fit this in RC2 | |
| 10:12:10 | gibi | but we have a broken gate so maybe not | |
| 10:12:35 | bauzas | yep, I dunno what to say | |
| 10:13:02 | bauzas | gibi: maybe we should accept the RC2 proposal for the moment and see whether the fix could be merged tomorrow for providing a RC3 | |
| 10:13:17 | gibi | bauzas: good point | |
| 10:13:22 | gibi | that is the safest bet I think | |
| 10:13:47 | bauzas | gibi: add a comment for our meeting | |
| 10:13:48 | gibi | so lets talk about this on the meeting today, accept the RC2 if nobody against it and I can push for RC3 | |
| 10:13:53 | bauzas | yep | |
| 10:14:12 | gibi | adding it to the meeting.. | |
| 10:15:26 | gibi | done | |
| 10:17:53 | bauzas | gibi: adding the xena-rc-potential tag to your bug report | |
| 10:18:29 | bauzas | can you triage it ? | |
| 10:18:30 | gibi | ack, there is a slight chance that it was similarly broken before xena too hence my hesitation in that | |
| 10:19:04 | gibi | I will triage it | |
| 10:24:46 | lyarwood | do we not have a bug for the Apache/Placement issue at the moment? | |
| 10:25:43 | gibi | lyarwood: I don't see any | |
| 10:25:47 | gibi | lyarwood: I only see the fix https://review.opendev.org/c/openstack/devstack/+/811303 | |
| 10:26:01 | gibi | partial fix, as grenade still needs love | |
| 10:26:01 | lyarwood | yeah same was just checking I hadn't missed something | |
| 10:26:17 | lyarwood | yeah I can't test that until the devstack fix lands | |
| 10:26:18 | gibi | lyarwood: I guess this is not even a placement bug but a devstack one | |
| 10:26:25 | lyarwood | actually I'll just pull both in | |
| 10:26:42 | gibi | lyarwood: wait, there are two grenade issues | |
| 10:26:48 | lyarwood | oh wait sorry I see what you mean re grenade, we need it fixed on both branches | |
| 10:26:56 | gibi | lyarwood: in the https://review.opendev.org/c/openstack/devstack/+/811303 grenade still fails with the placement issue | |
| 10:26:58 | lyarwood | the devstack issue | |
| 10:27:03 | lyarwood | right | |
| 10:27:20 | lyarwood | yeah so we need to land on stable/xena first | |
| 10:27:24 | lyarwood | then master | |
| 10:27:31 | lyarwood | good old grenade | |
| 10:27:38 | gibi | lyarwood: that was my guess but I wasn't sure | |
| 10:27:54 | gibi | lyarwood: so grenade used the devstack from the stable branch? | |
| 10:28:09 | lyarwood | gibi: yup I believe so | |
| 10:28:16 | gibi | nice | |
| 10:30:07 | elodilles | yes, grenade issues usually have to be fixed first at the oldest branch (where the problem exists) and merge "backward" | |
| 10:30:27 | elodilles | is this such an issue? | |
| 10:31:10 | lyarwood | yeah I think so if it's blocking nova-status checks during the initial install | |
| 10:31:39 | lyarwood | I guess it needs to go back as far as that apache update landed | |
| 10:32:19 | elodilles | :S | |
| 10:32:36 | gibi | and the apache update is in focal | |
| 10:32:46 | gibi | so it effect all our focal grenade jobs? | |
| 10:33:03 | lyarwood | it could, I'm just watching the stable/xena job install stable/wallaby now | |
| 10:33:10 | lyarwood | https://zuul.opendev.org/t/openstack/stream/540dea35d7a84633bb00d706fe02136d?logfile=console.log | |
| 10:33:30 | elodilles | if it is only related to focal then we need to fix it first on victoria afaik | |
| 10:34:39 | elodilles | (victoria is the first branch based on focal) | |
| 10:37:38 | lyarwood | yeah it failed | |
| 10:37:42 | lyarwood | fun times | |
| 10:38:25 | gibi | lyarwood: could you drop a mail to the ML thread about this fun finding? | |
| 10:38:56 | lyarwood | sure | |
| 10:41:31 | gibi | thanks | |
| 10:42:00 | gibi | http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025101.html | |
| 10:59:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811394 | |
| 11:00:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396 | |
| 11:07:09 | lyarwood | gibi: I'm going to hold off posting to the thread, I think people are likely aware of the stable requirement here | |
| 11:07:34 | lyarwood | gibi: but I'm slightly confused by the way forward, it seems that people want to add the trailing / but that's breaking keystone at least on stable/xena | |
| 11:13:32 | gibi | lyarwood: ohh, I haven't checked the stable/xena result yet | |
| 11:17:22 | gibi | lyarwood: do you think a non trailing / version would pass? | |
| 11:17:40 | lyarwood | gibi: it passed on master for the non-grenade jobs | |
| 11:17:46 | gibi | hm, true | |
| 11:18:08 | gibi | so then we have to go back and reason about the non trailing / case | |
| 11:24:56 | lyarwood | yup I think so | |
| 12:13:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396 | |
| 12:19:55 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811405 | |
| 12:21:11 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811407 | |
| 12:28:58 | gibi | bauzas: now I can confirm that https://bugs.launchpad.net/nova/+bug/1945310 is not fully a xena regression, at least parts of it is reproducible in wallaby too | |
| 12:29:17 | bauzas | gibi: ack, then remove the xena regression tag | |
| 12:29:49 | bauzas | gibi: can you backport the reproduction change to stable/wallaby so the CI would test it ? | |
| 12:30:04 | gibi | I'm doing it right now | |
| 12:30:06 | bauzas | cool | |
| 12:30:13 | bauzas | so we don't need a RC3 for it | |
| 12:30:15 | gibi | this is how I confirmed that it is a probelm in wallaby too | |
| 12:30:31 | bauzas | but we will merge it for a 25.0.1 release | |
| 12:30:37 | bauzas | huh | |
| 12:30:40 | bauzas | 24.0.1 | |
| 12:30:45 | gibi | :) | |
| 12:31:04 | bauzas | gibi: thanks for working on it, very appreciated | |
| 12:31:34 | gibi | I broke it :D | |
| 12:31:44 | gibi | so hell yeah I have to work on it | |
| 12:47:50 | jamesdenton_alt | Any ideas as to why Nova might return duplicates for hypervisors and services (same UUIDs)? https://paste.opendev.org/show/809646/ | |
| 13:16:17 | opendevreview | Ivan Kolodyazhny proposed openstack/nova master: Fix NVMe volume attach https://review.opendev.org/c/openstack/nova/+/811413 | |