| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-16 | |||
| 15:26:22 | dansmith | way ahead of you | |
| 15:26:50 | gibi | done | |
| 15:26:52 | bauzas | dansmith: man, you're way too much productive compared to me | |
| 15:27:08 | dansmith | I was just waiting for it after the mention above ;) | |
| 15:27:12 | melwitt | seeya soon, highlights o/ | |
| 15:27:16 | bauzas | ++ | |
| 15:27:33 | dansmith | oh man, an early morning melwitt spotting! | |
| 15:27:40 | dansmith | well, "early" | |
| 15:28:34 | melwitt | uh huh | |
| 15:29:01 | dansmith | :P | |
| 16:05:24 | fungi | elodilles: i noticed yesterday you were pushing forward on some nova stable branch changes (thanks!) and wanted to mention that the vmt is waiting for the rest of https://review.opendev.org/q/I95f68be76330ff09e5eabb5ef8dd9a18f5547866 to merge so we can publish errata for ossa-2021-002 | |
| 16:06:19 | fungi | sean-k-mooney: on that note, the failures on 806629 imply that fix may not be backportable as-is to train? | |
| 16:07:03 | fungi | looks like it could be as simple as a couple of missing imports though | |
| 16:10:07 | melwitt | fungi: hm, looks like the original fix didn't merge yet on train as well | |
| 16:10:38 | fungi | melwitt: oh, interesting yeah | |
| 16:11:31 | fungi | that being 791807 | |
| 16:11:40 | melwitt | fungi: I'll help review the ones that I didn't propose, I didn't realize we needed to wait for ussuri and train, I had thought the original fix notice went out after the victoria change merged | |
| 16:12:34 | melwitt | yeah | |
| 16:13:11 | fungi | melwitt: we need stable/ussuri merged first ideally in this case, since it's still in a maintained state (projected transition to extended maintenance is november) | |
| 16:13:37 | melwitt | fungi: ack, will prioritize | |
| 16:13:38 | fungi | i didn't realize the original train fix never got approved either | |
| 16:14:15 | fungi | but you're right that's less important, i can in theory link to the unmerged patch for train though i'm hesitant to do so unless it's actually passing ci jobs | |
| 16:15:18 | fungi | as i could be inadvertently encouraging someone to merge a change which just causes nova to start crashing | |
| 16:15:33 | fungi | er, to import a change into their deployment i mean | |
| 16:17:25 | opendevreview | Elod Illes proposed openstack/nova stable/ussuri: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806628 | |
| 16:17:26 | melwitt | fungi: yeah makes sense. the train change (the original) is a combo of two patches squashed together bc a test change was needed in order for the test to run on python < 3.6. and the followup should be rebased over it | |
| 16:18:04 | sean-k-mooney | so presumable if we rebase the train one on top of the first patch that might fix it? | |
| 16:18:06 | fungi | aha, that 'splains the errors. i can push that | |
| 16:18:12 | fungi | unless someone else is already on it | |
| 16:19:21 | fungi | ahh, though there are merge conflicts between them as well, so i'll defer to someone familiar with doing nova backports | |
| 16:19:36 | fungi | i don't want to get the conflicts stuff in the commit message wrong | |
| 16:19:47 | sean-k-mooney | im on a call but if melwitt does not get to it i can try and stack them | |
| 16:19:54 | fungi | thanks! | |
| 16:20:37 | elodilles | fungi melwitt : I've updated the commit message of the ussuri patch ^^^ | |
| 16:20:45 | melwitt | thanks elodilles | |
| 16:20:48 | fungi | thanks elodilles!!! | |
| 16:21:12 | elodilles | so that it's nice and clean (& ready to merge) | |
| 16:21:13 | fungi | trying to make sure this ossa update doesn't fall through the cracks | |
| 16:23:39 | melwitt | ++ thanks fungi | |
| 16:32:34 | melwitt | sean-k-mooney: I've got it | |
| 16:32:57 | opendevreview | Merged openstack/placement master: Update master for stable/xena https://review.opendev.org/c/openstack/placement/+/809365 | |
| 16:34:40 | opendevreview | melanie witt proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629 | |
| 16:34:46 | clarkb | Hello, I've got a cloud (the inmotion cloud you might see some of your CI jobs running on) that nodepool is failing to boot instances on. | |
| 16:35:11 | clarkb | the compute log says things like Instance f98ce366-90b1-43ba-8513-bf2ea559c931 has allocations against this compute host but is not found in the database. | |
| 16:35:27 | clarkb | and the conductor log says things like nova.exception_Remote.NoValidHost_Remote: No valid host was found. | |
| 16:35:48 | clarkb | `openstackclient limits show --absolute` shows that we aren't using any quota. | |
| 16:36:07 | clarkb | But it appears that nova is essentially saying the compute hosts are full up and it can't schedule more work? | |
| 16:36:37 | melwitt | clarkb: orphaned allocations in placement :( are making the hosts appear to be using resources when they shouldn't be | |
| 16:37:09 | clarkb | Is there a way to list those without digging into a database? osc server list is empty too | |
| 16:37:43 | melwitt | this is the doc for dealing with that https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html probably just need to run 'nova-manage placement heal_allocations' tool | |
| 16:37:49 | clarkb | thank you | |
| 16:39:44 | fungi | remember to lay your hands on the server when you chant "heal_allocations" too | |
| 16:40:22 | fungi | it doesn't really make the command work any better, but it looks cool if anyone happens to be walking by | |
| 16:40:37 | melwitt | 😂 | |
| 16:42:56 | melwitt | this is just another reminder to prioritize the healing service (that would run heal_allocations periodically on its own, among other things) that's been in the backlog that I haven't had time to work on | |
| 16:58:26 | opendevreview | Merged openstack/nova master: Add the Xena prelude section https://review.opendev.org/c/openstack/nova/+/807786 | |
| 17:40:34 | melwitt | elodilles: oh no, l-c job failure again https://review.opendev.org/c/openstack/nova/+/806628 | |
| 17:47:33 | sean-k-mooney | The user requested decorator>=3.4.0 | |
| 17:47:35 | sean-k-mooney | The user requested (constraint) decorator==3.4.0 | |
| 17:47:47 | sean-k-mooney | .... | |
| 17:47:54 | sean-k-mooney | that does not feel like it should be a conflict | |
| 17:48:25 | sean-k-mooney | since 3.4.0==3.4.0 is true and 3.4.0>=3.4.0 is also true | |
| 17:48:37 | melwitt | yeah | |
| 17:48:40 | melwitt | there's also this error "error in decorator setup command: use_2to3 is invalid." | |
| 17:48:41 | clarkb | I think fungi said those issues may come up when pypi serves us stale indexes | |
| 17:48:44 | melwitt | I don't know what that means | |
| 17:48:53 | sean-k-mooney | oh error in decorator setup command: use_2to3 is invalid. | |
| 17:48:53 | clarkb | the use_2to3 error is due to a new setuptools I think they removed that flag | |
| 17:49:03 | sean-k-mooney | so this is not deps related | |
| 17:49:09 | sean-k-mooney | ya | |
| 17:49:17 | melwitt | https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#792 | |
| 17:49:23 | sean-k-mooney | the deps are fine but its unhappy with setuptools | |
| 17:49:26 | clarkb | basically python software with modern pypa tools are expected to be python3 and not converted | |
| 17:50:01 | sean-k-mooney | clarkb: is there a compat flag we can enable | |
| 17:50:07 | sean-k-mooney | or jus tmove this to python 3 | |
| 17:50:47 | melwitt | aside: the jump to link doesn't seem to be working for me lately when I link to a line number in a zuul output | |
| 17:51:08 | sean-k-mooney | ya that only worked for me if the file is small | |
| 17:51:12 | sean-k-mooney | it highlights it | |
| 17:51:21 | sean-k-mooney | but does not move to it if its not loaded quick enough | |
| 17:51:39 | melwitt | yeah. hrm. I wonder if something changed. the above ^ link doesn't jump me to the line and it loads really fast | |
| 17:52:27 | clarkb | sean-k-mooney: I think pypa isn't interested in having compat flags. Updating or replacing deps is probably necessary | |
| 17:52:56 | sean-k-mooney | melwitt: if i open it https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#792 then change it to https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#791 | |
| 17:52:59 | sean-k-mooney | it works fine | |
| 17:53:51 | sean-k-mooney | clarkb: ya or droping lower constraints. i just checked and its on py3 | |
| 17:54:04 | sean-k-mooney | we have https://github.com/openstack/nova/blob/stable/train/tox.ini#L10 | |
| 17:54:20 | sean-k-mooney | and we do not override it for lower constraints | |
| 17:55:13 | sean-k-mooney | clarkb: the other option we have woudl be to downgrade setuptools | |
| 17:55:27 | sean-k-mooney | pin it in the tox env to one that works | |
| 17:55:48 | sean-k-mooney | perhaps usign requires https://tox.readthedocs.io/en/latest/config.html#conf-requires | |
| 17:56:25 | sean-k-mooney | that would be a stable only change i guess if we did that. | |
| 17:56:35 | sean-k-mooney | clarkb: do you know if this is affecting anyone else | |
| 17:57:00 | sean-k-mooney | i know a lot of project just deleted there old lower constriants jobs | |
| 17:58:52 | clarkb | the governance repo had problems with pydot2 which hasn't been maintained for years | |
| 17:59:21 | clarkb | I don't think we want ot downgrade setuptools if we can avoid it. We can't really control what version of setuptools others use effectively and wide compatibility is desireable | |
| 17:59:39 | clarkb | (note with the whole pyproject.toml stuff you do get a bit more control but openstack hasn'tdone any of that) | |
| 18:00:32 | sean-k-mooney | ya its more we have 3 options delete the job, update the dep or hack around to make it work | |
| 18:00:47 | sean-k-mooney | pinnign setup tools on an em branch is just that | |
| 18:00:48 | clarkb | can you bump the dep version up such that it works? | |
| 18:00:53 | sean-k-mooney | a hackaround to make it work | |
| 18:00:58 | clarkb | that is all lower constraints is supposed to track iirc. The oldest version that works | |