| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-24 | |||
| 12:35:53 | cdent | you're very welcome | |
| 12:36:12 | aspiers | no amount of rubber-ducking would have helped on that one, I fear | |
| 12:36:49 | aspiers | :) | |
| 12:38:30 | openstackgerrit | Adam Spiers proposed openstack/nova master: Move selective patching of open() to nova.test for reuse https://review.opendev.org/661266 | |
| 12:38:47 | aspiers | cdent: ^^^ | |
| 12:39:02 | aspiers | BTW I think I fixed the contextlib issue in the parent of that one | |
| 12:41:17 | cdent | \o/ | |
| 12:51:53 | efried | tonyb: done | |
| 12:54:59 | aspiers | cdent: just noticed some functional tests are calling os.path.exists('/etc/policy.d') | |
| 12:55:13 | aspiers | presumably that's not good in a functional test environment | |
| 12:55:42 | cdent | There's a few bleeds like that in the nova functional tests | |
| 12:55:59 | aspiers | also .netrc, yikes | |
| 12:56:14 | cdent | I remember there were some (since fixed) I think, which would be change behavior by using the /etc/nova/nova.conf | |
| 12:58:38 | aspiers | my new helpers might help clean some of those cases up | |
| 13:06:45 | openstackgerrit | Silvan Kaiser proposed openstack/nova stable/stein: Fixes multi-registry config in Quobyte driver https://review.opendev.org/660706 | |
| 13:24:10 | openstackgerrit | Eric Fried proposed openstack/nova master: Clarify --before help text in nova manage https://review.opendev.org/661289 | |
| 13:29:52 | stephenfin | efried: I did not. Thanks for the heads up :) | |
| 13:35:45 | stephenfin | efried: So TripleO is being handled (mschuppert is on the case) and I'm working through the Kolla change. That's pretty much all we need, right? | |
| 13:37:50 | efried | stephenfin: IIUC from yesterday's meeting, yes. | |
| 13:37:55 | lyarwood | mriedem: prior to https://review.opendev.org/#/q/topic:bug/1469179+(status:open+OR+status:merged) the only real way to avoid over allocation issues when booting from volumes was to use flavors with a disk size of 0 right? | |
| 13:38:06 | efried | stephenfin: Assume you caught up on those logs (which prompted your ML thread)? | |
| 13:49:24 | mriedem | lyarwood: sounds right | |
| 13:49:46 | openstackgerrit | Dan Smith proposed openstack/nova master: Make nova-next archive using --before https://review.opendev.org/661002 | |
| 13:50:24 | mriedem | can i get a novaclient core to review https://review.opendev.org/#/c/659886/ since it's holding up a release? both tssurya and i have things in the upcoming novaclient release that are needed for enabling changes in other projects (osc and watcher) | |
| 13:55:31 | efried | efried@efried-ThinkPad-W520:~/openstack/nova$ f "$(date)" | |
| 13:55:31 | efried | Fri | |
| 13:55:31 | efried | efried@efried-ThinkPad-W520:~/openstack/nova$ f $(date) | |
| 13:55:31 | efried | > } | |
| 13:55:31 | efried | > echo $1 | |
| 13:55:31 | efried | efried@efried-ThinkPad-W520:~/openstack/nova$ f() { | |
| 13:55:31 | efried | dansmith: | |
| 13:55:32 | efried | Fri May 24 08:53:45 CDT 2019 | |
| 13:55:57 | efried | unless argparse is doing something magical... | |
| 13:56:03 | dansmith | efried: what? | |
| 13:56:16 | efried | see comment on https://review.opendev.org/661002 | |
| 13:56:52 | aspiers | Yeah, that comment looks right | |
| 13:56:57 | dansmith | oh yeah | |
| 13:57:22 | openstackgerrit | Dan Smith proposed openstack/nova master: Make nova-next archive using --before https://review.opendev.org/661002 | |
| 13:57:26 | aspiers | efried: BTW I can't believe you are still using a W520 | |
| 13:57:41 | aspiers | I mean, I am too, but only as a VPN server hidden under my desk | |
| 13:58:10 | efried | aspiers: Heh, it was my IBM-provided workstation, did a nice SSD and memory upgrade, it was really stable and solid, so I bought it back when I left. | |
| 13:58:15 | aspiers | Mine weighed a ton and the battery used to last 2 hours on a good day | |
| 13:58:39 | efried | My Intel-provided HP thingy has roughly equivalent specs, weighs about a quarter as much... and performs like a dog. | |
| 13:58:57 | efried | ...and won't run not-Windows. | |
| 13:59:06 | efried | (because of Intel gorp) | |
| 13:59:19 | efried | (without major hoops of fire) | |
| 14:05:33 | openstackgerrit | Silvan Kaiser proposed openstack/nova stable/stein: Fixes multi-registry config in Quobyte driver https://review.opendev.org/660706 | |
| 14:10:14 | mriedem | efried: we might want to see https://review.opendev.org/#/c/661002/ working in action before +2ing :) | |
| 14:11:19 | efried | mriedem: It worked at a previous PS | |
| 14:11:22 | efried | as in, didn't blow up | |
| 14:11:28 | efried | syntax weirdness notwithstanding | |
| 14:11:47 | efried | i.e. archived rows older than [earlier date than intended] | |
| 14:12:15 | efried | presumably all the rows in that job are newer than "yesterday" anyway, so the change is a no-op? | |
| 14:12:17 | mriedem | http://logs.openstack.org/02/661002/1/check/nova-next/d88596c/logs/devstack-gate-post_test_hook.txt.gz#_2019-05-23_23_53_30_031 | |
| 14:12:21 | mriedem | nova-manage: error: unrecognized arguments: --max-rows | |
| 14:12:48 | efried | oh, that doesn't cause the test to fail? Guess that should be fixed :) | |
| 14:12:53 | mriedem | we should probably have a set -e around this b/c it's blowing up but not failing | |
| 14:13:08 | efried | okay | |
| 14:14:49 | mriedem | set -e in a separate change so we can backport it imo | |
| 14:15:29 | finucannot | efried: I did, yup, cheers. The kolla patch has a +2 already too. Anything else needed to get that approved now? :) | |
| 14:21:16 | openstackgerrit | Dan Smith proposed openstack/nova master: Make nova-next archive using --before https://review.opendev.org/661002 | |
| 14:30:51 | melwitt | dansmith: do you recall whether we exclude certain exceptions from incrementing the build_failed counter for the auto-disabling of computes? I thought it was discussed in the past but looking at the code I don't see anything like that https://github.com/openstack/nova/blob/stable/queens/nova/compute/manager.py#L1789 | |
| 14:31:24 | dansmith | yes, there are two classes | |
| 14:32:30 | dansmith | so things we catch and handle in a certain way don't count, | |
| 14:32:41 | dansmith | and anything we don't catch counts | |
| 14:32:48 | dansmith | and then some gray in the middle | |
| 14:33:12 | dansmith | and part of the problem is we don't do nearly as good at distinguishing as we thought we did | |
| 14:34:07 | dansmith | hmm, although looking again, maybe we lumped them all together in the end after that realization | |
| 14:34:56 | melwitt | yeah, I don't remember this, I thought there was some kind of whitelist that wouldn't increment the counter but not seeing it now | |
| 14:35:32 | dansmith | I think it was the failed/rescheduled result | |
| 14:35:40 | dansmith | but we increment for both now | |
| 14:35:55 | melwitt | I see | |
| 14:36:47 | openstackgerrit | Eric Fried proposed openstack/nova master: Clarify --before help text in nova manage https://review.opendev.org/661289 | |
| 14:36:50 | mriedem | there is no whitelist | |
| 14:36:52 | mriedem | we talked about it | |
| 14:36:55 | mriedem | but it got out of hand | |
| 14:37:08 | mriedem | i had a wip patch for a bit for one type of exception but it was gross and dropped it | |
| 14:37:32 | mriedem | there are about a billion ways you can get an exception in that build path in the compute | |
| 14:37:41 | dansmith | yeah, that's what I mean by we thought it would be clearer which things are terminal failures or not, but it's not | |
| 14:37:48 | mriedem | and since it's python and not strictly typing any of that stuff, we can't really just catch one | |
| 14:38:10 | dansmith | well, I dunno about _that_ :) | |
| 14:39:02 | mriedem | well, meaning like in java where you have to declare and explicitly handle exceptions that get raised up during compiling | |
| 14:39:14 | dansmith | ah sure | |
| 14:39:27 | dansmith | i.e. "the most annoying part of java" | |
| 14:39:35 | mriedem | zvm driver could add a new MainframeKaputError to raise and we'd not handle it | |
| 14:39:43 | mriedem | if you're ocd like me it's not so bad :) | |
| 14:40:30 | mriedem | i'd like some credit for my witty use of a german word in that zvm joke please... | |
| 14:40:46 | melwitt | ok, this is sounding familiar now, I vaguely remember your wip patch about it | |
| 14:40:58 | melwitt | oh yeah, kaputt, nicely done | |
| 14:41:11 | mriedem | melwitt: so is your customer on just a release that is too old to have the weigher? | |
| 14:41:48 | melwitt | no, they have the weigher, it's just incrementing the counter for user-caused build failures we think | |
| 14:41:48 | efried | finucannot: If two out of three of melwitt, mriedem, and sean-k-mooney ack it, I'll be happy to approve it (nova-consoleauth removal bp) | |
| 14:42:02 | dansmith | that's the whole thing | |
| 14:42:07 | dansmith | that's why we moved it to a weigher | |
| 14:42:21 | dansmith | because we suck so bad at knowing why a thing failed that auto-disable isn't really sane | |
| 14:42:23 | mriedem | melwitt: malicious user? | |
| 14:42:41 | dansmith | so we made it a weigher so it doesn't take it out of rotation and you can adjust how much score you give to that decision | |
| 14:42:51 | melwitt | mriedem: I doubt it but not sure | |
| 14:43:22 | mriedem | so jed at the bank wrote a script to boot 100 vms at once and it's DoSing their private cloud | |
| 14:43:58 | melwitt | dansmith: got it. yeah, IIUC this customer is trying to pack instances and they see it start spreading instances when they don't want it to, because of the user-caused failures | |