| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-16 | |||
| 16:56:17 | efried | I haven't looked into it at all because I haven't yet been bothered to f'ing unzip the logs. | |
| 16:56:35 | sean-k-mooney | has my patch merged to fix that yet | |
| 16:56:46 | sean-k-mooney | you rechecked it this morning/last night right | |
| 16:57:04 | efried | your patch did merge, yes. Not sure if any of the failed jobs started since then. | |
| 16:57:05 | sean-k-mooney | ya looks like it di | |
| 16:57:52 | efried | I don't even know if the failures are the same across the board. I just know many have been failing. (Most of the nova-next failures are due to 'multiple possible networks', which I'm fixing elsewhere.) | |
| 16:58:24 | sean-k-mooney | yes with your tempest patch | |
| 17:00:14 | sean-k-mooney | that should also be merged now. if no one else fixes it ill try and get though these backprots then i can look at ci but proably tomorow or monday. | |
| 17:08:07 | efried | So ++ | |
| 17:08:07 | efried | Besides, I don't see backports being hugely affected by this. | |
| 17:08:07 | efried | I get the gripe about making backports difficult, but the code base will just get messier and messier if we don't bite the bullet and clean this stuff up from time to time. | |
| 17:08:07 | efried | I guess controversy was in previous versions. | |
| 17:08:07 | efried | stephenfin: +2 on https://review.opendev.org/#/c/696745/ | |
| 17:08:23 | stephenfin | thanks | |
| 17:26:46 | efried | I can't even see why https://zuul.opendev.org/t/openstack/build/093f129de3ca47549239eed0d6acf288 is failing. The job output says devstack failed, but the devstack log looks okay to me. Anyone? Bueller? | |
| 17:29:48 | efried | ^ on the subnode. wtf? | |
| 17:29:48 | efried | /opt/stack/new/devstack/lib/databases/mysql: line 64: mysql: command not found | |
| 17:30:06 | artom | Wait, that .gz compression thing hasn't been fixed? | |
| 17:30:28 | efried | artom: sean-k-mooney fixed it somewheres, but this job may have started before that fix merged. | |
| 17:30:30 | efried | Real PITA. | |
| 17:30:39 | artom | Seriously, wit hummus. | |
| 17:30:56 | efried | mm, hummus. | |
| 17:31:41 | sean-k-mooney | artom: https://review.opendev.org/#/c/702465/ meged this morning | |
| 17:32:06 | sean-k-mooney | but if the job started before that merged then ya they would be compressed | |
| 17:32:22 | artom | sean-k-mooney, were they not gziped but transparently decompressed before? | |
| 17:32:34 | sean-k-mooney | yes | |
| 17:32:42 | sean-k-mooney | well no | |
| 17:32:50 | artom | So what killed the transparent decompression? | |
| 17:32:56 | sean-k-mooney | they were gzip and serve with hearders to decompress them | |
| 17:33:01 | sean-k-mooney | but that had issues with swift | |
| 17:33:06 | artom | Oh, the browser did it | |
| 17:33:11 | artom | Client-side | |
| 17:33:34 | sean-k-mooney | yes the borser did but the issue is they were getting double compressed wehn being served form swift | |
| 17:33:47 | sean-k-mooney | so that is why you see the garbage in your browser | |
| 17:34:12 | artom | Oh, does Swift automagically compress text or something? | |
| 17:34:14 | sean-k-mooney | it is still decompressing with deflate but not then decompressing that with gzip | |
| 17:34:41 | openstackgerrit | Vladyslav Drok proposed openstack/nova master: Make removal of host from aggregate consistent https://review.opendev.org/696392 | |
| 17:34:44 | sean-k-mooney | artom: i did not look into it fully but fungi would be able to actully explain it | |
| 17:35:11 | sean-k-mooney | artom: for now just do curl <url> | zcat | less | |
| 17:35:24 | sean-k-mooney | or lnav lnav is good | |
| 17:35:49 | artom | I want to make a `squat` and `rack` command | |
| 17:35:58 | artom | So I can squat | curl | rack | |
| 17:38:37 | sean-k-mooney | looking at the log failrue we are getting nova warning for using deprecated version of qemu and libvirt | |
| 17:39:04 | fungi | sean-k-mooney: we're not clear whether it's a behavior in rackspace's swift implementation or the cdn they put in front of it, but if we upload .gz files with a content-type of application/gzip then they will be transparently decompressed when browsers or other clients request them and indicate they accept that content-type. sounds convenient, right? until you save a .tar.gz file there and it's | |
| 17:39:06 | fungi | quietly decompressed on retrieval without altering the extension, so you've downloaded a file whose name claims it's a gzipped tarball but which is in reality now an uncompressed tar archive. now make that an intermediate step in a release publication job and you're pushing python sdists or similar tarballs somewhere when they're not actually installable (because tools think they're compressed when | |
| 17:39:08 | fungi | they're not) | |
| 17:40:41 | fungi | so we altered the way we store job artifacts in swift to make it so that we transparently compress everything on upload, unless it's already a compressed file, and assume that if the file has been precompressed then it should not be transparently decompressed for requesting clients | |
| 17:40:47 | fungi | artom: ^ | |
| 17:42:06 | efried | clearly fungi has had to explain this before :P | |
| 17:42:20 | artom | fungi, to be honest I didn't grok most of that. My takeaway is: layers do deep even onions are like "whoa dude" | |
| 17:42:23 | sean-k-mooney | yes | |
| 17:42:29 | artom | *so deep | |
| 17:42:54 | sean-k-mooney | but thanks for the explaination again. | |
| 17:43:53 | fungi | we fixed the obvious cases we knew about in abstract jobs/roles but there were some in devstack which had to get fixed too | |
| 17:44:16 | sean-k-mooney | fungi: funnialy enogh the last time i tried deploying zuul at home geting the logs to render properly in the bower when loadbalnace via nginx in k8s was actully one of the hardest parts | |
| 17:44:21 | fungi | and there are also some project-specific jobs which are also precompressing logs from things which will need to be adjusted to stop compressing within the job | |
| 17:44:42 | fungi | which should be updated as we encounter them | |
| 17:45:58 | efried | dansmith: Would you please look at https://review.opendev.org/#/c/696392/ ? I'm not clear whether this counts as an API change... | |
| 17:47:45 | efried | artom: so are you investigating the failure amid all of that? | |
| 17:48:01 | artom | efried, which failure? The missing MySQL? | |
| 17:48:20 | efried | yeah | |
| 17:48:33 | artom | Heh, I'm being volunteered, aren't I? ;) | |
| 17:48:38 | efried | whatever's causing nova-live-migration to fail | |
| 17:48:38 | artom | I'll take a look :) | |
| 17:48:49 | efried | thanks artom. Rather you than me. | |
| 17:49:28 | efried | I'll go check the other instances of n-l-m fail and see if they're the same. | |
| 17:50:18 | artom | sean-k-mooney, zless is a thing, btw :) | |
| 17:50:27 | artom | Don't need to | zcat | less | |
| 17:50:30 | artom | Straight up | zless | |
| 17:50:44 | sean-k-mooney | artom: sure but im actully sendign it to lnav | |
| 17:50:56 | sean-k-mooney | and you use to be able to send it to firefox | |
| 17:51:08 | sean-k-mooney | but they droped readign from standar in at some point | |
| 17:51:33 | sean-k-mooney | but good to know zless is a thing. zgrep is also a thing | |
| 17:51:58 | dansmith | efried: that patch doesn't change what can be raised on error, so not an api change needing a microversion, if that's what you're asking for | |
| 17:52:28 | dansmith | efried: it adds a scan of all cells on delete, which the user doesn't know about anyway | |
| 17:52:57 | efried | artom: so far 5/5 on the ones I've looked at, fails from the last 2h or so. | |
| 17:54:22 | efried | dansmith: okay. afaict it makes it so HostMappingNotFound can no longer be raised when removing host from agg. That's kind of a behavior change, so I wasn't really sure. | |
| 17:54:23 | artom | efried, so, mysql-client should be getting installed | |
| 17:54:35 | efried | artom: I don't disagree. Why isn't it? | |
| 17:54:37 | artom | And it isn't, afaict | |
| 17:54:43 | artom | efried, yeah, no idea what changed yet | |
| 17:54:59 | efried | okay. Maybe -infra knows? | |
| 17:55:22 | dansmith | efried: it's the same error handler as ComputeHostNotFound in the api code though so it doesn't change anything the user can see | |
| 17:56:03 | efried | right, what I mean is that with a certain set of conditions, previously there would *be* an error, but now there would *not* be. | |
| 17:56:17 | efried | ...with those same conditions. | |
| 17:56:23 | efried | I guess that's kind of the definition of a bug fix though | |
| 17:56:30 | efried | so I'm going to +2. | |
| 17:56:34 | dansmith | the user doesn't know about those conditions | |
| 17:56:56 | efried | Okay, I must have misinterpreted that based on the bug description. | |
| 17:56:56 | dansmith | it'd be like saying we failed on mysql 4.1 and succed on 4.2 so that's user-visible | |
| 17:57:04 | efried | okay. | |
| 17:58:38 | artom | efried, what was the last time that job passed? I want to compare logs, see if it installed the mysqlclient | |
| 17:59:14 | efried | I'll look for one... | |
| 17:59:42 | artom | Hrmm, so it installs it on the *main* node | |
| 18:00:03 | efried | artom: here's one from ~24h ago, not sure if that's the most recent https://review.opendev.org/#/c/696745/ | |
| 18:01:04 | artom | efried, that works, thanks | |
| 18:01:19 | artom | Ah, so, on the main node, because we install the server, that pulls in the client | |
| 18:01:59 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Re-proposes multiple vGPU types in libvirt https://review.opendev.org/702943 | |
| 18:06:27 | efried | artom: assume this has nothing to do with cirros images http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012024.html | |
| 18:07:21 | artom | efried, don't think so - looks like it has something to do with whether we configure glance or not | |