| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-16 | |||
| 17:31:41 | sean-k-mooney | artom: https://review.opendev.org/#/c/702465/ meged this morning | |
| 17:32:06 | sean-k-mooney | but if the job started before that merged then ya they would be compressed | |
| 17:32:22 | artom | sean-k-mooney, were they not gziped but transparently decompressed before? | |
| 17:32:34 | sean-k-mooney | yes | |
| 17:32:42 | sean-k-mooney | well no | |
| 17:32:50 | artom | So what killed the transparent decompression? | |
| 17:32:56 | sean-k-mooney | they were gzip and serve with hearders to decompress them | |
| 17:33:01 | sean-k-mooney | but that had issues with swift | |
| 17:33:06 | artom | Oh, the browser did it | |
| 17:33:11 | artom | Client-side | |
| 17:33:34 | sean-k-mooney | yes the borser did but the issue is they were getting double compressed wehn being served form swift | |
| 17:33:47 | sean-k-mooney | so that is why you see the garbage in your browser | |
| 17:34:12 | artom | Oh, does Swift automagically compress text or something? | |
| 17:34:14 | sean-k-mooney | it is still decompressing with deflate but not then decompressing that with gzip | |
| 17:34:41 | openstackgerrit | Vladyslav Drok proposed openstack/nova master: Make removal of host from aggregate consistent https://review.opendev.org/696392 | |
| 17:34:44 | sean-k-mooney | artom: i did not look into it fully but fungi would be able to actully explain it | |
| 17:35:11 | sean-k-mooney | artom: for now just do curl <url> | zcat | less | |
| 17:35:24 | sean-k-mooney | or lnav lnav is good | |
| 17:35:49 | artom | I want to make a `squat` and `rack` command | |
| 17:35:58 | artom | So I can squat | curl | rack | |
| 17:38:37 | sean-k-mooney | looking at the log failrue we are getting nova warning for using deprecated version of qemu and libvirt | |
| 17:39:04 | fungi | sean-k-mooney: we're not clear whether it's a behavior in rackspace's swift implementation or the cdn they put in front of it, but if we upload .gz files with a content-type of application/gzip then they will be transparently decompressed when browsers or other clients request them and indicate they accept that content-type. sounds convenient, right? until you save a .tar.gz file there and it's | |
| 17:39:06 | fungi | quietly decompressed on retrieval without altering the extension, so you've downloaded a file whose name claims it's a gzipped tarball but which is in reality now an uncompressed tar archive. now make that an intermediate step in a release publication job and you're pushing python sdists or similar tarballs somewhere when they're not actually installable (because tools think they're compressed when | |
| 17:39:08 | fungi | they're not) | |
| 17:40:41 | fungi | so we altered the way we store job artifacts in swift to make it so that we transparently compress everything on upload, unless it's already a compressed file, and assume that if the file has been precompressed then it should not be transparently decompressed for requesting clients | |
| 17:40:47 | fungi | artom: ^ | |
| 17:42:06 | efried | clearly fungi has had to explain this before :P | |
| 17:42:20 | artom | fungi, to be honest I didn't grok most of that. My takeaway is: layers do deep even onions are like "whoa dude" | |
| 17:42:23 | sean-k-mooney | yes | |
| 17:42:29 | artom | *so deep | |
| 17:42:54 | sean-k-mooney | but thanks for the explaination again. | |
| 17:43:53 | fungi | we fixed the obvious cases we knew about in abstract jobs/roles but there were some in devstack which had to get fixed too | |
| 17:44:16 | sean-k-mooney | fungi: funnialy enogh the last time i tried deploying zuul at home geting the logs to render properly in the bower when loadbalnace via nginx in k8s was actully one of the hardest parts | |
| 17:44:21 | fungi | and there are also some project-specific jobs which are also precompressing logs from things which will need to be adjusted to stop compressing within the job | |
| 17:44:42 | fungi | which should be updated as we encounter them | |
| 17:45:58 | efried | dansmith: Would you please look at https://review.opendev.org/#/c/696392/ ? I'm not clear whether this counts as an API change... | |
| 17:47:45 | efried | artom: so are you investigating the failure amid all of that? | |
| 17:48:01 | artom | efried, which failure? The missing MySQL? | |
| 17:48:20 | efried | yeah | |
| 17:48:33 | artom | Heh, I'm being volunteered, aren't I? ;) | |
| 17:48:38 | efried | whatever's causing nova-live-migration to fail | |
| 17:48:38 | artom | I'll take a look :) | |
| 17:48:49 | efried | thanks artom. Rather you than me. | |
| 17:49:28 | efried | I'll go check the other instances of n-l-m fail and see if they're the same. | |
| 17:50:18 | artom | sean-k-mooney, zless is a thing, btw :) | |
| 17:50:27 | artom | Don't need to | zcat | less | |
| 17:50:30 | artom | Straight up | zless | |
| 17:50:44 | sean-k-mooney | artom: sure but im actully sendign it to lnav | |
| 17:50:56 | sean-k-mooney | and you use to be able to send it to firefox | |
| 17:51:08 | sean-k-mooney | but they droped readign from standar in at some point | |
| 17:51:33 | sean-k-mooney | but good to know zless is a thing. zgrep is also a thing | |
| 17:51:58 | dansmith | efried: that patch doesn't change what can be raised on error, so not an api change needing a microversion, if that's what you're asking for | |
| 17:52:28 | dansmith | efried: it adds a scan of all cells on delete, which the user doesn't know about anyway | |
| 17:52:57 | efried | artom: so far 5/5 on the ones I've looked at, fails from the last 2h or so. | |
| 17:54:22 | efried | dansmith: okay. afaict it makes it so HostMappingNotFound can no longer be raised when removing host from agg. That's kind of a behavior change, so I wasn't really sure. | |
| 17:54:23 | artom | efried, so, mysql-client should be getting installed | |
| 17:54:35 | efried | artom: I don't disagree. Why isn't it? | |
| 17:54:37 | artom | And it isn't, afaict | |
| 17:54:43 | artom | efried, yeah, no idea what changed yet | |
| 17:54:59 | efried | okay. Maybe -infra knows? | |
| 17:55:22 | dansmith | efried: it's the same error handler as ComputeHostNotFound in the api code though so it doesn't change anything the user can see | |
| 17:56:03 | efried | right, what I mean is that with a certain set of conditions, previously there would *be* an error, but now there would *not* be. | |
| 17:56:17 | efried | ...with those same conditions. | |
| 17:56:23 | efried | I guess that's kind of the definition of a bug fix though | |
| 17:56:30 | efried | so I'm going to +2. | |
| 17:56:34 | dansmith | the user doesn't know about those conditions | |
| 17:56:56 | efried | Okay, I must have misinterpreted that based on the bug description. | |
| 17:56:56 | dansmith | it'd be like saying we failed on mysql 4.1 and succed on 4.2 so that's user-visible | |
| 17:57:04 | efried | okay. | |
| 17:58:38 | artom | efried, what was the last time that job passed? I want to compare logs, see if it installed the mysqlclient | |
| 17:59:14 | efried | I'll look for one... | |
| 17:59:42 | artom | Hrmm, so it installs it on the *main* node | |
| 18:00:03 | efried | artom: here's one from ~24h ago, not sure if that's the most recent https://review.opendev.org/#/c/696745/ | |
| 18:01:04 | artom | efried, that works, thanks | |
| 18:01:19 | artom | Ah, so, on the main node, because we install the server, that pulls in the client | |
| 18:01:59 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Re-proposes multiple vGPU types in libvirt https://review.opendev.org/702943 | |
| 18:06:27 | efried | artom: assume this has nothing to do with cirros images http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012024.html | |
| 18:07:21 | artom | efried, don't think so - looks like it has something to do with whether we configure glance or not | |
| 18:07:50 | artom | The broken one runs through this: | |
| 18:07:51 | artom | 2020-01-16 14:30:39.889 | + lib/glance:init_glance:277 : mkdir -p /opt/stack/data/glance/images | |
| 18:07:51 | artom | 2020-01-16 14:30:39.885 | + lib/glance:init_glance:276 : rm -rf /opt/stack/data/glance/images | |
| 18:07:51 | artom | 2020-01-16 14:30:39.883 | + ./stack.sh:main:1160 : init_glance | |
| 18:07:51 | artom | 2020-01-16 14:30:39.880 | + ./stack.sh:echo_summary:458 : echo -e Configuring Glance | |
| 18:07:53 | artom | 2020-01-16 14:30:39.893 | + lib/glance:init_glance:280 : recreate_database glance | |
| 18:08:51 | efried | gdi, why do I suck at logstash. Shouldn't it be as simple as searching for message:"mysql: command not found" ?? | |
| 18:11:44 | efried | ...unless logstash is also trying to read gzipped files and getting garbage. | |
| 18:12:07 | artom | Hrmm, hol'up | |
| 18:12:09 | artom | Working: is_service_enabled g-reg | |
| 18:12:21 | artom | Broken: is_glance_enabled | |
| 18:12:40 | artom | Which looks for g- in ENABLED_SERVICES, so g-api matches | |
| 18:13:59 | sean-k-mooney | artom: that is not somthign that would be changeing per fun but it would be different if patches had not been rebased i guess | |
| 18:15:17 | melwitt | do you think we would expect to run mysql commands on the subnode? that one is compute only and shouldn't have a database on it anyway | |
| 18:15:18 | efried | artom: https://review.opendev.org/#/c/702707/ | |
| 18:15:40 | artom | efried, bingo | |
| 18:16:06 | artom | That eventually triggers it to attempt to recreate the glance database on the subnode | |
| 18:16:15 | artom | Which calls the missing mysql binary | |
| 18:16:36 | dansmith | melwitt: that may be true for nova, but not all the services | |
| 18:16:36 | artom | I gotta run to a quick apt | |
| 18:16:56 | melwitt | dansmith: ack | |
| 18:17:12 | dansmith | and of course "mysql -h primary" could be some use case for sequencing (just making that up) | |