| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-16 | |||
| 17:42:06 | efried | clearly fungi has had to explain this before :P | |
| 17:42:20 | artom | fungi, to be honest I didn't grok most of that. My takeaway is: layers do deep even onions are like "whoa dude" | |
| 17:42:23 | sean-k-mooney | yes | |
| 17:42:29 | artom | *so deep | |
| 17:42:54 | sean-k-mooney | but thanks for the explaination again. | |
| 17:43:53 | fungi | we fixed the obvious cases we knew about in abstract jobs/roles but there were some in devstack which had to get fixed too | |
| 17:44:16 | sean-k-mooney | fungi: funnialy enogh the last time i tried deploying zuul at home geting the logs to render properly in the bower when loadbalnace via nginx in k8s was actully one of the hardest parts | |
| 17:44:21 | fungi | and there are also some project-specific jobs which are also precompressing logs from things which will need to be adjusted to stop compressing within the job | |
| 17:44:42 | fungi | which should be updated as we encounter them | |
| 17:45:58 | efried | dansmith: Would you please look at https://review.opendev.org/#/c/696392/ ? I'm not clear whether this counts as an API change... | |
| 17:47:45 | efried | artom: so are you investigating the failure amid all of that? | |
| 17:48:01 | artom | efried, which failure? The missing MySQL? | |
| 17:48:20 | efried | yeah | |
| 17:48:33 | artom | Heh, I'm being volunteered, aren't I? ;) | |
| 17:48:38 | efried | whatever's causing nova-live-migration to fail | |
| 17:48:38 | artom | I'll take a look :) | |
| 17:48:49 | efried | thanks artom. Rather you than me. | |
| 17:49:28 | efried | I'll go check the other instances of n-l-m fail and see if they're the same. | |
| 17:50:18 | artom | sean-k-mooney, zless is a thing, btw :) | |
| 17:50:27 | artom | Don't need to | zcat | less | |
| 17:50:30 | artom | Straight up | zless | |
| 17:50:44 | sean-k-mooney | artom: sure but im actully sendign it to lnav | |
| 17:50:56 | sean-k-mooney | and you use to be able to send it to firefox | |
| 17:51:08 | sean-k-mooney | but they droped readign from standar in at some point | |
| 17:51:33 | sean-k-mooney | but good to know zless is a thing. zgrep is also a thing | |
| 17:51:58 | dansmith | efried: that patch doesn't change what can be raised on error, so not an api change needing a microversion, if that's what you're asking for | |
| 17:52:28 | dansmith | efried: it adds a scan of all cells on delete, which the user doesn't know about anyway | |
| 17:52:57 | efried | artom: so far 5/5 on the ones I've looked at, fails from the last 2h or so. | |
| 17:54:22 | efried | dansmith: okay. afaict it makes it so HostMappingNotFound can no longer be raised when removing host from agg. That's kind of a behavior change, so I wasn't really sure. | |
| 17:54:23 | artom | efried, so, mysql-client should be getting installed | |
| 17:54:35 | efried | artom: I don't disagree. Why isn't it? | |
| 17:54:37 | artom | And it isn't, afaict | |
| 17:54:43 | artom | efried, yeah, no idea what changed yet | |
| 17:54:59 | efried | okay. Maybe -infra knows? | |
| 17:55:22 | dansmith | efried: it's the same error handler as ComputeHostNotFound in the api code though so it doesn't change anything the user can see | |
| 17:56:03 | efried | right, what I mean is that with a certain set of conditions, previously there would *be* an error, but now there would *not* be. | |
| 17:56:17 | efried | ...with those same conditions. | |
| 17:56:23 | efried | I guess that's kind of the definition of a bug fix though | |
| 17:56:30 | efried | so I'm going to +2. | |
| 17:56:34 | dansmith | the user doesn't know about those conditions | |
| 17:56:56 | efried | Okay, I must have misinterpreted that based on the bug description. | |
| 17:56:56 | dansmith | it'd be like saying we failed on mysql 4.1 and succed on 4.2 so that's user-visible | |
| 17:57:04 | efried | okay. | |
| 17:58:38 | artom | efried, what was the last time that job passed? I want to compare logs, see if it installed the mysqlclient | |
| 17:59:14 | efried | I'll look for one... | |
| 17:59:42 | artom | Hrmm, so it installs it on the *main* node | |
| 18:00:03 | efried | artom: here's one from ~24h ago, not sure if that's the most recent https://review.opendev.org/#/c/696745/ | |
| 18:01:04 | artom | efried, that works, thanks | |
| 18:01:19 | artom | Ah, so, on the main node, because we install the server, that pulls in the client | |
| 18:01:59 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Re-proposes multiple vGPU types in libvirt https://review.opendev.org/702943 | |
| 18:06:27 | efried | artom: assume this has nothing to do with cirros images http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012024.html | |
| 18:07:21 | artom | efried, don't think so - looks like it has something to do with whether we configure glance or not | |
| 18:07:50 | artom | The broken one runs through this: | |
| 18:07:51 | artom | 2020-01-16 14:30:39.889 | + lib/glance:init_glance:277 : mkdir -p /opt/stack/data/glance/images | |
| 18:07:51 | artom | 2020-01-16 14:30:39.885 | + lib/glance:init_glance:276 : rm -rf /opt/stack/data/glance/images | |
| 18:07:51 | artom | 2020-01-16 14:30:39.883 | + ./stack.sh:main:1160 : init_glance | |
| 18:07:51 | artom | 2020-01-16 14:30:39.880 | + ./stack.sh:echo_summary:458 : echo -e Configuring Glance | |
| 18:07:53 | artom | 2020-01-16 14:30:39.893 | + lib/glance:init_glance:280 : recreate_database glance | |
| 18:08:51 | efried | gdi, why do I suck at logstash. Shouldn't it be as simple as searching for message:"mysql: command not found" ?? | |
| 18:11:44 | efried | ...unless logstash is also trying to read gzipped files and getting garbage. | |
| 18:12:07 | artom | Hrmm, hol'up | |
| 18:12:09 | artom | Working: is_service_enabled g-reg | |
| 18:12:21 | artom | Broken: is_glance_enabled | |
| 18:12:40 | artom | Which looks for g- in ENABLED_SERVICES, so g-api matches | |
| 18:13:59 | sean-k-mooney | artom: that is not somthign that would be changeing per fun but it would be different if patches had not been rebased i guess | |
| 18:15:17 | melwitt | do you think we would expect to run mysql commands on the subnode? that one is compute only and shouldn't have a database on it anyway | |
| 18:15:18 | efried | artom: https://review.opendev.org/#/c/702707/ | |
| 18:15:40 | artom | efried, bingo | |
| 18:16:06 | artom | That eventually triggers it to attempt to recreate the glance database on the subnode | |
| 18:16:15 | artom | Which calls the missing mysql binary | |
| 18:16:36 | dansmith | melwitt: that may be true for nova, but not all the services | |
| 18:16:36 | artom | I gotta run to a quick apt | |
| 18:16:56 | melwitt | dansmith: ack | |
| 18:17:12 | dansmith | and of course "mysql -h primary" could be some use case for sequencing (just making that up) | |
| 18:17:14 | efried | artom: will this make it okay? https://review.opendev.org/#/c/702709/1 | |
| 18:17:47 | sean-k-mooney | dansmith: while you are right i dont think cinder cyborg or neutron do direct db acess form the compute nodes eitehr | |
| 18:18:18 | dansmith | sean-k-mooney: apparently glance is the service in question here? | |
| 18:18:38 | sean-k-mooney | ya | |
| 18:18:39 | dansmith | presumably other projects do or could use multinode for things | |
| 18:18:46 | dansmith | (or should) | |
| 18:19:04 | sean-k-mooney | glance should normally not be on the compute node but we can enable it there if we want | |
| 18:19:27 | sean-k-mooney | maybe its beign used for the multi image store testing or something like that | |
| 18:24:03 | efried | artom: I guess not, since they both merged at essentially the same time. | |
| 18:29:10 | openstackgerrit | Merged openstack/nova stable/pike: lxc: make use of filter python3 compatible https://review.opendev.org/676502 | |
| 18:33:14 | openstack | Launchpad bug 1860021 in OpenStack Compute (nova) "nova-live-migration fails 100% with "mysql: command not found" on subnode" [Undecided,New] | |
| 18:33:14 | efried | https://bugs.launchpad.net/nova/+bug/1860021 | |
| 18:34:19 | efried | Correct me if I'm wrong: we didn't need init_glance to be happening on the subnode, cause it wasn't before? | |
| 18:35:02 | sean-k-mooney | efried: well we dont need a second copy of glance running on the subnode no | |
| 18:35:23 | sean-k-mooney | unless we are specificly trying to test image that are in multiple data stores | |
| 18:35:30 | melwitt | is it only our multinode job that's failing? I would expect all would be | |
| 18:35:53 | sean-k-mooney | i think this was only nova livemigation? | |
| 18:35:54 | efried | melwitt: many patches have failed with just n-l-m in the red. A few have failed with other things, but I haven't looked into them. | |
| 18:36:06 | melwitt | meaning I would think all multinode jobs regardless of project would fail from this | |
| 18:36:09 | efried | (other than nova-next with 'multiple possible networks' as noted, fix in the pipe) | |
| 18:36:36 | sean-k-mooney | melwitt: it could be that nova live migration is a legacy job | |
| 18:36:41 | melwitt | hm, that is surprising. so that means there's some other boilerplate in other jobs that we don't have in nova-live-migration if we're the only ones failing | |
| 18:36:56 | efried | perhaps somehow we're installing mysql client on the other jobs | |
| 18:37:02 | melwitt | yeah, that's what I'm wondering | |
| 18:37:05 | sean-k-mooney | and our other multinode jobs may be zuul v3 so the glance service might not be enabled on the sub nodes | |
| 18:37:21 | melwitt | sean-k-mooney: in what way is it a legacy job? I remember moving it over to zuul v3 | |