| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-16 | |||
| 17:50:27 | artom | Don't need to | zcat | less | |
| 17:50:30 | artom | Straight up | zless | |
| 17:50:44 | sean-k-mooney | artom: sure but im actully sendign it to lnav | |
| 17:50:56 | sean-k-mooney | and you use to be able to send it to firefox | |
| 17:51:08 | sean-k-mooney | but they droped readign from standar in at some point | |
| 17:51:33 | sean-k-mooney | but good to know zless is a thing. zgrep is also a thing | |
| 17:51:58 | dansmith | efried: that patch doesn't change what can be raised on error, so not an api change needing a microversion, if that's what you're asking for | |
| 17:52:28 | dansmith | efried: it adds a scan of all cells on delete, which the user doesn't know about anyway | |
| 17:52:57 | efried | artom: so far 5/5 on the ones I've looked at, fails from the last 2h or so. | |
| 17:54:22 | efried | dansmith: okay. afaict it makes it so HostMappingNotFound can no longer be raised when removing host from agg. That's kind of a behavior change, so I wasn't really sure. | |
| 17:54:23 | artom | efried, so, mysql-client should be getting installed | |
| 17:54:35 | efried | artom: I don't disagree. Why isn't it? | |
| 17:54:37 | artom | And it isn't, afaict | |
| 17:54:43 | artom | efried, yeah, no idea what changed yet | |
| 17:54:59 | efried | okay. Maybe -infra knows? | |
| 17:55:22 | dansmith | efried: it's the same error handler as ComputeHostNotFound in the api code though so it doesn't change anything the user can see | |
| 17:56:03 | efried | right, what I mean is that with a certain set of conditions, previously there would *be* an error, but now there would *not* be. | |
| 17:56:17 | efried | ...with those same conditions. | |
| 17:56:23 | efried | I guess that's kind of the definition of a bug fix though | |
| 17:56:30 | efried | so I'm going to +2. | |
| 17:56:34 | dansmith | the user doesn't know about those conditions | |
| 17:56:56 | dansmith | it'd be like saying we failed on mysql 4.1 and succed on 4.2 so that's user-visible | |
| 17:56:56 | efried | Okay, I must have misinterpreted that based on the bug description. | |
| 17:57:04 | efried | okay. | |
| 17:58:38 | artom | efried, what was the last time that job passed? I want to compare logs, see if it installed the mysqlclient | |
| 17:59:14 | efried | I'll look for one... | |
| 17:59:42 | artom | Hrmm, so it installs it on the *main* node | |
| 18:00:03 | efried | artom: here's one from ~24h ago, not sure if that's the most recent https://review.opendev.org/#/c/696745/ | |
| 18:01:04 | artom | efried, that works, thanks | |
| 18:01:19 | artom | Ah, so, on the main node, because we install the server, that pulls in the client | |
| 18:01:59 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Re-proposes multiple vGPU types in libvirt https://review.opendev.org/702943 | |
| 18:06:27 | efried | artom: assume this has nothing to do with cirros images http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012024.html | |
| 18:07:21 | artom | efried, don't think so - looks like it has something to do with whether we configure glance or not | |
| 18:07:50 | artom | The broken one runs through this: | |
| 18:07:51 | artom | 2020-01-16 14:30:39.880 | + ./stack.sh:echo_summary:458 : echo -e Configuring Glance | |
| 18:07:51 | artom | 2020-01-16 14:30:39.883 | + ./stack.sh:main:1160 : init_glance | |
| 18:07:51 | artom | 2020-01-16 14:30:39.885 | + lib/glance:init_glance:276 : rm -rf /opt/stack/data/glance/images | |
| 18:07:51 | artom | 2020-01-16 14:30:39.889 | + lib/glance:init_glance:277 : mkdir -p /opt/stack/data/glance/images | |
| 18:07:53 | artom | 2020-01-16 14:30:39.893 | + lib/glance:init_glance:280 : recreate_database glance | |
| 18:08:51 | efried | gdi, why do I suck at logstash. Shouldn't it be as simple as searching for message:"mysql: command not found" ?? | |
| 18:11:44 | efried | ...unless logstash is also trying to read gzipped files and getting garbage. | |
| 18:12:07 | artom | Hrmm, hol'up | |
| 18:12:09 | artom | Working: is_service_enabled g-reg | |
| 18:12:21 | artom | Broken: is_glance_enabled | |
| 18:12:40 | artom | Which looks for g- in ENABLED_SERVICES, so g-api matches | |
| 18:13:59 | sean-k-mooney | artom: that is not somthign that would be changeing per fun but it would be different if patches had not been rebased i guess | |
| 18:15:17 | melwitt | do you think we would expect to run mysql commands on the subnode? that one is compute only and shouldn't have a database on it anyway | |
| 18:15:18 | efried | artom: https://review.opendev.org/#/c/702707/ | |
| 18:15:40 | artom | efried, bingo | |
| 18:16:06 | artom | That eventually triggers it to attempt to recreate the glance database on the subnode | |
| 18:16:15 | artom | Which calls the missing mysql binary | |
| 18:16:36 | artom | I gotta run to a quick apt | |
| 18:16:36 | dansmith | melwitt: that may be true for nova, but not all the services | |
| 18:16:56 | melwitt | dansmith: ack | |
| 18:17:12 | dansmith | and of course "mysql -h primary" could be some use case for sequencing (just making that up) | |
| 18:17:14 | efried | artom: will this make it okay? https://review.opendev.org/#/c/702709/1 | |
| 18:17:47 | sean-k-mooney | dansmith: while you are right i dont think cinder cyborg or neutron do direct db acess form the compute nodes eitehr | |
| 18:18:18 | dansmith | sean-k-mooney: apparently glance is the service in question here? | |
| 18:18:38 | sean-k-mooney | ya | |
| 18:18:39 | dansmith | presumably other projects do or could use multinode for things | |
| 18:18:46 | dansmith | (or should) | |
| 18:19:04 | sean-k-mooney | glance should normally not be on the compute node but we can enable it there if we want | |
| 18:19:27 | sean-k-mooney | maybe its beign used for the multi image store testing or something like that | |
| 18:24:03 | efried | artom: I guess not, since they both merged at essentially the same time. | |
| 18:29:10 | openstackgerrit | Merged openstack/nova stable/pike: lxc: make use of filter python3 compatible https://review.opendev.org/676502 | |
| 18:33:14 | efried | https://bugs.launchpad.net/nova/+bug/1860021 | |
| 18:33:14 | openstack | Launchpad bug 1860021 in OpenStack Compute (nova) "nova-live-migration fails 100% with "mysql: command not found" on subnode" [Undecided,New] | |
| 18:34:19 | efried | Correct me if I'm wrong: we didn't need init_glance to be happening on the subnode, cause it wasn't before? | |
| 18:35:02 | sean-k-mooney | efried: well we dont need a second copy of glance running on the subnode no | |
| 18:35:23 | sean-k-mooney | unless we are specificly trying to test image that are in multiple data stores | |
| 18:35:30 | melwitt | is it only our multinode job that's failing? I would expect all would be | |
| 18:35:53 | sean-k-mooney | i think this was only nova livemigation? | |
| 18:35:54 | efried | melwitt: many patches have failed with just n-l-m in the red. A few have failed with other things, but I haven't looked into them. | |
| 18:36:06 | melwitt | meaning I would think all multinode jobs regardless of project would fail from this | |
| 18:36:09 | efried | (other than nova-next with 'multiple possible networks' as noted, fix in the pipe) | |
| 18:36:36 | sean-k-mooney | melwitt: it could be that nova live migration is a legacy job | |
| 18:36:41 | melwitt | hm, that is surprising. so that means there's some other boilerplate in other jobs that we don't have in nova-live-migration if we're the only ones failing | |
| 18:36:56 | efried | perhaps somehow we're installing mysql client on the other jobs | |
| 18:37:02 | melwitt | yeah, that's what I'm wondering | |
| 18:37:05 | sean-k-mooney | and our other multinode jobs may be zuul v3 so the glance service might not be enabled on the sub nodes | |
| 18:37:21 | melwitt | sean-k-mooney: in what way is it a legacy job? I remember moving it over to zuul v3 | |
| 18:37:39 | efried | the right thing would be to figure out how to make the subnode not init_glance; but I would be happy enough with a solution that got mysql client installed there so we could go back to the way we were. | |
| 18:37:46 | sean-k-mooney | oh maybe im thinking of nova-next | |
| 18:37:47 | efried | n-l-m is legacy | |
| 18:37:49 | efried | isn't it? | |
| 18:37:52 | melwitt | I didn't think so | |
| 18:38:01 | melwitt | let me look it up because maybe I'm wrong | |
| 18:38:06 | sean-k-mooney | well there is one way to find out | |
| 18:38:17 | efried | if it weren't, wouldn't we be not having the gz problem? | |
| 18:38:25 | sean-k-mooney | https://github.com/openstack/nova/tree/stable/stein/playbooks/legacy | |
| 18:38:41 | melwitt | oh yeah, I see https://review.opendev.org/#/c/693364 | |
| 18:38:58 | melwitt | so moving it to the .zuul.yaml does not make it zuul v3 apparently. I didn't know that | |
| 18:39:31 | sean-k-mooney | so it has playbooks in the legacy folder and https://github.com/openstack/nova/blob/stable/stein/.zuul.yaml#L114-L122 still runs them so ya | |
| 18:39:51 | melwitt | ok. I had previously thought that migrating jobs in tree made them zuul v3 but that was a wrong thought | |
| 18:39:58 | melwitt | sorry about that | |
| 18:40:04 | sean-k-mooney | melwitt: yep matt start to move it but did not have time to finish it | |
| 18:40:51 | sean-k-mooney | also you gerrit foo is better then mine i spent 20 minuts trying to find that last week an gave up | |
| 18:41:15 | sean-k-mooney | maybe i filtered out abandoned chagnes | |
| 18:41:20 | melwitt | lol. I just did "message:nova-live" nothing fancy | |
| 18:41:31 | sean-k-mooney | message:? | |