| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-16 | |||
| 18:11:44 | efried | ...unless logstash is also trying to read gzipped files and getting garbage. | |
| 18:12:07 | artom | Hrmm, hol'up | |
| 18:12:09 | artom | Working: is_service_enabled g-reg | |
| 18:12:21 | artom | Broken: is_glance_enabled | |
| 18:12:40 | artom | Which looks for g- in ENABLED_SERVICES, so g-api matches | |
| 18:13:59 | sean-k-mooney | artom: that is not somthign that would be changeing per fun but it would be different if patches had not been rebased i guess | |
| 18:15:17 | melwitt | do you think we would expect to run mysql commands on the subnode? that one is compute only and shouldn't have a database on it anyway | |
| 18:15:18 | efried | artom: https://review.opendev.org/#/c/702707/ | |
| 18:15:40 | artom | efried, bingo | |
| 18:16:06 | artom | That eventually triggers it to attempt to recreate the glance database on the subnode | |
| 18:16:15 | artom | Which calls the missing mysql binary | |
| 18:16:36 | artom | I gotta run to a quick apt | |
| 18:16:36 | dansmith | melwitt: that may be true for nova, but not all the services | |
| 18:16:56 | melwitt | dansmith: ack | |
| 18:17:12 | dansmith | and of course "mysql -h primary" could be some use case for sequencing (just making that up) | |
| 18:17:14 | efried | artom: will this make it okay? https://review.opendev.org/#/c/702709/1 | |
| 18:17:47 | sean-k-mooney | dansmith: while you are right i dont think cinder cyborg or neutron do direct db acess form the compute nodes eitehr | |
| 18:18:18 | dansmith | sean-k-mooney: apparently glance is the service in question here? | |
| 18:18:38 | sean-k-mooney | ya | |
| 18:18:39 | dansmith | presumably other projects do or could use multinode for things | |
| 18:18:46 | dansmith | (or should) | |
| 18:19:04 | sean-k-mooney | glance should normally not be on the compute node but we can enable it there if we want | |
| 18:19:27 | sean-k-mooney | maybe its beign used for the multi image store testing or something like that | |
| 18:24:03 | efried | artom: I guess not, since they both merged at essentially the same time. | |
| 18:29:10 | openstackgerrit | Merged openstack/nova stable/pike: lxc: make use of filter python3 compatible https://review.opendev.org/676502 | |
| 18:33:14 | efried | https://bugs.launchpad.net/nova/+bug/1860021 | |
| 18:33:14 | openstack | Launchpad bug 1860021 in OpenStack Compute (nova) "nova-live-migration fails 100% with "mysql: command not found" on subnode" [Undecided,New] | |
| 18:34:19 | efried | Correct me if I'm wrong: we didn't need init_glance to be happening on the subnode, cause it wasn't before? | |
| 18:35:02 | sean-k-mooney | efried: well we dont need a second copy of glance running on the subnode no | |
| 18:35:23 | sean-k-mooney | unless we are specificly trying to test image that are in multiple data stores | |
| 18:35:30 | melwitt | is it only our multinode job that's failing? I would expect all would be | |
| 18:35:53 | sean-k-mooney | i think this was only nova livemigation? | |
| 18:35:54 | efried | melwitt: many patches have failed with just n-l-m in the red. A few have failed with other things, but I haven't looked into them. | |
| 18:36:06 | melwitt | meaning I would think all multinode jobs regardless of project would fail from this | |
| 18:36:09 | efried | (other than nova-next with 'multiple possible networks' as noted, fix in the pipe) | |
| 18:36:36 | sean-k-mooney | melwitt: it could be that nova live migration is a legacy job | |
| 18:36:41 | melwitt | hm, that is surprising. so that means there's some other boilerplate in other jobs that we don't have in nova-live-migration if we're the only ones failing | |
| 18:36:56 | efried | perhaps somehow we're installing mysql client on the other jobs | |
| 18:37:02 | melwitt | yeah, that's what I'm wondering | |
| 18:37:05 | sean-k-mooney | and our other multinode jobs may be zuul v3 so the glance service might not be enabled on the sub nodes | |
| 18:37:21 | melwitt | sean-k-mooney: in what way is it a legacy job? I remember moving it over to zuul v3 | |
| 18:37:39 | efried | the right thing would be to figure out how to make the subnode not init_glance; but I would be happy enough with a solution that got mysql client installed there so we could go back to the way we were. | |
| 18:37:46 | sean-k-mooney | oh maybe im thinking of nova-next | |
| 18:37:47 | efried | n-l-m is legacy | |
| 18:37:49 | efried | isn't it? | |
| 18:37:52 | melwitt | I didn't think so | |
| 18:38:01 | melwitt | let me look it up because maybe I'm wrong | |
| 18:38:06 | sean-k-mooney | well there is one way to find out | |
| 18:38:17 | efried | if it weren't, wouldn't we be not having the gz problem? | |
| 18:38:25 | sean-k-mooney | https://github.com/openstack/nova/tree/stable/stein/playbooks/legacy | |
| 18:38:41 | melwitt | oh yeah, I see https://review.opendev.org/#/c/693364 | |
| 18:38:58 | melwitt | so moving it to the .zuul.yaml does not make it zuul v3 apparently. I didn't know that | |
| 18:39:31 | sean-k-mooney | so it has playbooks in the legacy folder and https://github.com/openstack/nova/blob/stable/stein/.zuul.yaml#L114-L122 still runs them so ya | |
| 18:39:51 | melwitt | ok. I had previously thought that migrating jobs in tree made them zuul v3 but that was a wrong thought | |
| 18:39:58 | melwitt | sorry about that | |
| 18:40:04 | sean-k-mooney | melwitt: yep matt start to move it but did not have time to finish it | |
| 18:40:51 | sean-k-mooney | also you gerrit foo is better then mine i spent 20 minuts trying to find that last week an gave up | |
| 18:41:15 | sean-k-mooney | maybe i filtered out abandoned chagnes | |
| 18:41:20 | melwitt | lol. I just did "message:nova-live" nothing fancy | |
| 18:41:31 | sean-k-mooney | message:? | |
| 18:41:40 | melwitt | yeah means look in commit message | |
| 18:41:41 | sean-k-mooney | oh that checks the commt message? | |
| 18:41:51 | melwitt | "comment:" looks in comments | |
| 18:41:54 | sean-k-mooney | see you gerrit foo is better then mine | |
| 18:41:59 | melwitt | gerrit search is pretty cool | |
| 18:43:28 | sean-k-mooney | so ya i would guess that this has somethign to do with devstack-gate | |
| 18:43:47 | sean-k-mooney | the legacy jobs are going to be using devstack-gate and the non legacy ones will not | |
| 18:44:05 | melwitt | I'm trying to look at tempest-multinode-full-py3 to see if it's legacy but I think you could figure that out faster than me | |
| 18:44:11 | melwitt | that's an example from neutron | |
| 18:44:16 | melwitt | which is passing | |
| 18:44:50 | sean-k-mooney | i think that is not a legacy job | |
| 18:45:09 | sean-k-mooney | the legacy ones usually specify run: playbook | |
| 18:45:37 | melwitt | I see, ok. so that's a data point in favor of it being related to legacy-ness | |
| 18:46:17 | melwitt | I don't know what it is about the non-legacy stuff that can handle the mysql commands. do they have the client available or something more involved | |
| 18:47:09 | sean-k-mooney | is it related to this? https://github.com/openstack/devstack-gate/commit/183a58eab185c2e6f48afb4d99417d49e88c0f5a | |
| 18:47:28 | sean-k-mooney | specificlly https://github.com/openstack/devstack-gate/commit/183a58eab185c2e6f48afb4d99417d49e88c0f5a#diff-987768d02fbf43ebb74ad9a359222903R27 | |
| 18:48:13 | sean-k-mooney | matt enabled glance api on the subnode | |
| 18:48:33 | artom | efried, back - no, it won't, because is_glance_enabled checks for 'g-' in the ENABLED_SERVICES | |
| 18:48:34 | sean-k-mooney | that was also like 27 days ago | |
| 18:48:37 | artom | So g-api triggers it too | |
| 18:48:57 | efried | Looking for some way to redefine is_glance_enabled as a stopgap | |
| 18:49:14 | artom | We might want to consult with the devstack folks | |
| 18:49:31 | efried | wherezat? -infra? -qa? | |
| 18:49:37 | sean-k-mooney | qa | |
| 18:49:59 | sean-k-mooney | artom: are you sure its not https://github.com/openstack/devstack-gate/commit/183a58eab185c2e6f48afb4d99417d49e88c0f5a#diff-987768d02fbf43ebb74ad9a359222903R27 | |
| 18:50:27 | artom | sean-k-mooney, I'm never sure of anything, but I don't think so | |
| 18:50:54 | sean-k-mooney | i guess that is GRENADE_SUBNODE_MASTER | |
| 18:50:55 | efried | could be that, and then the glance-enabled check exposed it. | |
| 18:51:12 | artom | Wait, what did that change? | |
| 18:51:18 | artom | Added g-api to the subnode? | |
| 18:52:54 | sean-k-mooney | yes | |
| 18:53:08 | sean-k-mooney | although maybe only in the grenade jobs | |
| 19:05:43 | openstackgerrit | Eric Fried proposed openstack/nova master: Test fix for bug 1860021 https://review.opendev.org/702961 | |
| 19:05:43 | openstack | bug 1860021 in devstack "nova-live-migration fails 100% with "mysql: command not found" on subnode" [Undecided,In progress] https://launchpad.net/bugs/1860021 - Assigned to Radosław Piliszek (yoctozepto) | |
| 19:05:53 | efried | artom: ^ | |
| 19:08:43 | artom | efried, thanks! | |
| 19:09:00 | efried | artom: thanks for helping triage this. | |
| 19:09:10 | artom | \o/ I'm useful! | |
| 19:22:01 | openstackgerrit | Merged openstack/nova stable/rocky: Cache security group driver https://review.opendev.org/697505 | |
| 19:57:01 | openstackgerrit | Merged openstack/nova stable/rocky: Block deleting compute services with in-progress migrations https://review.opendev.org/698113 | |