| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-26 | |||
| 11:16:33 | mriedem | at least it's in gerrit | |
| 11:16:45 | sdague | yeh, well there is that :) | |
| 11:16:51 | sdague | then you get to see who signed off on things | |
| 11:17:19 | sdague | all those bzr changes that were imported usually have no rhyme or reason why they were merged | |
| 11:17:45 | mriedem | seemed like a good idea at the time... | |
| 11:18:00 | mriedem | this is why the ptl talk had the part about 'we're not historians' | |
| 11:19:58 | sdague | Chesterton's Fence | |
| 11:20:30 | sdague | early morning is a good time to look through old commits and figure out why would folks ever have done a thing :) | |
| 11:27:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused migrate_data kwarg from virt driver destroy() method https://review.openstack.org/452958 | |
| 11:50:41 | sdague | that feeling when you pdb trace a failing unit test, and figure out the issue is tox env doesn't include the required version of oslo.middleware | |
| 11:54:51 | mriedem | -r | |
| 12:07:35 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Use plain routes list for server ips endpoint https://review.openstack.org/468334 | |
| 12:08:20 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Use plain routes list for instance actions endpoint https://review.openstack.org/468338 | |
| 12:08:46 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Use plain routes list for os-fping endpoint https://review.openstack.org/468349 | |
| 12:11:47 | sdague | mriedem: amirth pointed out some interesting trove fail about a failing to schedule instance which is stuck in scheduling and not put into ERROR | |
| 12:11:49 | sdague | http://logs.openstack.org/47/467447/14/check/gate-trove-scenario-dsvm-mariadb-multi-ubuntu-xenial-nv/6efec50/logs/screen-n-cond.txt.gz#_May_26_03_24_31_161873 | |
| 12:12:06 | amrith | sdague ... | |
| 12:12:12 | amrith | http://paste.openstack.org/show/610724/ | |
| 12:12:14 | sdague | it looks like when conductor swings around to error it it can't find the instance | |
| 12:12:44 | amrith | and trove hangs on the scheduling | |
| 12:14:27 | mriedem | so it's trying to bury the instance in cell0, and send a notification but while it's doing that it's trying to save the instance (probably updating vm/task state) and that's loading the instance up again for some reason which is making it blow up... | |
| 12:14:57 | mriedem | this is all changes on master for trove? | |
| 12:15:00 | mriedem | or random? | |
| 12:15:14 | sdague | amrith: ^^?? | |
| 12:15:54 | amrith | sdague i dont understand the question | |
| 12:16:06 | mriedem | amrith: is this random or 100% CI fail? | |
| 12:16:18 | amrith | this is 100% repeatable in https://review.openstack.org/#/c/467447/ | |
| 12:16:23 | amrith | look at the multi jobs | |
| 12:16:29 | mriedem | so just that change? | |
| 12:16:29 | amrith | I repro it on my laptop | |
| 12:16:34 | amrith | just that one change | |
| 12:16:42 | amrith | this is the change to get around the nova pass through API's | |
| 12:16:55 | amrith | it is the first time we're seeing 8.0.0 | |
| 12:17:06 | amrith | and whatever else nova backend brings to the party | |
| 12:17:24 | amrith | to repro it just recheck that change https://review.openstack.org/#/c/467447/ | |
| 12:17:29 | amrith | and watch the multi job | |
| 12:17:50 | amrith | it'll get to the wait_for_non_affinity_repl_to_fail thing | |
| 12:17:51 | amrith | and hang | |
| 12:18:00 | amrith | because nova lists the instance (in nova list) as scheduling | |
| 12:18:04 | amrith | while it has actually failed | |
| 12:18:10 | sdague | mriedem: it's not just that change | |
| 12:18:21 | sdague | the entire trove gate looks like this is busted | |
| 12:18:27 | sdague | it's the multinode for mariadb | |
| 12:18:29 | amrith | in my laptop now ... http://paste.openstack.org/show/610724/ | |
| 12:18:51 | amrith | you can see that scheduler gave up, conductor errored but nova list says scheduling | |
| 12:18:52 | amrith | and yes to what sdague said | |
| 12:18:54 | amrith | currnetly the trove gate is borked | |
| 12:18:54 | mriedem | so, how does the trove CI setup cells v2? is it using devstack? | |
| 12:18:55 | sdague | https://review.openstack.org/#/c/468102/ is a change to CONTRIBUTING.rst that has the same pattern | |
| 12:19:13 | mriedem | did this just start? | |
| 12:19:18 | amrith | yes | |
| 12:19:31 | amrith | I thought it was related to the 8.0.0 stuff | |
| 12:19:34 | amrith | but not true | |
| 12:19:59 | amrith | this may have been the last successful run https://review.openstack.org/#/c/457223/ | |
| 12:20:24 | amrith | certainly didnt happen > 4 days ago | |
| 12:20:42 | mriedem | http://logs.openstack.org/47/467447/14/check/gate-trove-scenario-dsvm-mariadb-multi-ubuntu-xenial-nv/6efec50/logs/screen-n-sch.txt.gz#_May_26_03_24_31_029544 | |
| 12:20:52 | mriedem | the anti-affinity filter kicked it out, which is why you're getting NoValidHost, | |
| 12:20:57 | mriedem | also, the scheduler is only seeing 1 host | |
| 12:21:02 | sdague | mriedem: right, that's expected | |
| 12:21:06 | mriedem | are you using devstack? | |
| 12:21:08 | sdague | that's actually the point of the test | |
| 12:21:28 | amrith | mriedem it is devstack driven through trovestack (formerly redstack) | |
| 12:21:49 | amrith | i can send you my local confs if that will help | |
| 12:22:01 | amrith | but you can see the confs in the CI as well | |
| 12:22:02 | sdague | the real question is actually why nova-conductor fails to actually error the the node | |
| 12:22:16 | sdague | not that it fails to schedule | |
| 12:22:28 | mriedem | yes i realize | |
| 12:22:33 | sdague | ok, cool | |
| 12:24:39 | mriedem | odd that i can't get hits on this in logstash | |
| 12:24:51 | sdague | is conductor indexed? | |
| 12:24:58 | sdague | or these jobs? | |
| 12:25:11 | mriedem | conductor is certainly indexed, | |
| 12:25:21 | mriedem | but i didn't think it was per job | |
| 12:25:24 | mriedem | it's per file | |
| 12:26:26 | mriedem | i think i see the problem, maybe | |
| 12:26:28 | mriedem | patch incoming | |
| 12:28:48 | sdague | cool | |
| 12:28:51 | sdague | thanks | |
| 12:28:59 | mriedem | amrith: got a bug? | |
| 12:30:23 | sdague | mriedem: I don't think he does, and he just had to head off for holiday weekend | |
| 12:31:08 | mriedem | ok, i'll put up a trove patch with depends-on to test this out | |
| 12:31:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Use targeted context when burying instances in cell0 https://review.openstack.org/468401 | |
| 12:31:38 | mriedem | dansmith: ^ | |
| 12:32:38 | openstackgerrit | Sean Dague proposed openstack/python-novaclient master: Have python-novaclient support global_request_id https://review.openstack.org/468126 | |
| 12:33:36 | mriedem | aha https://bugs.launchpad.net/nova/+bug/1693438 | |
| 12:33:36 | openstack | Launchpad bug 1693438 in OpenStack Compute (nova) "error instances remain in "Build" status and can't delete it" [Undecided,New] - Assigned to Tetsuro Nakamura (tetsuro0907) | |
| 12:33:42 | mriedem | already reported yesterday by someone doing a good job at CD | |
| 12:34:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Use targeted context when burying instances in cell0 https://review.openstack.org/468401 | |
| 12:35:51 | sdague | mriedem: nice | |
| 12:39:47 | mriedem | amrith: fyi https://review.openstack.org/#/c/468402/ | |
| 12:40:45 | mriedem | and now i'm afraid i must get cleaned up | |
| 12:42:08 | sdague | mriedem: what was the dogpile cache issue with nova? | |
| 12:44:24 | amrith | mriedem your trove change will fail because of the nova client 8.0.0. if you'd like please dep my change (which aims to fix that) on your proposed fix | |
| 12:44:28 | amrith | sdague ^^ | |
| 12:44:47 | sdague | amrith: yeh, I can update that | |
| 13:00:39 | leakypipes | morning folks | |
| 13:02:56 | sdague | leakypipes: in reading through test logs of my failing patches, I saw a lot of sqla warnings are being emitted around the traits tests | |
| 13:03:03 | sdague | because of unicode mismatch | |
| 13:03:17 | cdent | sfinucan: leakypipes put his +2 on https://review.openstack.org/#/c/461494/ so it is safe for you to raise your +1 to +2 if you like. or anyone else if they are feeling like a quick win | |
| 13:03:43 | sfinucan | cdent: Yup, done | |