| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-26 | |||
| 11:05:53 | cdent | right, but the end state (scheduler being aware of what's on the computes) is effectively the same? | |
| 11:05:54 | mriedem | i've got https://review.openstack.org/#/c/453916/ to be smarter about that if you're not using the affinity filters | |
| 11:06:03 | mriedem | cdent: yes i believe that's the idea | |
| 11:06:29 | mriedem | i also want to kill the TypeAffinityFilter https://review.openstack.org/#/c/457812/ | |
| 11:06:47 | mriedem | because then i can change this stuff to not send full objects but just the ids | |
| 11:06:59 | mriedem | for https://blueprints.launchpad.net/nova/+spec/put-host-manager-instance-info-on-a-diet | |
| 11:07:10 | mriedem | if we get the right stuff in placement for affinity filtering before then, great | |
| 11:07:19 | sdague | mriedem: thanks for getting that spec landed yesterday | |
| 11:07:25 | mriedem | but right now we send a full InstanceList from all computes every minute to the single poor scheduler | |
| 11:07:29 | mriedem | sdague: np | |
| 11:07:45 | cdent | 453916 is saying "people not using the filter scheduler". if we kill the caching scheduler is there much left besides filter? (in terms of active users) | |
| 11:08:06 | mriedem | cdent: nope | |
| 11:08:18 | sdague | also, my new favorite git feature discovered this month | |
| 11:08:18 | mriedem | cdent: that could be further trimmed to the affinity filters, but it gets murky | |
| 11:08:23 | cdent | instances by reference (rather than object) sounds like a very good thing | |
| 11:08:26 | sdague | git commit -c/-C | |
| 11:08:28 | mriedem | cdent: but that gets messy with people that have customer filters | |
| 11:08:34 | sdague | which is steal the commit message from another commit | |
| 11:08:47 | mriedem | stealing is bad mkay | |
| 11:09:07 | mriedem | sdague: maybe you'd like to deprecate this filter https://review.openstack.org/#/c/457812/ | |
| 11:11:17 | sdague | I always like deprecating things | |
| 11:11:27 | mriedem | sdague: you're going to have functional api samples failures on this https://review.openstack.org/#/c/467998/ | |
| 11:11:35 | mriedem | because of the GET / tests | |
| 11:12:07 | sdague | mriedem: good call, I'll fix that as soon as I'm done with your review | |
| 11:13:19 | sdague | zomg, we can blame jogo for that one :) | |
| 11:13:31 | sdague | I was assuming it was code to support rax | |
| 11:14:29 | mriedem | yup i saw that too when i dug through writing that change | |
| 11:15:16 | sdague | https://review.openstack.org/#/c/7571 | |
| 11:15:30 | sdague | it honestly confuses me to see review ids that short | |
| 11:16:33 | mriedem | at least it's in gerrit | |
| 11:16:45 | sdague | yeh, well there is that :) | |
| 11:16:51 | sdague | then you get to see who signed off on things | |
| 11:17:19 | sdague | all those bzr changes that were imported usually have no rhyme or reason why they were merged | |
| 11:17:45 | mriedem | seemed like a good idea at the time... | |
| 11:18:00 | mriedem | this is why the ptl talk had the part about 'we're not historians' | |
| 11:19:58 | sdague | Chesterton's Fence | |
| 11:20:30 | sdague | early morning is a good time to look through old commits and figure out why would folks ever have done a thing :) | |
| 11:27:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused migrate_data kwarg from virt driver destroy() method https://review.openstack.org/452958 | |
| 11:50:41 | sdague | that feeling when you pdb trace a failing unit test, and figure out the issue is tox env doesn't include the required version of oslo.middleware | |
| 11:54:51 | mriedem | -r | |
| 12:07:35 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Use plain routes list for server ips endpoint https://review.openstack.org/468334 | |
| 12:08:20 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Use plain routes list for instance actions endpoint https://review.openstack.org/468338 | |
| 12:08:46 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Use plain routes list for os-fping endpoint https://review.openstack.org/468349 | |
| 12:11:47 | sdague | mriedem: amirth pointed out some interesting trove fail about a failing to schedule instance which is stuck in scheduling and not put into ERROR | |
| 12:11:49 | sdague | http://logs.openstack.org/47/467447/14/check/gate-trove-scenario-dsvm-mariadb-multi-ubuntu-xenial-nv/6efec50/logs/screen-n-cond.txt.gz#_May_26_03_24_31_161873 | |
| 12:12:06 | amrith | sdague ... | |
| 12:12:12 | amrith | http://paste.openstack.org/show/610724/ | |
| 12:12:14 | sdague | it looks like when conductor swings around to error it it can't find the instance | |
| 12:12:44 | amrith | and trove hangs on the scheduling | |
| 12:14:27 | mriedem | so it's trying to bury the instance in cell0, and send a notification but while it's doing that it's trying to save the instance (probably updating vm/task state) and that's loading the instance up again for some reason which is making it blow up... | |
| 12:14:57 | mriedem | this is all changes on master for trove? | |
| 12:15:00 | mriedem | or random? | |
| 12:15:14 | sdague | amrith: ^^?? | |
| 12:15:54 | amrith | sdague i dont understand the question | |
| 12:16:06 | mriedem | amrith: is this random or 100% CI fail? | |
| 12:16:18 | amrith | this is 100% repeatable in https://review.openstack.org/#/c/467447/ | |
| 12:16:23 | amrith | look at the multi jobs | |
| 12:16:29 | mriedem | so just that change? | |
| 12:16:29 | amrith | I repro it on my laptop | |
| 12:16:34 | amrith | just that one change | |
| 12:16:42 | amrith | this is the change to get around the nova pass through API's | |
| 12:16:55 | amrith | it is the first time we're seeing 8.0.0 | |
| 12:17:06 | amrith | and whatever else nova backend brings to the party | |
| 12:17:24 | amrith | to repro it just recheck that change https://review.openstack.org/#/c/467447/ | |
| 12:17:29 | amrith | and watch the multi job | |
| 12:17:50 | amrith | it'll get to the wait_for_non_affinity_repl_to_fail thing | |
| 12:17:51 | amrith | and hang | |
| 12:18:00 | amrith | because nova lists the instance (in nova list) as scheduling | |
| 12:18:04 | amrith | while it has actually failed | |
| 12:18:10 | sdague | mriedem: it's not just that change | |
| 12:18:21 | sdague | the entire trove gate looks like this is busted | |
| 12:18:27 | sdague | it's the multinode for mariadb | |
| 12:18:29 | amrith | in my laptop now ... http://paste.openstack.org/show/610724/ | |
| 12:18:51 | amrith | you can see that scheduler gave up, conductor errored but nova list says scheduling | |
| 12:18:52 | amrith | and yes to what sdague said | |
| 12:18:54 | amrith | currnetly the trove gate is borked | |
| 12:18:54 | mriedem | so, how does the trove CI setup cells v2? is it using devstack? | |
| 12:18:55 | sdague | https://review.openstack.org/#/c/468102/ is a change to CONTRIBUTING.rst that has the same pattern | |
| 12:19:13 | mriedem | did this just start? | |
| 12:19:18 | amrith | yes | |
| 12:19:31 | amrith | I thought it was related to the 8.0.0 stuff | |
| 12:19:34 | amrith | but not true | |
| 12:19:59 | amrith | this may have been the last successful run https://review.openstack.org/#/c/457223/ | |
| 12:20:24 | amrith | certainly didnt happen > 4 days ago | |
| 12:20:42 | mriedem | http://logs.openstack.org/47/467447/14/check/gate-trove-scenario-dsvm-mariadb-multi-ubuntu-xenial-nv/6efec50/logs/screen-n-sch.txt.gz#_May_26_03_24_31_029544 | |
| 12:20:52 | mriedem | the anti-affinity filter kicked it out, which is why you're getting NoValidHost, | |
| 12:20:57 | mriedem | also, the scheduler is only seeing 1 host | |
| 12:21:02 | sdague | mriedem: right, that's expected | |
| 12:21:06 | mriedem | are you using devstack? | |
| 12:21:08 | sdague | that's actually the point of the test | |
| 12:21:28 | amrith | mriedem it is devstack driven through trovestack (formerly redstack) | |
| 12:21:49 | amrith | i can send you my local confs if that will help | |
| 12:22:01 | amrith | but you can see the confs in the CI as well | |
| 12:22:02 | sdague | the real question is actually why nova-conductor fails to actually error the the node | |
| 12:22:16 | sdague | not that it fails to schedule | |
| 12:22:28 | mriedem | yes i realize | |
| 12:22:33 | sdague | ok, cool | |
| 12:24:39 | mriedem | odd that i can't get hits on this in logstash | |
| 12:24:51 | sdague | is conductor indexed? | |
| 12:24:58 | sdague | or these jobs? | |