Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-05
14:50:23 mriedem and this isn't really the place to debug that level of detail probably
14:51:06 mriedem mordred: ok so back to your thing, this stacktrace is when the compute starts up and calls update_instance_info before the host is mapped http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-sch.txt.gz#_Jun_02_12_45_15_755877
14:51:14 mriedem then we map the host here http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/devstacklog.txt.gz#_2017-06-02_12_46_19_540
14:52:07 mriedem the server create request hits the scheduler here http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-sch.txt.gz#_Jun_02_12_46_59_055921
14:52:18 mriedem 40 seconds later
14:55:12 mriedem mordred: methinks the problem isn't with the compute host not being mapped,
14:55:20 mriedem but rather the compute host failing to register with placement
14:55:49 mriedem b/c the scheduler is asking placement for resource providers (comptue nodes) to satisfy the request
14:56:44 openstackgerrit Sean Dague proposed openstack/nova master: Be more tolerant of keystone catalog configuration https://review.openstack.org/438049
14:58:48 openstackgerrit Eric Berglund proposed openstack/nova master: Add PowerVM to nova support matrix https://review.openstack.org/470999
14:59:26 mordred mriedem: ah -ok that makes sense to me
14:59:42 mriedem mordred: still tracing the n-cpu logs
14:59:59 mriedem sdague: fyi while you were out, a few of the clients you needed for the global request id stuff got blacklisted
15:00:06 mriedem the versions that is, neutronclient 6.3.0 for sure
15:00:36 cdent dansmith: that was just an example of a pattern, not a plan
15:00:37 openstackgerrit Eric Berglund proposed openstack/nova master: Add PowerVM to nova support matrix https://review.openstack.org/470999
15:00:53 sdague mriedem: yeh, saw that, but the glance one looks fine now
15:01:02 edleafe dansmith: what cdent said
15:01:09 sdague mriedem: so I flipped that around
15:01:19 dansmith cdent: okay it seemed like it would be specifically targeted stuff for nova's scheduler though
15:01:50 edleafe dansmith: it doesn't have to be
15:01:50 jaypipes cdent, alex_xu, edleafe, dansmith: join #novaschedulercontd
15:02:00 dansmith jaypipes: ugh, can't we hangout?
15:02:13 jaypipes dansmith: I'm happy to hangout.
15:02:16 dansmith jaypipes: new irc channels ftl, even if short-lived
15:02:30 cdent i can hangout
15:02:37 jaypipes ok, let's do a hangout.
15:02:42 jaypipes one sec, I will set up
15:02:45 dansmith can we have five mins for food/coffee?
15:02:50 dansmith I have had neither yet
15:02:54 edleafe dansmith: +1
15:03:12 edleafe How about at <yourtime>:15 ?
15:03:24 cdent wfm
15:03:29 mriedem sfinucan: looks like we deprecated the use_neutron config option, which is going to be a bit confusing if you're using nova-network with cells v1
15:03:38 openstackgerrit Chris Dent proposed openstack/nova master: Sync os-traits to Traits database https://review.openstack.org/469578
15:04:19 dansmith yeah
15:04:31 jaypipes dansmith, cdent, edleafe, alex_xu: https://hangouts.google.com/call/fxu4kc4ufrakhiwjyq6idxrmcqu
15:04:31 dansmith I can fill out three or four employment applications in 15 mins
15:04:54 jaypipes I'll just sit in the hangout. join as you will.
15:05:02 cfriesen_ did anyone sort out the issue with jenkins getting stricter about releasenotes?
15:06:05 cfriesen_ I'm looking at this in particular, hitting it on my local machine as well: https://review.openstack.org/#/c/435141
15:07:06 lei-zh jaypipes, for requesting traits part, are we going to let users to specify traits associated with top resource providers and nested ones?
15:09:34 jaypipes lei-zh: my initial thoughts on that matter were that we would continue to use the flavor as the thing that users would request, and flavors would have a "required" attribute that was a list of string traits that were required by that flavor.
15:10:30 jaypipes lei-zh: long-term, I would love to have a compute API that allowed the user (or certain users I suppose) to request specific traits directly and not have to request a particular flavor.
15:10:39 jaypipes lei-zh: but we're a ways from that. :)
15:11:23 alex_xu jaypipes: lei-zh , I'm thinking the meaning of required_traits is that 'give me a resource provider or it's nested provider includes those required traits'
15:11:51 jaypipes alex_xu: correct. (also shared provider)
15:13:30 mordred mriedem: in general, please let me know if there is anything I can do to help - I'm pretty sure we've passed my level of understanding of being helpful with debugging further, but I'll dig in to whatever if it's useful
15:14:09 mriedem mordred: sure, if it's placement it should be fairly easy for me to see what's up
15:14:34 mordred mriedem: also, is there any way as a user to get "no hosts could schedule this request" is the cause of the error?
15:19:25 sdague mriedem: did you ever get the mysql specific test case merged from a couple of weeks ago?
15:20:47 mriedem sdague: nope
15:20:53 mriedem sdague: haven't found time to get back to it yet
15:21:00 sdague mriedem: was it working?
15:21:02 sdague or not?
15:21:27 mriedem sort of
15:21:42 mriedem https://review.openstack.org/#/c/467772/
15:21:54 mriedem https://review.openstack.org/#/c/467772/2/nova/tests/functional/test_servers.py@1029
15:22:21 mriedem i think we have to do some things manually for setting up the cell mapping pointed at the mysql backend
15:22:34 mriedem and then create the HostMapping and map that to the CellMapping manually also
15:22:42 mriedem for our single nova-compute service
15:22:58 mriedem mordred: the user gets a 202 so they don't see the cause
15:22:59 sdague ok, maybe I'll play with it
15:23:12 mriedem mordred: the best you can get is an instance fault recorded on the instance, but those are visible only to admins by default
15:28:09 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Test removing live_migration_uri from devstack https://review.openstack.org/471019
15:29:47 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/470937
15:32:47 mordred mriedem: yah. that was what I was seeing. do you think it would be worth trying to write up a spec on exposing just a little more data in the error case?
15:33:25 mordred mriedem: I know we've hit issues before where nova couldn't boot an instance but the error was ultimately "this image is in an invalid format" - but is shows up just as "can't boot"
15:33:50 mordred mriedem: I'd be happy to try to write something up - but if it's a bad or unworkable area there's no need to bother of course
15:34:50 mriedem mordred: it's been tried before, usually ends up being a problem because of leaking internal details about the cloud so could be used for attacks
15:35:12 mriedem cfriesen_ had a spec and some code for improving the filtering in the scheduler logs to help operators, but that doesn't help the end user
15:35:24 mriedem except get their ticket resolved faster maybe
15:35:59 mordred mriedem: yah - that's what I'd be worried about from a "this spec cant' happen' pov
15:37:56 mriedem mordred: so looking at your issue,
15:38:02 mriedem nova-compute is failing to talk to placement http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-cpu.txt.gz#_Jun_02_12_46_14_477554
15:38:06 mriedem getting 503 back
15:38:26 mriedem which means as far as placement and the scheduler are concerned, there are no compute nodes available
15:39:15 mriedem mordred: the compute node finally gets registered here: http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-cpu.txt.gz#_Jun_02_12_47_14_576829
15:39:34 mriedem which is 15 seconds after your test failed
15:42:15 mordred mriedem: ah - so there was an early issue, but it eventually heals
15:43:00 mordred mriedem: so in terms of the hacky thing I had for the tests, I could actually check for "Created resource provider record via placement API...$(hostname)" in the n-cpu logs
15:43:12 mordred as a check for "is the cloud up and ready for action" before starting the tests
15:44:04 mriedem well,
15:44:19 mriedem the better less hacky thing would be to GET /resource_providers against the placement endpoint
15:44:36 mriedem until you get something back that's not an empty body
15:44:47 mordred mriedem: ah - yah. could do that for sure
15:44:53 mriedem note that placement is admin-only by default
15:45:18 mordred mriedem: that's ok - we've got admin accounts in the functional tets
15:45:20 mordred tests
15:45:22 mriedem i'm also curious why placement is failing in the first place
15:45:28 mhenkel_ mriedem: when you find some time, would you mind to take a look into https://review.openstack.org/#/c/441183/ ?
15:45:34 mriedem that should be started well before n-cpu and n-sch
15:45:57 mhenkel_ it passed ci and I addressed all the open points
15:46:03 mordred I don't think this is actually a thing where shade sohuld be checking for the error in normal usage -it seems like a pretty specific first-install devstack race from an api consumer perspective
15:46:17 mriedem mordred: also, i'm not sure why this would have regressed in the last few days, i don't see anything related to changing the startup order in devstack
15:46:39 mordred mriedem: it might just have always been a race and something else caused something else to be slow so it just started showing up
15:47:09 cfriesen_ mriedem: mordred: in our environment the users are generally very tightly couple to the admins, so we expose all the sched failure logs directly to the user
15:47:13 mordred mriedem: or, actually,it could have always been here, but now something is improved so we get all the way to tests before the self-healing
15:47:32 mordred cfriesen_: yah - I have admin on almost none of my clouds :)
15:47:50 mordred so I'm usually the annoying voice of "how do I learn this as a non-admin"

Earlier   Later