Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-20
15:11:14 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Modify signature of _filter_non_requested_pfs https://review.openstack.org/527473
15:11:14 openstackgerrit Stephen Finucane proposed openstack/nova master: Add PCI NUMA policies https://review.openstack.org/527472
15:11:17 edleafe jaypipes: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L139-L143
15:11:47 edleafe it calls it an iterator, which is technically correct, but it's a generator
15:11:55 mdbooth jaypipes: So [foo for foo in foos] is a list comprehension
15:11:56 stephenfin cfriesen, bauzas: Fancy taking a look at ^^^ again? Has been changed quite a lot (hopefully for the better)
15:12:11 mdbooth And (foo for foo in foos) is a generator expression which iterates over the same thing
15:13:37 jaypipes hmm, interesting.
15:14:34 mdbooth So if that last line is returning a generator, it's essentially a closure over seen_nodes, which is local, and self.host_state_map, which is global.
15:15:08 jaypipes yeah, I understand that now. Why don't we just list() that instead of returning a generator?
15:15:26 mdbooth Well that would still be a race, just a shorter one
15:15:52 openstack Launchpad bug 1739440 in OpenStack Compute (nova) "Ironic virt driver is hardcoding Ironic API microversion" [Undecided,New]
15:15:52 dtantsur hey folks! I've filed https://bugs.launchpad.net/nova/+bug/1739440 to track ironic API version negotiation in the virt driver.
15:16:02 dtantsur do you think it makes sense as a bug or a blueprint or a spec or ...?
15:16:03 jaypipes cool, thanks dtantsur
15:16:13 edleafe jaypipes: in a large deployment, it was feared that the number of hosts returned would be huge
15:16:15 jaypipes dtantsur: up to mriedem_away
15:16:35 jaypipes edleafe: so premature optimization, then. just like 90% of the code in the scheduler.'
15:16:44 edleafe yeah, pretty much
15:16:50 dtantsur jaypipes: how away is mriedem_away? :)
15:16:57 jaypipes dtantsur: no idea :)
15:17:08 mdbooth dtantsur: He wasn't away within the last 30 minutes.
15:17:17 edleafe I don't know if that was a result of real-world problems, or just premature, TBH
15:17:34 dtantsur okay, so not like away-for-the-reminder-of-the-year state of being away :)
15:17:50 mdbooth dtantsur: mriedem_away isn't allowed PTO
15:18:04 dtantsur I see, makes sense :D
15:22:43 jaypipes mdbooth: bauzas changed the return of _get_host_states() from an iterator over host_state_map's values to the generator expression that included the local seen_nodes variable here: https://github.com/openstack/nova/commit/4660333d0d97d8e00cf290ea1d4ed932f5edc1dc#diff-978b9f8734365934eaf8fbb01f11a7d7L624
15:23:37 jaypipes mdbooth: might be worth trying to reproduce this race before and after that patch to see if it makes a diff
15:24:32 openstackgerrit Matthew Booth proposed openstack/nova master: Fix an error in _get_host_states when deleting a compute node https://review.openstack.org/529352
15:24:38 jaypipes note that I'm not saying anything bad about bauzas' patch! :) I'm just pointing out that's the patch that changed from returning an iterator over the global state to a generator expression over local state
15:25:03 mdbooth jaypipes: Looking
15:25:24 openstackgerrit Chris Dent proposed openstack/nova master: Provide example for placement last-modified header of now https://review.openstack.org/529354
15:25:32 jaypipes mdbooth: of course, trying to reproduce this reliably is virtually impossible...
15:25:57 mdbooth jaypipes: I've written a couple of unit tests now which play with execution order
15:26:03 mdbooth They're always really hard to read, though
15:26:59 jaypipes yta
15:27:01 jaypipes ya
15:28:51 mdbooth Although as this is a generator it might be pretty easy
15:37:18 mdbooth jaypipes: mriedem_away Reproduced it
15:37:37 mdbooth Not sure if the unit test is worth anything, though, as it's irrelevant with my patch in place
15:38:55 mdbooth You have to create an iterator from the returned list in order to trigger a bug evaluating the iterator, which really doesn't make sense if it doesn't return an iterator
15:40:44 mdbooth http://paste.openstack.org/show/629469/
15:41:09 mdbooth ^^^ There's the test and its execution, anyway. But I don't think it makes sense to include it if we're not returning an iterator any more.
15:53:59 openstackgerrit Jay Pipes proposed openstack/nova master: WIP Support aggregate affinity filters https://review.openstack.org/529201
15:53:59 openstackgerrit Jay Pipes proposed openstack/nova master: Remove server group sched filter support caching https://review.openstack.org/529200
16:12:48 mriedem just read the scrollback
16:12:59 mriedem https://review.openstack.org/529343
16:13:05 mriedem i thought it was a tuple of HostState objects too
16:13:18 mriedem because of the change jaypipes pointed out where we stopped using six.itervalues
16:14:38 mriedem mdbooth: so with your change, we lose any yield behavior on the generator for the 10s of thousands of nodes optimization?
16:19:34 mnaser ok reading scrollback
16:19:43 mriedem ah v
16:19:44 mriedem http://intermediatepythonista.com/python-generators#generator-expressions_1
16:19:50 mriedem ok i'm sorely lacking in this area then
16:20:20 mnaser ok but forgive me if i'm being unclear but i think the problem is a lot simpler than that
16:20:35 mnaser seen_nodes contains a tuple list of (host, node) of all hosts in the system
16:20:47 mnaser dead_notes contains a tuple list of (host, node) of all dead hosts in the system
16:21:11 mnaser as part of the for loop right before the generator, all dead_nodes are removed from self.host_state_map
16:21:32 mnaser which leaves self.host_state_map with keys of the alive hosts *only*
16:21:52 mnaser the generator loops over all of self.host_state_map but uses seen_nodes as the key (which is all hosts)
16:22:03 mnaser which includes dead nodes, that were removed, in the loop just prior
16:22:09 mriedem mnaser: dead_nodes is the set of entries in host_state_map that aren't also in seen_nodes
16:22:21 mriedem so it's removing existing dead entries from host_state_map
16:22:23 mriedem that's what the set is for
16:22:36 mnaser OH
16:22:39 mnaser i see what you mean
16:22:56 mriedem so i think mdbooth's analysis in the commit message here is correct https://review.openstack.org/#/c/529352/
16:23:11 mriedem because since we're using a generator (which i didn't realize),
16:23:28 mriedem the host_state_map results could change while processing a request,
16:23:39 mriedem if a compute node is deleted in between, in a busy cloud, such as someone's public cloud
16:24:01 mnaser ok but im pretty sure this compute node wasnt deleted, it still exists but it is dead
16:24:06 mnaser let me please double check
16:24:10 mriedem mdbooth: rathre than change it from returning a generator to a list, why not just check if the key is still in the host_state_map before returning the next item?
16:25:10 mriedem mnaser: maybe the service was deleted?
16:25:19 mriedem you can't delete a compute node through the REST API, but you can delete a service
16:25:32 mriedem https://review.openstack.org/#/c/529352/1/nova/scheduler/host_manager.py@686
16:26:19 mnaser on one conductor i see 40 instances of this with the most recent happening on the 20th
16:26:29 mriedem so i think a simple test could be you run _get_host_states once with 2 computes and 2 services, then delete one of the services and run _get_host_states again, and iterate the results and it should blow up
16:27:15 mnaser # grep KeyError /var/log/nova/nova-scheduler.log | grep req | wc -l => 20
16:27:20 mnaser one one of the schedulers
16:29:03 mnaser happened with 9 different compute nodes in the KeyError
16:29:29 mnaser and i know for a fact some of thoser havent been deleted, especially those a few days ago too
16:29:31 mriedem do those computes have corresponding services table entries?
16:29:52 mnaser mriedem: i mean i see them in 'nova service-list'
16:29:58 mnaser and their updated_at is recent so they're reporting in
16:30:13 mnaser afaik the only way for that service record to go away is... if someone deleted it?
16:30:38 mriedem correct
16:31:10 mriedem we do start auto-disabling computes that fail to build 10 times in a row
16:31:20 mriedem but in the scheduler code here, we include disabled services
16:31:54 mnaser mriedem: i think we disabled that because scheduling bugs would shut down our entire cloud lol
16:32:12 mnaser ex: failing to place numa on server will result in a build fail count being increased
16:38:57 mriedem mdbooth: ok i think we can still use part of your test and maintain the generator
16:39:02 mriedem i'm going to update your patch quick
16:39:18 mdbooth mriedem: on a call, but that's cool
16:42:46 cdent jaypipes, mriedem: if you're both still around, this is a good and relatively straightforward win for placement behaving in a version discovery world: https://review.openstack.org/#/c/522002/
16:43:21 mdbooth mriedem: Did you see my unit test, btw?
16:43:38 mnaser mriedem: im not sure if this is relevant or not but do you think the fact we run 3x nova-schedulers could play a role in this .. somehow
16:44:10 mriedem mdbooth: yes i pulled part of the test and applied it to an existing test to recreate the bug
16:44:21 mriedem mnaser: separate processes?
16:44:30 mnaser yes, on seperate machines too

Earlier   Later