Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-17
20:42:26 zzzeek melwitt: in here I see "db.create_context_manager()" which seems pretty clean. what determines the other settings for this CELL db, they are taken from the global settings? like how do we know it's utf-8 encoding, stuff like that ?
20:43:08 melwitt zzzeek: they all use the same config, they're just clones of what we call the "main DB" they all have the same schema
20:43:36 dansmith clone in the schema sense
20:43:52 melwitt yeah, sorry
20:44:15 zzzeek melwitt: yup. OK so oslo.db could have a featrue like this. like in enginefacade, you give it the URL and some flag, "use_existing" or something, pulls it from a dictionaar like you have here keyed to URL
20:47:47 melwitt zzzeek: okay, cool. thanks. just wanted a sanity check of whether there were more appropriate objects to use that we didn't use
20:48:39 sdague efried: totally feel free to, it was a nice add
20:52:18 efried sdague Pursuant to your colors-by-hash idea, it occurred to me that it might be more useful to do that by service, now that we've got converged logs.
20:53:21 sdague efried: you don't have access that early
20:53:22 efried Just thinking whether consumer is more likely to grep away by req-id or by service - and I'm thinking it's the former by a mile.
20:53:29 sdague that's set by systemd itself
20:53:49 sdague efried: you don't need to grep by service though
20:53:57 sdague because -u does that for you
20:54:00 efried sdague Yeah, exactly my point.
20:54:18 efried It would be a visual break showing you when you're jumping services.
20:54:20 cfriesen task_states.MIGRATING is only for live-migration, right?
20:54:37 sdague right, so the reason I thought it would be nice to rainbow hash the req-id is that it becomes more clear when you have competing flows happening at the some time
20:54:52 sdague most of our races are actually related to that
20:54:57 efried sdague Makes sense.
20:55:19 sdague it's potentially totally gratuitous, but it was enough to wonder "hey, that might be useful"
20:55:56 efried sdague If it's just coloring the req-id string itself (not the whole message), it should be harmless at worst... unless you wind up printing black-on-black.
20:57:08 melwitt cells meeting in a few min
20:57:27 sdague efried: yeh, I was thinking just the req-id string
21:34:02 melwitt dansmith: got a bunch of unit test fails on my patch bc I think I need to clear that cache between tests
21:34:10 dansmith melwitt: yep
21:34:14 dansmith just like we do for the others
21:34:35 melwitt I just grepped for CELLS in the tests and didn't find anything. where are you doing it?
21:34:43 melwitt er, didn't find a clearing of it
21:36:12 melwitt oh, derp, I grepped in nova/tests but it's in nova/test.py
21:36:19 melwitt nvm
21:36:19 dansmith yepo
21:41:08 cfriesen mriedem: dansmith: would something like this seem okay to you? http://paste.openstack.org/show/609851/ I'm a little bit worried about _rollback_live_migration() which only calls setup_networks_on_host() and not get_instance_nw_info() like the network-changed notifier does
21:53:30 openstackgerrit melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042
22:02:40 mriedem cfriesen: i don't really care for the instance.refresh thing
22:46:04 cburgess Is it actually possible to select additional scheduler filters to use on the boot command?
22:46:56 cfriesen mriedem: the point of that is to query from the DB the current task_state to see if it's migrating...got a better way to do that?
22:49:08 dansmith cfriesen: I don't see why we should skip refreshes altogether for migrating instances, nor do I think it's reasonable to do that heavy lookup every time
22:49:21 dansmith we already send events to both host and destination during migration
22:49:52 dansmith if you ask for any pre-review, can you do it in gerrit please? it's annoying to see only ten lines of context
22:50:31 cfriesen dansmith: we've seen the refresh take 35+ seconds due to lock contention with the incoming network-change events when there are 16 vifs. the events kept on acquiring the lock even though the migration code tried to get it first.
22:50:41 cfriesen dansmith: lack of fair locking was giving grief
22:52:59 cfriesen dansmith: with the 2-sec delay due to neutron batching, the DB lookup from instance.refresh() is way faster
22:53:31 cfriesen dansmith: and the reason to skip them for live-migrating instances is that after the migration is done we're going to refresh the cache explicitly
22:53:33 dansmith but for everyone else that's an extra db lookup on every event
22:55:23 cfriesen dansmith: got any suggestions? fair locking would help unblock the migration code but we'd still have the 2-sec neutron delays (and that seems serialized somehow so it takes ~8sec to finish getting all the network-changed events after a migration
22:56:34 mriedem cfriesen: do you really need the refresh at all?
22:56:56 mriedem what are the chances that the live migration completes and the task_state changes in the window of time between when the api gets the instances for the events and when you check in the compute?
22:58:08 cfriesen mriedem: that's why I asked you earlier...seems like there'd be a small race window where we might ignore an event we should have processed
22:58:38 cfriesen whether that would actually cause problems...I don't know
22:58:58 melwitt cburgess: what do you mean by select? other than adding them to enabled_filters in your nova.conf?
23:00:06 cfriesen mriedem: dansmith: to ignore it the instance would have been migrating at the time nova-api pulled the instance from the DB, so I *think* it should be valid to ignore the event
23:00:16 cburgess melwitt Exactly. We have available_filters and default_filters in nova.conf. default gets used when you do a launch but is there a way as part of a boot to add a filter to that list or use a different filter?
23:01:38 cfriesen cburgess: I don't see a way...get_filtered_hosts() calls self.filter_handler.get_filtered_objects(self.enabled_filters....) where self.enabled_filters is used as the list of filters to evaluate
23:01:45 melwitt cburgess: you mean potentially use different filters for different boot requests? I don't think so
23:02:02 dansmith cfriesen: I don't see why it's valid to ignore it ever, but especially not just because you might've lost the race where we already looked it up in the api
23:04:06 cfriesen dansmith: the event handler for the network-changed message just calls self.network_api.get_instance_nw_info(). the live-migration code will call that explicitly when the migration finishes.
23:04:32 cfriesen that's a relatively expensive call, so we'd like to minimize the number of times we call it
23:05:03 cfriesen and we'd like to avoid the main migration code contending on the lock with the events
23:05:24 cburgess melwitt cfriesen So whats what I thought. Which begs the question.. why do we have 2 config values?
23:05:39 melwitt cburgess: I think the only way to get different behavior for different boot would be scheduler hints (and add a filter that handles that hint) or extra specs to do it via flavor
23:06:03 cburgess melwitt Ahh that could be why hints. I haven't looked at that.
23:06:11 melwitt cburgess: that's a good question ... checking
23:06:29 cfriesen cburgess: 2 config values for what/
23:06:32 cfriesen ?
23:06:54 melwitt like available filters vs enabled filters
23:07:00 cburgess cfriesen scheduler_default_filters vs scheduler_available_filters
23:07:02 dansmith cfriesen: some virt driver could need to wait for some update during live migration and need to see that refresh to proceed, no?
23:07:03 melwitt I actually don't know why
23:07:12 dansmith cfriesen: sounds like you want to fix the locking problem instead of just whack-a-mole'ing your immediate problem
23:08:52 cfriesen dansmith: there are a few issues: 1) lock isn't fair, 2) the call to neutron is expensive, 3) what does it buy us to call self.network_api.get_instance_nw_info() a bunch of times during a single live-migration?
23:09:48 dansmith cfriesen: fix #1, skip duplicate instances as you are, and I just wrote a virt driver that depends on those refreshes to complete phases of my live migration.
23:12:09 cburgess melwitt Yeah ok thats what I thought.. superfluous... I'll keep digging into the hints thing but it really look to me like its redundant.
23:12:40 cfriesen dansmith: would a virt driver ever use InstanceInfoCache?
23:13:11 dansmith cfriesen: sure, the one I just wrote does
23:13:21 melwitt cburgess: yeah, I'm not seeing any clues as to why there are two options. maybe a legacy thing that used to have a purpose. jaypipes, bauzas, or edleafe might know
23:13:29 cfriesen :)
23:14:24 cfriesen dansmith: okay, so assuming we do the locking thing...do you think we would want to modify the standard lock (likely making it somewhat slower in the fastpath) or introduce a new "fair" lock and hope services switch to using it?
23:15:03 dansmith cfriesen: there's lots of network_info walking throughout the xen driver at least..
23:15:35 dansmith cfriesen: I have no idea what you want to do, and would suggest you ask the lockutils people what they think
23:16:25 cfriesen dansmith: I think it'd be nuts to rely on incoming network-changed events during a live migration since they can be processed in any order due to RPC and locking contention. But who knows what crazy stuff xen might do.
23:16:33 cfriesen dansmith: fair enough
23:17:02 dansmith if you're waiting for some property on a vif to become a value, I see no reason why that wouldn't be reasonable
23:17:21 dansmith anyway, EOD and EOP for me
23:17:33 cfriesen thanks for the feedback
23:24:39 mriedem SteelyDanHypervisor?
23:26:05 mriedem i'm going to need someone to run the nova team meeting tomorrow...
23:26:13 cfriesen 10thDanBlackBeltHypervisor
23:27:41 mriedem also, i wonder if dan is aware of dan's opinions on out of tree virt drivers
23:31:49 mriedem oomichi: were you interested in taking over either https://blueprints.launchpad.net/nova/+spec/live-migration-per-instance-timeout or https://blueprints.launchpad.net/nova/+spec/live-migration-force-after-timeout ?
23:51:07 openstackgerrit Ed Leafe proposed openstack/nova master: Have scheduler claim resources https://review.openstack.org/463993
23:51:07 openstackgerrit Ed Leafe proposed openstack/nova master: Remove selecting host based on host_subset_size https://review.openstack.org/465174
23:51:08 openstackgerrit Ed Leafe proposed openstack/nova master: Make the method to put allocations public https://review.openstack.org/464086
23:51:08 openstackgerrit Ed Leafe proposed openstack/nova master: Don't delete allocation if instance being scheduled https://review.openstack.org/460177
23:51:09 openstackgerrit Ed Leafe proposed openstack/nova master: Pass a list of instance UUIDs to scheduler https://review.openstack.org/465171
23:51:09 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - Pass the number of hosts to return explicitly https://review.openstack.org/465767
23:51:10 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - return multiple hosts from select_destinations https://review.openstack.org/465768
23:51:27 mriedem melwitt: do you want to tweak the reno at all in this? https://review.openstack.org/#/c/465042/
23:51:29 mriedem if not, i'm +2
23:52:48 melwitt mriedem: yeah, I can change it to your suggest and flip the test assertions. I wasn't thinking when I copied them
23:52:54 melwitt *suggestion

Earlier   Later