Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-17
20:55:56 efried sdague If it's just coloring the req-id string itself (not the whole message), it should be harmless at worst... unless you wind up printing black-on-black.
20:57:08 melwitt cells meeting in a few min
20:57:27 sdague efried: yeh, I was thinking just the req-id string
21:34:02 melwitt dansmith: got a bunch of unit test fails on my patch bc I think I need to clear that cache between tests
21:34:10 dansmith melwitt: yep
21:34:14 dansmith just like we do for the others
21:34:35 melwitt I just grepped for CELLS in the tests and didn't find anything. where are you doing it?
21:34:43 melwitt er, didn't find a clearing of it
21:36:12 melwitt oh, derp, I grepped in nova/tests but it's in nova/test.py
21:36:19 melwitt nvm
21:36:19 dansmith yepo
21:41:08 cfriesen mriedem: dansmith: would something like this seem okay to you? http://paste.openstack.org/show/609851/ I'm a little bit worried about _rollback_live_migration() which only calls setup_networks_on_host() and not get_instance_nw_info() like the network-changed notifier does
21:53:30 openstackgerrit melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042
22:02:40 mriedem cfriesen: i don't really care for the instance.refresh thing
22:46:04 cburgess Is it actually possible to select additional scheduler filters to use on the boot command?
22:46:56 cfriesen mriedem: the point of that is to query from the DB the current task_state to see if it's migrating...got a better way to do that?
22:49:08 dansmith cfriesen: I don't see why we should skip refreshes altogether for migrating instances, nor do I think it's reasonable to do that heavy lookup every time
22:49:21 dansmith we already send events to both host and destination during migration
22:49:52 dansmith if you ask for any pre-review, can you do it in gerrit please? it's annoying to see only ten lines of context
22:50:31 cfriesen dansmith: we've seen the refresh take 35+ seconds due to lock contention with the incoming network-change events when there are 16 vifs. the events kept on acquiring the lock even though the migration code tried to get it first.
22:50:41 cfriesen dansmith: lack of fair locking was giving grief
22:52:59 cfriesen dansmith: with the 2-sec delay due to neutron batching, the DB lookup from instance.refresh() is way faster
22:53:31 cfriesen dansmith: and the reason to skip them for live-migrating instances is that after the migration is done we're going to refresh the cache explicitly
22:53:33 dansmith but for everyone else that's an extra db lookup on every event
22:55:23 cfriesen dansmith: got any suggestions? fair locking would help unblock the migration code but we'd still have the 2-sec neutron delays (and that seems serialized somehow so it takes ~8sec to finish getting all the network-changed events after a migration
22:56:34 mriedem cfriesen: do you really need the refresh at all?
22:56:56 mriedem what are the chances that the live migration completes and the task_state changes in the window of time between when the api gets the instances for the events and when you check in the compute?
22:58:08 cfriesen mriedem: that's why I asked you earlier...seems like there'd be a small race window where we might ignore an event we should have processed
22:58:38 cfriesen whether that would actually cause problems...I don't know
22:58:58 melwitt cburgess: what do you mean by select? other than adding them to enabled_filters in your nova.conf?
23:00:06 cfriesen mriedem: dansmith: to ignore it the instance would have been migrating at the time nova-api pulled the instance from the DB, so I *think* it should be valid to ignore the event
23:00:16 cburgess melwitt Exactly. We have available_filters and default_filters in nova.conf. default gets used when you do a launch but is there a way as part of a boot to add a filter to that list or use a different filter?
23:01:38 cfriesen cburgess: I don't see a way...get_filtered_hosts() calls self.filter_handler.get_filtered_objects(self.enabled_filters....) where self.enabled_filters is used as the list of filters to evaluate
23:01:45 melwitt cburgess: you mean potentially use different filters for different boot requests? I don't think so
23:02:02 dansmith cfriesen: I don't see why it's valid to ignore it ever, but especially not just because you might've lost the race where we already looked it up in the api
23:04:06 cfriesen dansmith: the event handler for the network-changed message just calls self.network_api.get_instance_nw_info(). the live-migration code will call that explicitly when the migration finishes.
23:04:32 cfriesen that's a relatively expensive call, so we'd like to minimize the number of times we call it
23:05:03 cfriesen and we'd like to avoid the main migration code contending on the lock with the events
23:05:24 cburgess melwitt cfriesen So whats what I thought. Which begs the question.. why do we have 2 config values?
23:05:39 melwitt cburgess: I think the only way to get different behavior for different boot would be scheduler hints (and add a filter that handles that hint) or extra specs to do it via flavor
23:06:03 cburgess melwitt Ahh that could be why hints. I haven't looked at that.
23:06:11 melwitt cburgess: that's a good question ... checking
23:06:29 cfriesen cburgess: 2 config values for what/
23:06:32 cfriesen ?
23:06:54 melwitt like available filters vs enabled filters
23:07:00 cburgess cfriesen scheduler_default_filters vs scheduler_available_filters
23:07:02 dansmith cfriesen: some virt driver could need to wait for some update during live migration and need to see that refresh to proceed, no?
23:07:03 melwitt I actually don't know why
23:07:12 dansmith cfriesen: sounds like you want to fix the locking problem instead of just whack-a-mole'ing your immediate problem
23:08:52 cfriesen dansmith: there are a few issues: 1) lock isn't fair, 2) the call to neutron is expensive, 3) what does it buy us to call self.network_api.get_instance_nw_info() a bunch of times during a single live-migration?
23:09:48 dansmith cfriesen: fix #1, skip duplicate instances as you are, and I just wrote a virt driver that depends on those refreshes to complete phases of my live migration.
23:12:09 cburgess melwitt Yeah ok thats what I thought.. superfluous... I'll keep digging into the hints thing but it really look to me like its redundant.
23:12:40 cfriesen dansmith: would a virt driver ever use InstanceInfoCache?
23:13:11 dansmith cfriesen: sure, the one I just wrote does
23:13:21 melwitt cburgess: yeah, I'm not seeing any clues as to why there are two options. maybe a legacy thing that used to have a purpose. jaypipes, bauzas, or edleafe might know
23:13:29 cfriesen :)
23:14:24 cfriesen dansmith: okay, so assuming we do the locking thing...do you think we would want to modify the standard lock (likely making it somewhat slower in the fastpath) or introduce a new "fair" lock and hope services switch to using it?
23:15:03 dansmith cfriesen: there's lots of network_info walking throughout the xen driver at least..
23:15:35 dansmith cfriesen: I have no idea what you want to do, and would suggest you ask the lockutils people what they think
23:16:25 cfriesen dansmith: I think it'd be nuts to rely on incoming network-changed events during a live migration since they can be processed in any order due to RPC and locking contention. But who knows what crazy stuff xen might do.
23:16:33 cfriesen dansmith: fair enough
23:17:02 dansmith if you're waiting for some property on a vif to become a value, I see no reason why that wouldn't be reasonable
23:17:21 dansmith anyway, EOD and EOP for me
23:17:33 cfriesen thanks for the feedback
23:24:39 mriedem SteelyDanHypervisor?
23:26:05 mriedem i'm going to need someone to run the nova team meeting tomorrow...
23:26:13 cfriesen 10thDanBlackBeltHypervisor
23:27:41 mriedem also, i wonder if dan is aware of dan's opinions on out of tree virt drivers
23:31:49 mriedem oomichi: were you interested in taking over either https://blueprints.launchpad.net/nova/+spec/live-migration-per-instance-timeout or https://blueprints.launchpad.net/nova/+spec/live-migration-force-after-timeout ?
23:51:07 openstackgerrit Ed Leafe proposed openstack/nova master: Have scheduler claim resources https://review.openstack.org/463993
23:51:07 openstackgerrit Ed Leafe proposed openstack/nova master: Remove selecting host based on host_subset_size https://review.openstack.org/465174
23:51:08 openstackgerrit Ed Leafe proposed openstack/nova master: Make the method to put allocations public https://review.openstack.org/464086
23:51:08 openstackgerrit Ed Leafe proposed openstack/nova master: Don't delete allocation if instance being scheduled https://review.openstack.org/460177
23:51:09 openstackgerrit Ed Leafe proposed openstack/nova master: Pass a list of instance UUIDs to scheduler https://review.openstack.org/465171
23:51:09 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - Pass the number of hosts to return explicitly https://review.openstack.org/465767
23:51:10 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - return multiple hosts from select_destinations https://review.openstack.org/465768
23:51:27 mriedem melwitt: do you want to tweak the reno at all in this? https://review.openstack.org/#/c/465042/
23:51:29 mriedem if not, i'm +2
23:52:48 melwitt mriedem: yeah, I can change it to your suggest and flip the test assertions. I wasn't thinking when I copied them
23:52:54 melwitt *suggestion
23:53:55 openstackgerrit Hongbin Lu proposed openstack/nova master: Handle exception on adding secgroup https://review.openstack.org/465173
23:55:31 cfriesen mriedem: so it looks like _build_network_info_model() costs about 200ms plus about 125ms per port since we query each port separatly from neutron. and the refresh_cache lock is held the whole time
23:56:18 cfriesen mriedem: I wonder if we could issues those requests in parallel for all the ports?
23:56:33 mriedem cfriesen: we've talked for years about needing to do bulk queries from neutron
23:57:45 mriedem well,
23:57:57 mriedem where do we re-query per port, after we get the list of ports from data = client.list_ports(**search_opts)
23:58:23 cfriesen _nw_info_get_ips and _nw_info_get_subnets both take noticeable time
23:58:32 cfriesen for each port
23:59:28 mriedem the first is going to iterate over all fixed_ips for a port and get floating IPs for each fixed_ip
#openstack-nova - 2017-05-18
00:01:32 mriedem i wonder if we could bulk query the floating IPs by the list of fixed IPs and the port_id
00:01:46 openstackgerrit melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042
00:02:07 mriedem "When you specify multiple filters, the Networking API v2.0 returns only objects that meet all filtering criteria. The operation applies an AND condition among the filters."
00:02:17 mriedem so that wouldn't really work,
00:02:27 mriedem because fixed_ip1 AND fixed_ip2 wouldn't resolve to a floating IP
00:02:45 cfriesen mriedem: we get an event for each vif, and we have to get info for all vifs on each event, so when we have N vifs, the live-migration overhead goes up as N^2
00:02:47 mriedem we could trim the fat on the fields we actually need to get back
00:03:07 cfriesen mriedem: it'd be better to have the vif specified in the network-changed event
00:03:24 cfriesen mriedem: and avoid having to scan all the vifs
00:03:37 mriedem cfriesen: yeah i'm not sure how feasible that is from the neutron side
00:03:46 mriedem i guess it would depend on how they calculate the network-changed event

Earlier   Later