Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-17
23:00:16 cburgess melwitt Exactly. We have available_filters and default_filters in nova.conf. default gets used when you do a launch but is there a way as part of a boot to add a filter to that list or use a different filter?
23:01:38 cfriesen cburgess: I don't see a way...get_filtered_hosts() calls self.filter_handler.get_filtered_objects(self.enabled_filters....) where self.enabled_filters is used as the list of filters to evaluate
23:01:45 melwitt cburgess: you mean potentially use different filters for different boot requests? I don't think so
23:02:02 dansmith cfriesen: I don't see why it's valid to ignore it ever, but especially not just because you might've lost the race where we already looked it up in the api
23:04:06 cfriesen dansmith: the event handler for the network-changed message just calls self.network_api.get_instance_nw_info(). the live-migration code will call that explicitly when the migration finishes.
23:04:32 cfriesen that's a relatively expensive call, so we'd like to minimize the number of times we call it
23:05:03 cfriesen and we'd like to avoid the main migration code contending on the lock with the events
23:05:24 cburgess melwitt cfriesen So whats what I thought. Which begs the question.. why do we have 2 config values?
23:05:39 melwitt cburgess: I think the only way to get different behavior for different boot would be scheduler hints (and add a filter that handles that hint) or extra specs to do it via flavor
23:06:03 cburgess melwitt Ahh that could be why hints. I haven't looked at that.
23:06:11 melwitt cburgess: that's a good question ... checking
23:06:29 cfriesen cburgess: 2 config values for what/
23:06:32 cfriesen ?
23:06:54 melwitt like available filters vs enabled filters
23:07:00 cburgess cfriesen scheduler_default_filters vs scheduler_available_filters
23:07:02 dansmith cfriesen: some virt driver could need to wait for some update during live migration and need to see that refresh to proceed, no?
23:07:03 melwitt I actually don't know why
23:07:12 dansmith cfriesen: sounds like you want to fix the locking problem instead of just whack-a-mole'ing your immediate problem
23:08:52 cfriesen dansmith: there are a few issues: 1) lock isn't fair, 2) the call to neutron is expensive, 3) what does it buy us to call self.network_api.get_instance_nw_info() a bunch of times during a single live-migration?
23:09:48 dansmith cfriesen: fix #1, skip duplicate instances as you are, and I just wrote a virt driver that depends on those refreshes to complete phases of my live migration.
23:12:09 cburgess melwitt Yeah ok thats what I thought.. superfluous... I'll keep digging into the hints thing but it really look to me like its redundant.
23:12:40 cfriesen dansmith: would a virt driver ever use InstanceInfoCache?
23:13:11 dansmith cfriesen: sure, the one I just wrote does
23:13:21 melwitt cburgess: yeah, I'm not seeing any clues as to why there are two options. maybe a legacy thing that used to have a purpose. jaypipes, bauzas, or edleafe might know
23:13:29 cfriesen :)
23:14:24 cfriesen dansmith: okay, so assuming we do the locking thing...do you think we would want to modify the standard lock (likely making it somewhat slower in the fastpath) or introduce a new "fair" lock and hope services switch to using it?
23:15:03 dansmith cfriesen: there's lots of network_info walking throughout the xen driver at least..
23:15:35 dansmith cfriesen: I have no idea what you want to do, and would suggest you ask the lockutils people what they think
23:16:25 cfriesen dansmith: I think it'd be nuts to rely on incoming network-changed events during a live migration since they can be processed in any order due to RPC and locking contention. But who knows what crazy stuff xen might do.
23:16:33 cfriesen dansmith: fair enough
23:17:02 dansmith if you're waiting for some property on a vif to become a value, I see no reason why that wouldn't be reasonable
23:17:21 dansmith anyway, EOD and EOP for me
23:17:33 cfriesen thanks for the feedback
23:24:39 mriedem SteelyDanHypervisor?
23:26:05 mriedem i'm going to need someone to run the nova team meeting tomorrow...
23:26:13 cfriesen 10thDanBlackBeltHypervisor
23:27:41 mriedem also, i wonder if dan is aware of dan's opinions on out of tree virt drivers
23:31:49 mriedem oomichi: were you interested in taking over either https://blueprints.launchpad.net/nova/+spec/live-migration-per-instance-timeout or https://blueprints.launchpad.net/nova/+spec/live-migration-force-after-timeout ?
23:51:07 openstackgerrit Ed Leafe proposed openstack/nova master: Remove selecting host based on host_subset_size https://review.openstack.org/465174
23:51:07 openstackgerrit Ed Leafe proposed openstack/nova master: Have scheduler claim resources https://review.openstack.org/463993
23:51:08 openstackgerrit Ed Leafe proposed openstack/nova master: Don't delete allocation if instance being scheduled https://review.openstack.org/460177
23:51:08 openstackgerrit Ed Leafe proposed openstack/nova master: Make the method to put allocations public https://review.openstack.org/464086
23:51:09 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - Pass the number of hosts to return explicitly https://review.openstack.org/465767
23:51:09 openstackgerrit Ed Leafe proposed openstack/nova master: Pass a list of instance UUIDs to scheduler https://review.openstack.org/465171
23:51:10 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - return multiple hosts from select_destinations https://review.openstack.org/465768
23:51:27 mriedem melwitt: do you want to tweak the reno at all in this? https://review.openstack.org/#/c/465042/
23:51:29 mriedem if not, i'm +2
23:52:48 melwitt mriedem: yeah, I can change it to your suggest and flip the test assertions. I wasn't thinking when I copied them
23:52:54 melwitt *suggestion
23:53:55 openstackgerrit Hongbin Lu proposed openstack/nova master: Handle exception on adding secgroup https://review.openstack.org/465173
23:55:31 cfriesen mriedem: so it looks like _build_network_info_model() costs about 200ms plus about 125ms per port since we query each port separatly from neutron. and the refresh_cache lock is held the whole time
23:56:18 cfriesen mriedem: I wonder if we could issues those requests in parallel for all the ports?
23:56:33 mriedem cfriesen: we've talked for years about needing to do bulk queries from neutron
23:57:45 mriedem well,
23:57:57 mriedem where do we re-query per port, after we get the list of ports from data = client.list_ports(**search_opts)
23:58:23 cfriesen _nw_info_get_ips and _nw_info_get_subnets both take noticeable time
23:58:32 cfriesen for each port
23:59:28 mriedem the first is going to iterate over all fixed_ips for a port and get floating IPs for each fixed_ip
#openstack-nova - 2017-05-18
00:01:32 mriedem i wonder if we could bulk query the floating IPs by the list of fixed IPs and the port_id
00:01:46 openstackgerrit melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042
00:02:07 mriedem "When you specify multiple filters, the Networking API v2.0 returns only objects that meet all filtering criteria. The operation applies an AND condition among the filters."
00:02:17 mriedem so that wouldn't really work,
00:02:27 mriedem because fixed_ip1 AND fixed_ip2 wouldn't resolve to a floating IP
00:02:45 cfriesen mriedem: we get an event for each vif, and we have to get info for all vifs on each event, so when we have N vifs, the live-migration overhead goes up as N^2
00:02:47 mriedem we could trim the fat on the fields we actually need to get back
00:03:07 cfriesen mriedem: it'd be better to have the vif specified in the network-changed event
00:03:24 cfriesen mriedem: and avoid having to scan all the vifs
00:03:37 mriedem cfriesen: yeah i'm not sure how feasible that is from the neutron side
00:03:46 mriedem i guess it would depend on how they calculate the network-changed event
00:03:52 edleafe I've written a post to the dev list about continuing the discussion on claims in the scheduler: http://lists.openstack.org/pipermail/openstack-dev/2017-May/116949.html
00:03:52 cfriesen mriedem: gotta run, family is calling for supper
00:04:00 edleafe ^^ cdent jaypipes dansmith bauzas johnthetubaguy
00:04:27 edleafe It has a link to my blog post and the patch series
00:04:36 mriedem edleafe: ack
00:04:37 edleafe And with that, I'm off to PyCon
00:04:42 mriedem have fun
00:04:46 edleafe See you all next week!
00:04:53 edleafe mriedem: oh, it's all work work work
00:15:21 mriedem cfriesen: here is something to look into later - the network-changed event in neutron does have a port in scope, it just doesn't put it in the external event request https://github.com/openstack/neutron/blob/master/neutron/notifiers/nova.py#L161
00:15:30 mriedem but we should be able to scope that event to the single vif
00:16:03 mriedem cfriesen: it would be helpful if we had a bug to track some of this
00:41:08 openstack Launchpad bug 1691602 in OpenStack Compute (nova) "live migration generates several network-changed events which lock up refreshing the nw info cache" [Undecided,New]
00:41:08 mriedem cfriesen: https://bugs.launchpad.net/neutron/+bug/1691602 for tracking
00:59:53 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add policy description for os-host https://review.openstack.org/465458
01:18:23 openstackgerrit Alex Xu proposed openstack/nova master: Limit the min length of string for integer JSON-Schema https://review.openstack.org/415103
01:21:53 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Catch exception.OverQuota when create image for volume backed instance https://review.openstack.org/464548
01:24:45 openstackgerrit Alex Xu proposed openstack/nova master: Limit the min length of string for integer JSON-Schema https://review.openstack.org/415103
01:36:36 dikonoor pumaranikar: Hi
01:39:37 dikonoor pumaranikar: I am trying to understand if this new support added is still experimental > https://github.com/openstack/nova/blob/master/nova/conf/service_token.py#L43
01:39:48 dikonoor Can it be turned on ?
01:40:47 mriedem dikonoor: yes it can be turned on for testing, we test with it on in the nova-next CI job
01:41:08 mriedem dikonoor: would actually appreciate someone to run with it in rally or longer-running operation scenarios like with live migration
01:41:18 mriedem and report back if it's helping
02:16:41 openstackgerrit Jay Pipes proposed openstack/nova master: Ensure image metadata when unshelving BFV instance https://review.openstack.org/347063
02:22:55 jaypipes alex_xu: looks sensible to me: https://review.openstack.org/#/c/353862/23
02:23:02 jaypipes alex_xu: XenAPI resource tracking fix.
02:34:46 openstackgerrit Alex Xu proposed openstack/nova master: Correct the error message for query parameter validation https://review.openstack.org/465785
02:35:28 openstackgerrit Tao Li proposed openstack/nova master: Fix the display of updated_at time when using memcache driver. https://review.openstack.org/414463
02:38:22 mriedem oh bug smashing glory
02:43:29 mriedem kevinbenton: looking at this https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L331

Earlier   Later