Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-21
20:57:03 mriedem i can fix it up
20:59:10 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Don't try to delete build request during a reschedule https://review.openstack.org/529697
21:14:57 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Don't try to delete build request during a reschedule https://review.openstack.org/529703
21:37:18 mriedem edleafe: got that rebase done locally, just running tests
21:39:17 edleafe k
21:56:04 openstackgerrit Merged openstack/nova master: conf: libvirt: Cleanup CPU modelling related options https://review.openstack.org/527691
22:04:55 mriedem here it comes
22:05:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
22:05:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't try to delete build request during a reschedule https://review.openstack.org/528835
22:05:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
22:09:45 mriedem edleafe: you might want to look this over quick https://review.openstack.org/#/c/511358/48..49/
22:09:48 mriedem make sure i didn't eff anything up
22:14:54 edleafe mriedem: looks sane
22:15:14 mriedem alright, bombs away
22:27:21 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608
22:27:47 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483
22:30:10 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: SEA https://review.openstack.org/523216
22:30:37 openstackgerrit Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094
22:33:08 openstackgerrit Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094
22:38:43 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.59 - List/Show all server migration types https://review.openstack.org/430839
23:13:42 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Fix parameter order in rebuild https://review.openstack.org/529718
23:15:21 Guest98938 Hello guys ..... Has anybody here seen an issue with nova-scheduler where max_instances_per_host host aggregate tag is not honored ?
23:16:01 Guest98938 I am seeing number of instances in those hosts go beyond max_instances_per_host limit set. This is on newton.
23:17:33 mriedem Guest98938: no, but it's also likely not tested very well and it's also likely racey
23:18:11 mriedem or if you bypass the scheduler by forcing a host during live migration or evacuate
23:18:29 Guest98938 I did further analysis on this and found stats in compute_node object to be empty, hence scheduler thinks num_instances to be 0 and schedules to this host
23:21:24 mriedem it doesn't look like stats is where the filter pulls that value from
23:21:43 mriedem it checks the HostState.instances value
23:21:59 mriedem *HostState.num_instances,
23:22:03 mriedem so maybe that comes from stats?
23:22:18 mriedem ah you're right
23:22:34 Guest98938 mriedem: Do you know of any condition which maybe causing this stats to be empty for a compute_node ? I see this happening every some time
23:22:53 mriedem not without digging
23:23:02 mriedem i also see that HostState._locked_consume_from_request overrides the num_instances value
23:23:29 mriedem there is also HostState.instances, which is a dict tracking the instances on that host
23:23:34 Guest98938 I basically have setup a heat autoscaling with cooldown of 60 secs, to continuously try to spawn instances
23:23:37 mriedem so apparently there are 3 different ways to screw this up
23:24:44 Guest98938 oh
23:24:52 mriedem idk this stats stuff all looks like garbage to me
23:25:53 Guest98938 yea I thought of a fix for this problem by relying on running_vms attribute rather than on stats(which is missing sometimes)
23:26:22 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/filters/num_instances_filter.py#L37
23:26:22 mriedem i would try this
23:26:32 mriedem num_instances = host_state.num_instances or len(host_state.instances)
23:27:00 mriedem or even just compare host_state.num_instances to len(host_state.instances) and if those are different, there is a problem somewhere and should probably log a warning
23:27:05 mriedem could help with your debug
23:30:13 Guest98938 yea that can help
23:31:25 Guest98938 or how about making scheduler to pick num_instances from compute_node.running_vms instead of self.stats.get('num_instances', 0) ?
23:31:58 mriedem ?
23:31:58 mriedem well, you could have stopped vms on a node right
23:32:22 mriedem although cn.running_vms = self.stats.num_instances
23:32:26 mriedem i guess those are the same
23:33:55 mriedem i would think that if stats.num_instances is wrong, that compute_node.running_vms will also be wrong
23:34:00 mriedem because the latter is set based on the former
23:34:13 Guest98938 so does running_vms include stopped vms ?
23:34:20 mriedem yeah i think so
23:34:49 Guest98938 yea, I think the problem here is self.stats are missing and compute_node.running_vms exists from my observation
23:35:24 Guest98938 which is why I though of relying on running_vms
23:37:33 mriedem it's possible there could be a problem where stats.num_instances doesn't get updated properly for a move operation
23:37:37 mriedem i.e. a migration to another host
23:39:48 mriedem this change might have regressed something in newton
23:39:49 mriedem https://review.openstack.org/#/c/307124/21/nova/compute/resource_tracker.py@245
23:39:52 mriedem see how that comment is wrong
23:40:02 mriedem it might have just meant _update_usage_from_migration
23:40:04 mriedem and was a typo
23:40:33 mriedem i think that's a typo,
23:40:42 mriedem but i also think that a move operation isn't properly updating stats.num_instances
23:41:35 mriedem the update_available_resource periodic task which runs by default every minute should update stats.num_instances for the instances that currently live on that host,
23:41:50 mriedem but you could hit a window between the periodic task running and a move happening
23:41:55 mriedem which throws off your filter
23:43:43 Guest98938 yea, but in my case there is no migration happening
23:50:09 mriedem ok, idk then
23:50:19 mriedem i'd still check this in the filter
23:50:20 mriedem num_instances = host_state.num_instances or len(host_state.instances)
23:50:37 mriedem if those 2 values on the right are different, i'd log a warning
23:50:59 tonyb mriedem: followign up from the meetign I just missed ... I'll do the newton EOL in Jan
23:51:07 mriedem tonyb: ok
23:57:02 Guest98938 yea, I will try the filter approach too, thanks
#openstack-nova - 2017-12-22
00:06:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix functional tests for USE_NEUTRON https://review.openstack.org/529456
00:12:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix functional tests for USE_NEUTRON https://review.openstack.org/529456
00:59:59 yikun https://launchpad.net/bugs/1739571 and https://review.openstack.org/#/c/529519/
01:00:00 openstack Launchpad bug 1739571 in OpenStack Compute (nova) "Can't delete root resource provider because of the self foreign key" [Undecided,In progress] - Assigned to Yikun Jiang (yikunkero)
01:00:03 yikun hello, morning or evening, ^^^ I found a bug, but this bug only reproduced in MySQL environment.
01:00:08 yikun but seems our testcases only run in SQLite, so, is there some apporach to test it?
01:44:48 openstackgerrit Minho Ban proposed openstack/nova master: prevent removing numa_topology usage info from DB https://review.openstack.org/529236
02:06:49 alex_xu mriedem: oomichi do you like using the 'controller' instead of 'extension'?
02:31:18 mriedem alex_xu: i'll do either
02:32:04 mriedem sec
02:32:04 mriedem yikun: there is a trick to make foreign keys work with sqlite,
02:33:07 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Fix error when deleting any root resource provider https://review.openstack.org/529519
02:33:12 mriedem sqlite 3.7+ should work with foreign keys
02:33:13 mriedem see TestDatabaseArchive
02:33:36 mriedem engine.connect().execute("PRAGMA foreign_keys = ON")
02:33:48 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Test allocation candidates: multiple aggregates https://review.openstack.org/518633
02:33:49 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix accumulated nits in refactor series https://review.openstack.org/521189
02:33:49 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Test helper: validate provider summaries https://review.openstack.org/518982
02:33:50 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix missing rps in allocation candidates https://review.openstack.org/522409
02:33:50 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Add aggregates check in allocation candidates https://review.openstack.org/522407
02:33:51 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: placement: func tests for multiple shared RPs https://review.openstack.org/498737
02:34:03 yikun o? I will take a look, and try it in my local env, thanks. : )
02:34:08 yikun mriedem,

Earlier   Later