| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-21 | |||
| 22:15:14 | mriedem | alright, bombs away | |
| 22:27:21 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608 | |
| 22:27:47 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 22:30:10 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: SEA https://review.openstack.org/523216 | |
| 22:30:37 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 22:33:08 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 22:38:43 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.59 - List/Show all server migration types https://review.openstack.org/430839 | |
| 23:13:42 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Fix parameter order in rebuild https://review.openstack.org/529718 | |
| 23:15:21 | Guest98938 | Hello guys ..... Has anybody here seen an issue with nova-scheduler where max_instances_per_host host aggregate tag is not honored ? | |
| 23:16:01 | Guest98938 | I am seeing number of instances in those hosts go beyond max_instances_per_host limit set. This is on newton. | |
| 23:17:33 | mriedem | Guest98938: no, but it's also likely not tested very well and it's also likely racey | |
| 23:18:11 | mriedem | or if you bypass the scheduler by forcing a host during live migration or evacuate | |
| 23:18:29 | Guest98938 | I did further analysis on this and found stats in compute_node object to be empty, hence scheduler thinks num_instances to be 0 and schedules to this host | |
| 23:21:24 | mriedem | it doesn't look like stats is where the filter pulls that value from | |
| 23:21:43 | mriedem | it checks the HostState.instances value | |
| 23:21:59 | mriedem | *HostState.num_instances, | |
| 23:22:03 | mriedem | so maybe that comes from stats? | |
| 23:22:18 | mriedem | ah you're right | |
| 23:22:34 | Guest98938 | mriedem: Do you know of any condition which maybe causing this stats to be empty for a compute_node ? I see this happening every some time | |
| 23:22:53 | mriedem | not without digging | |
| 23:23:02 | mriedem | i also see that HostState._locked_consume_from_request overrides the num_instances value | |
| 23:23:29 | mriedem | there is also HostState.instances, which is a dict tracking the instances on that host | |
| 23:23:34 | Guest98938 | I basically have setup a heat autoscaling with cooldown of 60 secs, to continuously try to spawn instances | |
| 23:23:37 | mriedem | so apparently there are 3 different ways to screw this up | |
| 23:24:44 | Guest98938 | oh | |
| 23:24:52 | mriedem | idk this stats stuff all looks like garbage to me | |
| 23:25:53 | Guest98938 | yea I thought of a fix for this problem by relying on running_vms attribute rather than on stats(which is missing sometimes) | |
| 23:26:22 | mriedem | i would try this | |
| 23:26:22 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/num_instances_filter.py#L37 | |
| 23:26:32 | mriedem | num_instances = host_state.num_instances or len(host_state.instances) | |
| 23:27:00 | mriedem | or even just compare host_state.num_instances to len(host_state.instances) and if those are different, there is a problem somewhere and should probably log a warning | |
| 23:27:05 | mriedem | could help with your debug | |
| 23:30:13 | Guest98938 | yea that can help | |
| 23:31:25 | Guest98938 | or how about making scheduler to pick num_instances from compute_node.running_vms instead of self.stats.get('num_instances', 0) ? | |
| 23:31:58 | mriedem | well, you could have stopped vms on a node right | |
| 23:31:58 | mriedem | ? | |
| 23:32:22 | mriedem | although cn.running_vms = self.stats.num_instances | |
| 23:32:26 | mriedem | i guess those are the same | |
| 23:33:55 | mriedem | i would think that if stats.num_instances is wrong, that compute_node.running_vms will also be wrong | |
| 23:34:00 | mriedem | because the latter is set based on the former | |
| 23:34:13 | Guest98938 | so does running_vms include stopped vms ? | |
| 23:34:20 | mriedem | yeah i think so | |
| 23:34:49 | Guest98938 | yea, I think the problem here is self.stats are missing and compute_node.running_vms exists from my observation | |
| 23:35:24 | Guest98938 | which is why I though of relying on running_vms | |
| 23:37:33 | mriedem | it's possible there could be a problem where stats.num_instances doesn't get updated properly for a move operation | |
| 23:37:37 | mriedem | i.e. a migration to another host | |
| 23:39:48 | mriedem | this change might have regressed something in newton | |
| 23:39:49 | mriedem | https://review.openstack.org/#/c/307124/21/nova/compute/resource_tracker.py@245 | |
| 23:39:52 | mriedem | see how that comment is wrong | |
| 23:40:02 | mriedem | it might have just meant _update_usage_from_migration | |
| 23:40:04 | mriedem | and was a typo | |
| 23:40:33 | mriedem | i think that's a typo, | |
| 23:40:42 | mriedem | but i also think that a move operation isn't properly updating stats.num_instances | |
| 23:41:35 | mriedem | the update_available_resource periodic task which runs by default every minute should update stats.num_instances for the instances that currently live on that host, | |
| 23:41:50 | mriedem | but you could hit a window between the periodic task running and a move happening | |
| 23:41:55 | mriedem | which throws off your filter | |
| 23:43:43 | Guest98938 | yea, but in my case there is no migration happening | |
| 23:50:09 | mriedem | ok, idk then | |
| 23:50:19 | mriedem | i'd still check this in the filter | |
| 23:50:20 | mriedem | num_instances = host_state.num_instances or len(host_state.instances) | |
| 23:50:37 | mriedem | if those 2 values on the right are different, i'd log a warning | |
| 23:50:59 | tonyb | mriedem: followign up from the meetign I just missed ... I'll do the newton EOL in Jan | |
| 23:51:07 | mriedem | tonyb: ok | |
| 23:57:02 | Guest98938 | yea, I will try the filter approach too, thanks | |
| #openstack-nova - 2017-12-22 | |||
| 00:06:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix functional tests for USE_NEUTRON https://review.openstack.org/529456 | |
| 00:12:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix functional tests for USE_NEUTRON https://review.openstack.org/529456 | |
| 00:59:59 | yikun | https://launchpad.net/bugs/1739571 and https://review.openstack.org/#/c/529519/ | |
| 01:00:00 | openstack | Launchpad bug 1739571 in OpenStack Compute (nova) "Can't delete root resource provider because of the self foreign key" [Undecided,In progress] - Assigned to Yikun Jiang (yikunkero) | |
| 01:00:03 | yikun | hello, morning or evening, ^^^ I found a bug, but this bug only reproduced in MySQL environment. | |
| 01:00:08 | yikun | but seems our testcases only run in SQLite, so, is there some apporach to test it? | |
| 01:44:48 | openstackgerrit | Minho Ban proposed openstack/nova master: prevent removing numa_topology usage info from DB https://review.openstack.org/529236 | |
| 02:06:49 | alex_xu | mriedem: oomichi do you like using the 'controller' instead of 'extension'? | |
| 02:31:18 | mriedem | alex_xu: i'll do either | |
| 02:32:04 | mriedem | yikun: there is a trick to make foreign keys work with sqlite, | |
| 02:32:04 | mriedem | sec | |
| 02:33:07 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Fix error when deleting any root resource provider https://review.openstack.org/529519 | |
| 02:33:12 | mriedem | sqlite 3.7+ should work with foreign keys | |
| 02:33:13 | mriedem | see TestDatabaseArchive | |
| 02:33:36 | mriedem | engine.connect().execute("PRAGMA foreign_keys = ON") | |
| 02:33:48 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Test allocation candidates: multiple aggregates https://review.openstack.org/518633 | |
| 02:33:49 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Test helper: validate provider summaries https://review.openstack.org/518982 | |
| 02:33:49 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix accumulated nits in refactor series https://review.openstack.org/521189 | |
| 02:33:50 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add aggregates check in allocation candidates https://review.openstack.org/522407 | |
| 02:33:50 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix missing rps in allocation candidates https://review.openstack.org/522409 | |
| 02:33:51 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: placement: func tests for multiple shared RPs https://review.openstack.org/498737 | |
| 02:34:03 | yikun | o? I will take a look, and try it in my local env, thanks. : ) | |
| 02:34:08 | yikun | mriedem, | |
| 02:34:10 | mriedem | yw | |
| 02:38:33 | mriedem | yikun: fyi https://review.openstack.org/#/c/529672/ | |
| 02:38:37 | mriedem | turns out we needed that after all | |
| 02:42:33 | yikun | mriedem, emm, yes, +1, the deadlock on action event start is possible, | |
| 02:42:40 | yikun | because action and event doesn't have some relation ship, so sqlalchemy session mechanism can't ensure the update order. | |
| 02:43:57 | yikun | some ref: http://www.aosabook.org/en/sqlalchemy.html # 20.9. Unit of Work | |
| 02:45:16 | Kevin_Zheng | consider add relationship? | |
| 02:45:34 | openstackgerrit | Merged openstack/nova master: Fix unit test failures when direct IO not supported https://review.openstack.org/529568 | |
| 02:46:00 | Kevin_Zheng | mriedem we were thinking about a full scan about these kind of things in Nova | |
| 02:46:37 | Kevin_Zheng | https://review.openstack.org/#/c/527362/ | |
| 02:47:33 | Kevin_Zheng | ^ the deadlock retry method is not perfect, so we might be able to improve performance here | |
| 02:58:45 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Test allocation candidates: multiple aggregates https://review.openstack.org/518633 | |
| 02:58:46 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Test helper: validate provider summaries https://review.openstack.org/518982 | |