| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-21 | |||
| 20:03:51 | melwitt | k :) | |
| 20:39:07 | openstackgerrit | Merged openstack/nova master: Pass RequestSpec to ConductorTaskAPI.build_instances https://review.openstack.org/515495 | |
| 20:41:12 | cfriesen | mriedem: do you know why we're calling self._update() at https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L590 ? Having that means that every time we call_update_available_resource() we'll save invalid data, then overwrite it a bit later with valid data. | |
| 20:42:30 | mriedem | cfriesen: oh boy | |
| 20:42:39 | mriedem | you don't know?! | |
| 20:42:47 | mriedem | cfriesen: https://review.openstack.org/#/c/520024/ | |
| 20:43:04 | mriedem | feel free to update that change | |
| 20:43:08 | mriedem | maciejjozefczyk: ^ | |
| 20:43:17 | mriedem | nova meeting in 17 minutes | |
| 20:43:20 | mriedem | last one for the year | |
| 20:43:35 | cfriesen | mriedem: sweet, thanks | |
| 20:49:44 | openstackgerrit | Merged openstack/nova master: Fix an error in _get_host_states when deleting a compute node https://review.openstack.org/529352 | |
| 20:49:51 | openstackgerrit | Merged openstack/nova master: VMware: expose max vCPUs and max memory per ESX host https://review.openstack.org/516635 | |
| 20:49:58 | openstackgerrit | Merged openstack/nova master: VMware: implement get_inventory() driver method https://review.openstack.org/506175 | |
| 20:50:09 | openstackgerrit | Merged openstack/nova master: Pass mountpoint to volume attachment_create with connector https://review.openstack.org/527479 | |
| 20:50:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use volume shared_targets to lock during attach/detach https://review.openstack.org/529695 | |
| 20:50:24 | mriedem | ildikov: jgriffith: ^ | |
| 20:50:53 | ildikov | mriedem: thanks! | |
| 20:51:46 | mriedem | edleafe: you might hit a merge conflict now that https://review.openstack.org/#/c/515495/ merged | |
| 20:56:03 | jgriffith | mriedem: ack, I'll check it out here shortly. Thanks! | |
| 20:56:07 | edleafe | mriedem: oh joy | |
| 20:56:39 | mriedem | edleafe: maybe, but not sure | |
| 20:56:58 | mriedem | eh, yeah it's in merge conflict | |
| 20:57:03 | mriedem | i can fix it up | |
| 20:59:10 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Don't try to delete build request during a reschedule https://review.openstack.org/529697 | |
| 21:14:57 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Don't try to delete build request during a reschedule https://review.openstack.org/529703 | |
| 21:37:18 | mriedem | edleafe: got that rebase done locally, just running tests | |
| 21:39:17 | edleafe | k | |
| 21:56:04 | openstackgerrit | Merged openstack/nova master: conf: libvirt: Cleanup CPU modelling related options https://review.openstack.org/527691 | |
| 22:04:55 | mriedem | here it comes | |
| 22:05:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't try to delete build request during a reschedule https://review.openstack.org/528835 | |
| 22:05:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 22:05:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 22:09:45 | mriedem | edleafe: you might want to look this over quick https://review.openstack.org/#/c/511358/48..49/ | |
| 22:09:48 | mriedem | make sure i didn't eff anything up | |
| 22:14:54 | edleafe | mriedem: looks sane | |
| 22:15:14 | mriedem | alright, bombs away | |
| 22:27:21 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608 | |
| 22:27:47 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 22:30:10 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: SEA https://review.openstack.org/523216 | |
| 22:30:37 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 22:33:08 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 22:38:43 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.59 - List/Show all server migration types https://review.openstack.org/430839 | |
| 23:13:42 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Fix parameter order in rebuild https://review.openstack.org/529718 | |
| 23:15:21 | Guest98938 | Hello guys ..... Has anybody here seen an issue with nova-scheduler where max_instances_per_host host aggregate tag is not honored ? | |
| 23:16:01 | Guest98938 | I am seeing number of instances in those hosts go beyond max_instances_per_host limit set. This is on newton. | |
| 23:17:33 | mriedem | Guest98938: no, but it's also likely not tested very well and it's also likely racey | |
| 23:18:11 | mriedem | or if you bypass the scheduler by forcing a host during live migration or evacuate | |
| 23:18:29 | Guest98938 | I did further analysis on this and found stats in compute_node object to be empty, hence scheduler thinks num_instances to be 0 and schedules to this host | |
| 23:21:24 | mriedem | it doesn't look like stats is where the filter pulls that value from | |
| 23:21:43 | mriedem | it checks the HostState.instances value | |
| 23:21:59 | mriedem | *HostState.num_instances, | |
| 23:22:03 | mriedem | so maybe that comes from stats? | |
| 23:22:18 | mriedem | ah you're right | |
| 23:22:34 | Guest98938 | mriedem: Do you know of any condition which maybe causing this stats to be empty for a compute_node ? I see this happening every some time | |
| 23:22:53 | mriedem | not without digging | |
| 23:23:02 | mriedem | i also see that HostState._locked_consume_from_request overrides the num_instances value | |
| 23:23:29 | mriedem | there is also HostState.instances, which is a dict tracking the instances on that host | |
| 23:23:34 | Guest98938 | I basically have setup a heat autoscaling with cooldown of 60 secs, to continuously try to spawn instances | |
| 23:23:37 | mriedem | so apparently there are 3 different ways to screw this up | |
| 23:24:44 | Guest98938 | oh | |
| 23:24:52 | mriedem | idk this stats stuff all looks like garbage to me | |
| 23:25:53 | Guest98938 | yea I thought of a fix for this problem by relying on running_vms attribute rather than on stats(which is missing sometimes) | |
| 23:26:22 | mriedem | i would try this | |
| 23:26:22 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/num_instances_filter.py#L37 | |
| 23:26:32 | mriedem | num_instances = host_state.num_instances or len(host_state.instances) | |
| 23:27:00 | mriedem | or even just compare host_state.num_instances to len(host_state.instances) and if those are different, there is a problem somewhere and should probably log a warning | |
| 23:27:05 | mriedem | could help with your debug | |
| 23:30:13 | Guest98938 | yea that can help | |
| 23:31:25 | Guest98938 | or how about making scheduler to pick num_instances from compute_node.running_vms instead of self.stats.get('num_instances', 0) ? | |
| 23:31:58 | mriedem | well, you could have stopped vms on a node right | |
| 23:31:58 | mriedem | ? | |
| 23:32:22 | mriedem | although cn.running_vms = self.stats.num_instances | |
| 23:32:26 | mriedem | i guess those are the same | |
| 23:33:55 | mriedem | i would think that if stats.num_instances is wrong, that compute_node.running_vms will also be wrong | |
| 23:34:00 | mriedem | because the latter is set based on the former | |
| 23:34:13 | Guest98938 | so does running_vms include stopped vms ? | |
| 23:34:20 | mriedem | yeah i think so | |
| 23:34:49 | Guest98938 | yea, I think the problem here is self.stats are missing and compute_node.running_vms exists from my observation | |
| 23:35:24 | Guest98938 | which is why I though of relying on running_vms | |
| 23:37:33 | mriedem | it's possible there could be a problem where stats.num_instances doesn't get updated properly for a move operation | |
| 23:37:37 | mriedem | i.e. a migration to another host | |
| 23:39:48 | mriedem | this change might have regressed something in newton | |
| 23:39:49 | mriedem | https://review.openstack.org/#/c/307124/21/nova/compute/resource_tracker.py@245 | |
| 23:39:52 | mriedem | see how that comment is wrong | |
| 23:40:02 | mriedem | it might have just meant _update_usage_from_migration | |
| 23:40:04 | mriedem | and was a typo | |
| 23:40:33 | mriedem | i think that's a typo, | |
| 23:40:42 | mriedem | but i also think that a move operation isn't properly updating stats.num_instances | |
| 23:41:35 | mriedem | the update_available_resource periodic task which runs by default every minute should update stats.num_instances for the instances that currently live on that host, | |
| 23:41:50 | mriedem | but you could hit a window between the periodic task running and a move happening | |
| 23:41:55 | mriedem | which throws off your filter | |
| 23:43:43 | Guest98938 | yea, but in my case there is no migration happening | |
| 23:50:09 | mriedem | ok, idk then | |
| 23:50:19 | mriedem | i'd still check this in the filter | |
| 23:50:20 | mriedem | num_instances = host_state.num_instances or len(host_state.instances) | |
| 23:50:37 | mriedem | if those 2 values on the right are different, i'd log a warning | |
| 23:50:59 | tonyb | mriedem: followign up from the meetign I just missed ... I'll do the newton EOL in Jan | |
| 23:51:07 | mriedem | tonyb: ok | |
| 23:57:02 | Guest98938 | yea, I will try the filter approach too, thanks | |