| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-21 | |||
| 19:44:13 | mriedem | :) | |
| 19:44:36 | mriedem | oomichi: ok so i'll change the 'debug' part | |
| 19:44:37 | mriedem | and leave the rest | |
| 19:44:46 | oomichi | hehe | |
| 19:45:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Document testing guide for new API contributions https://review.openstack.org/529618 | |
| 19:46:14 | mriedem | https://docs.openstack.org/nova/latest/contributor/api.html#modularity | |
| 19:46:19 | mriedem | "The Nova REST API is separated into different controllers in the directory ‘nova/api/openstack/compute/’" | |
| 19:46:29 | openstackgerrit | Merged openstack/nova stable/pike: [placement] Fix an error message in API validation https://review.openstack.org/526465 | |
| 19:46:30 | mriedem | since the doc already says "controllers" maybe we should just be consistent? | |
| 19:50:14 | melwitt | I was originally going to put the noop quotas directory under os-quota-sets/ but thought maybe the top level one was less confusing. I can try to move it under os-quota-sets/ | |
| 19:52:32 | melwitt | another way I tried was putting the noop samples at the same level in os-quota-sets/ without their own directory and have the noop tests look for a noop prefix, but thought maybe that was worse ¯\_(ツ)_/¯ | |
| 20:00:21 | mriedem | i was just giving you shit about that | |
| 20:03:51 | melwitt | k :) | |
| 20:39:07 | openstackgerrit | Merged openstack/nova master: Pass RequestSpec to ConductorTaskAPI.build_instances https://review.openstack.org/515495 | |
| 20:41:12 | cfriesen | mriedem: do you know why we're calling self._update() at https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L590 ? Having that means that every time we call_update_available_resource() we'll save invalid data, then overwrite it a bit later with valid data. | |
| 20:42:30 | mriedem | cfriesen: oh boy | |
| 20:42:39 | mriedem | you don't know?! | |
| 20:42:47 | mriedem | cfriesen: https://review.openstack.org/#/c/520024/ | |
| 20:43:04 | mriedem | feel free to update that change | |
| 20:43:08 | mriedem | maciejjozefczyk: ^ | |
| 20:43:17 | mriedem | nova meeting in 17 minutes | |
| 20:43:20 | mriedem | last one for the year | |
| 20:43:35 | cfriesen | mriedem: sweet, thanks | |
| 20:49:44 | openstackgerrit | Merged openstack/nova master: Fix an error in _get_host_states when deleting a compute node https://review.openstack.org/529352 | |
| 20:49:51 | openstackgerrit | Merged openstack/nova master: VMware: expose max vCPUs and max memory per ESX host https://review.openstack.org/516635 | |
| 20:49:58 | openstackgerrit | Merged openstack/nova master: VMware: implement get_inventory() driver method https://review.openstack.org/506175 | |
| 20:50:09 | openstackgerrit | Merged openstack/nova master: Pass mountpoint to volume attachment_create with connector https://review.openstack.org/527479 | |
| 20:50:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use volume shared_targets to lock during attach/detach https://review.openstack.org/529695 | |
| 20:50:24 | mriedem | ildikov: jgriffith: ^ | |
| 20:50:53 | ildikov | mriedem: thanks! | |
| 20:51:46 | mriedem | edleafe: you might hit a merge conflict now that https://review.openstack.org/#/c/515495/ merged | |
| 20:56:03 | jgriffith | mriedem: ack, I'll check it out here shortly. Thanks! | |
| 20:56:07 | edleafe | mriedem: oh joy | |
| 20:56:39 | mriedem | edleafe: maybe, but not sure | |
| 20:56:58 | mriedem | eh, yeah it's in merge conflict | |
| 20:57:03 | mriedem | i can fix it up | |
| 20:59:10 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Don't try to delete build request during a reschedule https://review.openstack.org/529697 | |
| 21:14:57 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Don't try to delete build request during a reschedule https://review.openstack.org/529703 | |
| 21:37:18 | mriedem | edleafe: got that rebase done locally, just running tests | |
| 21:39:17 | edleafe | k | |
| 21:56:04 | openstackgerrit | Merged openstack/nova master: conf: libvirt: Cleanup CPU modelling related options https://review.openstack.org/527691 | |
| 22:04:55 | mriedem | here it comes | |
| 22:05:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't try to delete build request during a reschedule https://review.openstack.org/528835 | |
| 22:05:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 22:05:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 22:09:45 | mriedem | edleafe: you might want to look this over quick https://review.openstack.org/#/c/511358/48..49/ | |
| 22:09:48 | mriedem | make sure i didn't eff anything up | |
| 22:14:54 | edleafe | mriedem: looks sane | |
| 22:15:14 | mriedem | alright, bombs away | |
| 22:27:21 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608 | |
| 22:27:47 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 22:30:10 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: SEA https://review.openstack.org/523216 | |
| 22:30:37 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 22:33:08 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 22:38:43 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.59 - List/Show all server migration types https://review.openstack.org/430839 | |
| 23:13:42 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Fix parameter order in rebuild https://review.openstack.org/529718 | |
| 23:15:21 | Guest98938 | Hello guys ..... Has anybody here seen an issue with nova-scheduler where max_instances_per_host host aggregate tag is not honored ? | |
| 23:16:01 | Guest98938 | I am seeing number of instances in those hosts go beyond max_instances_per_host limit set. This is on newton. | |
| 23:17:33 | mriedem | Guest98938: no, but it's also likely not tested very well and it's also likely racey | |
| 23:18:11 | mriedem | or if you bypass the scheduler by forcing a host during live migration or evacuate | |
| 23:18:29 | Guest98938 | I did further analysis on this and found stats in compute_node object to be empty, hence scheduler thinks num_instances to be 0 and schedules to this host | |
| 23:21:24 | mriedem | it doesn't look like stats is where the filter pulls that value from | |
| 23:21:43 | mriedem | it checks the HostState.instances value | |
| 23:21:59 | mriedem | *HostState.num_instances, | |
| 23:22:03 | mriedem | so maybe that comes from stats? | |
| 23:22:18 | mriedem | ah you're right | |
| 23:22:34 | Guest98938 | mriedem: Do you know of any condition which maybe causing this stats to be empty for a compute_node ? I see this happening every some time | |
| 23:22:53 | mriedem | not without digging | |
| 23:23:02 | mriedem | i also see that HostState._locked_consume_from_request overrides the num_instances value | |
| 23:23:29 | mriedem | there is also HostState.instances, which is a dict tracking the instances on that host | |
| 23:23:34 | Guest98938 | I basically have setup a heat autoscaling with cooldown of 60 secs, to continuously try to spawn instances | |
| 23:23:37 | mriedem | so apparently there are 3 different ways to screw this up | |
| 23:24:44 | Guest98938 | oh | |
| 23:24:52 | mriedem | idk this stats stuff all looks like garbage to me | |
| 23:25:53 | Guest98938 | yea I thought of a fix for this problem by relying on running_vms attribute rather than on stats(which is missing sometimes) | |
| 23:26:22 | mriedem | i would try this | |
| 23:26:22 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/num_instances_filter.py#L37 | |
| 23:26:32 | mriedem | num_instances = host_state.num_instances or len(host_state.instances) | |
| 23:27:00 | mriedem | or even just compare host_state.num_instances to len(host_state.instances) and if those are different, there is a problem somewhere and should probably log a warning | |
| 23:27:05 | mriedem | could help with your debug | |
| 23:30:13 | Guest98938 | yea that can help | |
| 23:31:25 | Guest98938 | or how about making scheduler to pick num_instances from compute_node.running_vms instead of self.stats.get('num_instances', 0) ? | |
| 23:31:58 | mriedem | well, you could have stopped vms on a node right | |
| 23:31:58 | mriedem | ? | |
| 23:32:22 | mriedem | although cn.running_vms = self.stats.num_instances | |
| 23:32:26 | mriedem | i guess those are the same | |
| 23:33:55 | mriedem | i would think that if stats.num_instances is wrong, that compute_node.running_vms will also be wrong | |
| 23:34:00 | mriedem | because the latter is set based on the former | |
| 23:34:13 | Guest98938 | so does running_vms include stopped vms ? | |
| 23:34:20 | mriedem | yeah i think so | |
| 23:34:49 | Guest98938 | yea, I think the problem here is self.stats are missing and compute_node.running_vms exists from my observation | |
| 23:35:24 | Guest98938 | which is why I though of relying on running_vms | |
| 23:37:33 | mriedem | it's possible there could be a problem where stats.num_instances doesn't get updated properly for a move operation | |
| 23:37:37 | mriedem | i.e. a migration to another host | |
| 23:39:48 | mriedem | this change might have regressed something in newton | |
| 23:39:49 | mriedem | https://review.openstack.org/#/c/307124/21/nova/compute/resource_tracker.py@245 | |
| 23:39:52 | mriedem | see how that comment is wrong | |
| 23:40:02 | mriedem | it might have just meant _update_usage_from_migration | |
| 23:40:04 | mriedem | and was a typo | |
| 23:40:33 | mriedem | i think that's a typo, | |