| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-10 | |||
| 15:39:42 | efried | Okay, well, keep me posted. I'll be interested in contributing to (or at least reviewing) the code if we go there. | |
| 15:41:13 | mriedem | cdent: ha "After that every 60s or so, five requests are made:" | |
| 15:41:14 | mriedem | right on | |
| 15:41:39 | mriedem | tracking here btw https://bugs.launchpad.net/nova/+bug/1742467 | |
| 15:41:41 | openstack | Launchpad bug 1742467 in OpenStack Compute (nova) "Compute unnecessarily gets resource provider aggregates during every update_available_resource run" [Undecided,New] | |
| 15:43:06 | cdent | mriedem: in your thinking just now did you get any clearer picture on the why of double inventory GET? | |
| 15:43:11 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver https://review.openstack.org/532287 | |
| 15:43:11 | openstackgerrit | Mark Goddard proposed openstack/nova master: WIP: Send traits to ironic on server boot https://review.openstack.org/508116 | |
| 15:43:12 | openstackgerrit | Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288 | |
| 15:43:12 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add has_any_traits() to provider tree https://review.openstack.org/532289 | |
| 15:43:13 | openstackgerrit | Mark Goddard proposed openstack/nova master: Call get_traits() in the resource tracker https://review.openstack.org/532290 | |
| 15:43:13 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add support to scheduler client for setting traits https://review.openstack.org/532539 | |
| 15:43:59 | mriedem | cdent: yes it's the RT | |
| 15:44:09 | mriedem | _update_available_resource is the call from the compute periodic task, | |
| 15:44:13 | mriedem | which calls _init_compute_node | |
| 15:44:23 | mriedem | when we already have the compute node, it calls _update | |
| 15:44:32 | mriedem | which eventually does the update_inventory_attempt stuff in the report client | |
| 15:44:42 | mriedem | then at the end of _update_available_resource, | |
| 15:44:44 | mriedem | we call _update again | |
| 15:44:53 | mriedem | so that's your 2 inventory updates | |
| 15:45:06 | mriedem | which, johnthetubaguy changed in queens | |
| 15:45:16 | mriedem | or wait,no | |
| 15:45:37 | mriedem | https://review.openstack.org/#/c/520024/ | |
| 15:45:44 | mriedem | that would fix the double GET inventories | |
| 15:48:00 | mriedem | maciejjozefczyk: have you figured out anything more about https://review.openstack.org/#/c/520024/ ? | |
| 15:49:25 | maciejjozefczyk | mriedem: hah! | |
| 15:49:32 | maciejjozefczyk | mriedem: aready working on this | |
| 15:49:51 | maciejjozefczyk | and yes, I found something strange, but I need big prove about it | |
| 15:49:57 | maciejjozefczyk | I'll post it today in review | |
| 15:52:01 | maciejjozefczyk | mriedem: basically: each time self._provider_tree.has_inventory_changed() returns False here: | |
| 15:52:02 | maciejjozefczyk | https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L696 | |
| 15:52:10 | maciejjozefczyk | without my change | |
| 15:53:00 | maciejjozefczyk | with my change: It tries to update inventory_data to placement and then placement raises this strange Exception | |
| 15:54:09 | maciejjozefczyk | so for now in upstream we update only DB (once with faulty values, second time with proper ones) | |
| 15:54:53 | mriedem | so it's failing this check? https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L231 | |
| 15:55:22 | maciejjozefczyk | mriedem: exactly | |
| 15:56:48 | mriedem | do you see this debug message with the PUT inventory request body in it when this fails? https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L765 | |
| 15:57:43 | maciejjozefczyk | mriedem: yes | |
| 15:57:49 | maciejjozefczyk | WARNING nova.scheduler.client.report [None req-89e951da-7283-473f-9de0-53854741839a None None] [req-b6f884cd-883a-4964-8e99-a8f0 f754c0df] Failed to update inventory for resource provider 52559824-5fb1-424b-a4cf-79da9199447d: 400 {"errors": [{"status": 400, "request_id": "req-b6f884cd-883a-4964-8e99-a8f0f754c0df", "detail": "The server could not comply with the request since it is e ither | |
| 15:57:51 | maciejjozefczyk | malformed or otherwise incorrect.\n\n Unable to update inventory for resource provider 52559824-5fb1-424b-a4cf-79da9199447 d: Invalid inventory for 'VCPU' on resource provider '52559824-5fb1-424b-a4cf-79da9199447d'. The reserved value is greater than or equal to total. ", "title": "Bad Request"}]} | |
| 15:58:46 | mriedem | that's the warning, do you have this debug log? https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L765 | |
| 15:59:12 | mriedem | i want to see if the request body has "The reserved value is greater than or equal to total." in it | |
| 15:59:26 | cdent | greater than empty value? | |
| 15:59:33 | maciejjozefczyk | mriedem: checking | |
| 16:02:23 | mriedem | cdent: so, we can likely at least turn down the 5 calls per periodic to 2 if we nix the 2 aggregate calls and turn the 2 inventory calls to 1; and then i think the GET /allocations in here is only if you have ocata computes or are using the ironic driver | |
| 16:02:43 | cdent | that makes sense | |
| 16:05:26 | melwitt | lyarwood: thanks for the stable/pike reviews, could you please hit these stable/ocata versions too? https://review.openstack.org/531422 and https://review.openstack.org/#/c/523911 | |
| 16:06:53 | lyarwood | melwitt: ack np, I'll get to them tonight | |
| 16:07:16 | melwitt | thanks | |
| 16:09:29 | maciejjozefczyk | mriedem: debug log: DEBUG nova.scheduler.client.report [None req-ad586aa8-27d1-494d-9c4d-bb8f15439fca None None] [req-a4ea518f-1da0-43fd-8348-64704210cb49] Failed inventory update request for resource provider 52559824-5fb1-424b-a4cf-79da9199447d with body: {'resource_provider_generation': 4, 'inventories': {'VCPU': {'allocation_ratio': 0.0, 'total': 2, 'reserved': 0, 'step_size': 1, 'min_unit': 1, | |
| 16:09:31 | maciejjozefczyk | 'max_unit': 2}, 'MEMORY_MB': {'allocation_ratio': 0.0, 'total': 29449, 'reserved': 512, 'ste p_size': 1, 'min_unit': 1, 'max_unit': 29449}, 'DISK_GB': {'allocation_ratio': 0.0, 'total': 193, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 193}}} {{(pid=19609) _update_inventory_attempt /opt/stack/nova/nova/scheduler/cli ent/report.py:765}} | |
| 16:09:55 | cdent | allocation_ratio being 0 is not supposed to happen | |
| 16:10:02 | cdent | we've had bug fixes for that since then | |
| 16:10:18 | mriedem | reserved < total in all of those | |
| 16:11:04 | cdent | but capacity is a calculation that involved allocation_ratio as a multiplier | |
| 16:11:07 | cdent | if it is 0 | |
| 16:11:10 | cdent | ... | |
| 16:11:34 | mriedem | yeah | |
| 16:11:38 | mriedem | recheck gerrit restart | |
| 16:11:40 | cdent | I suspect we've got bad exception trapping happening | |
| 16:11:40 | mriedem | ooops | |
| 16:11:45 | mriedem | return int((self.total - self.reserved) * self.allocation_ratio) | |
| 16:12:07 | mriedem | maciejjozefczyk: is that from master with your patch? or pike/ocata? | |
| 16:12:35 | maciejjozefczyk | master, I think from 20 DEC 2017~ when I worked on that | |
| 16:12:43 | maciejjozefczyk | should I pull? | |
| 16:13:03 | mriedem | cdent: BASE_INVENTORY_SCHEMA shows only a max for allocation_ratio, not a min | |
| 16:13:07 | mriedem | in pike anyway | |
| 16:13:11 | maciejjozefczyk | Im on 04c8fa469109098a0ba8e8774f6176c43b7ed19a | |
| 16:13:13 | cdent | mriedem: that's still true, I checked | |
| 16:13:26 | cdent | where things got changed was on the resource tracker side | |
| 16:13:44 | cdent | where it was possible to default to 0, but that was changed, but I'm not sure when/where | |
| 16:14:21 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098 | |
| 16:14:22 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246 | |
| 16:14:22 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187 | |
| 16:14:23 | openstackgerrit | Eric Fried proposed openstack/nova master: Raise conflict exception on RP create 409 https://review.openstack.org/532563 | |
| 16:14:23 | openstackgerrit | Eric Fried proposed openstack/nova master: Fix nits in update_provider_tree series https://review.openstack.org/531260 | |
| 16:14:24 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: SchedulerReportClient.set_traits_for_provider https://review.openstack.org/532564 | |
| 16:14:30 | mriedem | cdent: _normalize_inventory_from_cn_obj ? | |
| 16:14:50 | cdent | not sure, but sounds likely | |
| 16:14:59 | efried | mgoddard WIP: SchedulerReportClient.set_traits_for_provider https://review.openstack.org/532564 is where I was planning to go. | |
| 16:15:14 | mriedem | that's the thing in the RT that sets the allocation_ratio in the inventory payload if the driver.get_inventory() method didn't include allocation ratios | |
| 16:15:38 | mriedem | and all 3 of those default to 0.0 in config :) | |
| 16:15:54 | mriedem | so yeah...we're always sending an inventory update with a capacity that's not going to be accepted | |
| 16:16:12 | efried | ouch | |
| 16:17:08 | maciejjozefczyk | mriedem: yes | |
| 16:18:29 | mriedem | left notes in https://review.openstack.org/#/c/520024/ | |
| 16:18:33 | mriedem | so how does this work today? | |
| 16:19:03 | maciejjozefczyk | today, you mean without my patch? | |
| 16:19:07 | cdent | i swear we've seen this before and changed it | |
| 16:19:11 | mriedem | maciejjozefczyk: yeah | |
| 16:19:37 | maciejjozefczyk | mriedem: https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L696 always pass | |
| 16:19:51 | maciejjozefczyk | so we dont send anything to scheduler | |
| 16:19:54 | maciejjozefczyk | pff, placement* | |
| 16:19:58 | maciejjozefczyk | only DB is updated | |
| 16:20:18 | ameeda | Hello novaers, when I try to deploy overcloud on baremetal using undercloud "installed at vm" I got this error "No compute node record for host undercloud: ComputeHostNotFound_Remote: Compute host undercloud could not be found." from nova-compute.log file | |
| 16:20:25 | cfriesen | mriedem: reading your "working towards feature freeze" email...given reviewer resources is there any point in trying to refresh the live-migration resource tracking patches for Q or is that now basically an R thing? | |
| 16:20:45 | mriedem | live migration resource tracking patches? | |
| 16:21:02 | mriedem | maciejjozefczyk: but at some point we have to set the initial inventory for the provider | |
| 16:21:14 | cfriesen | mriedem: yeah, the ones that have been around forever to fix pinned cpus, hugepages, etc. on live migration | |
| 16:21:41 | mriedem | cfriesen: i didn't realize you were trying to get those into Q | |