Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-10
15:28:30 sean-k-mooney efried: syntaks for traits to resouces. there was discusstion of a resouce_1=<class x>,required1=<trait y>,<trait z> query arg syntaks for get allocation candiates
15:28:40 cdent mriedem: that may not be right, have a look at point B1 in the list posting above
15:29:29 openstackgerrit Balazs Gibizer proposed openstack/nova master: Deduplicate service status notification samples https://review.openstack.org/531381
15:29:39 cdent (it maybe also be right, though, but without logs, hard to say)
15:29:53 sean-k-mooney efried: i dont think we have a similar fuctionality for the resouce providres api however.
15:31:17 mriedem cdent: yeah, you might be right, because if we hit an inventory conflict, we delete the rp_uuid from the cache and then get the resource provider again to update the generation, but i don't see a GET to just /resource_providers in klindgren_'s output
15:31:51 mriedem so likely hitting something that's not a 409, but would need logs
15:33:59 efried sean-k-mooney Correct.
15:34:19 efried mriedem FYI, I'm trying to rework all of this atm
15:34:43 efried mriedem But are you looking for a way to get this "fixed" in pike?
15:38:25 mriedem efried: well, kind of depends on how much operators that are rolling up to ocata and pike are going to be complaining about the new load their computes are making becaues of a lot of http traffic to placement every minute
15:38:43 mriedem i think klindgren_ is working around it by turning down the update_available_resource report interval so it doesn't run every minute
15:39:42 efried Okay, well, keep me posted. I'll be interested in contributing to (or at least reviewing) the code if we go there.
15:41:13 mriedem cdent: ha "After that every 60s or so, five requests are made:"
15:41:14 mriedem right on
15:41:39 mriedem tracking here btw https://bugs.launchpad.net/nova/+bug/1742467
15:41:41 openstack Launchpad bug 1742467 in OpenStack Compute (nova) "Compute unnecessarily gets resource provider aggregates during every update_available_resource run" [Undecided,New]
15:43:06 cdent mriedem: in your thinking just now did you get any clearer picture on the why of double inventory GET?
15:43:11 openstackgerrit Mark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver https://review.openstack.org/532287
15:43:11 openstackgerrit Mark Goddard proposed openstack/nova master: WIP: Send traits to ironic on server boot https://review.openstack.org/508116
15:43:12 openstackgerrit Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288
15:43:12 openstackgerrit Mark Goddard proposed openstack/nova master: Add has_any_traits() to provider tree https://review.openstack.org/532289
15:43:13 openstackgerrit Mark Goddard proposed openstack/nova master: Call get_traits() in the resource tracker https://review.openstack.org/532290
15:43:13 openstackgerrit Mark Goddard proposed openstack/nova master: Add support to scheduler client for setting traits https://review.openstack.org/532539
15:43:59 mriedem cdent: yes it's the RT
15:44:09 mriedem _update_available_resource is the call from the compute periodic task,
15:44:13 mriedem which calls _init_compute_node
15:44:23 mriedem when we already have the compute node, it calls _update
15:44:32 mriedem which eventually does the update_inventory_attempt stuff in the report client
15:44:42 mriedem then at the end of _update_available_resource,
15:44:44 mriedem we call _update again
15:44:53 mriedem so that's your 2 inventory updates
15:45:06 mriedem which, johnthetubaguy changed in queens
15:45:16 mriedem or wait,no
15:45:37 mriedem https://review.openstack.org/#/c/520024/
15:45:44 mriedem that would fix the double GET inventories
15:48:00 mriedem maciejjozefczyk: have you figured out anything more about https://review.openstack.org/#/c/520024/ ?
15:49:25 maciejjozefczyk mriedem: hah!
15:49:32 maciejjozefczyk mriedem: aready working on this
15:49:51 maciejjozefczyk and yes, I found something strange, but I need big prove about it
15:49:57 maciejjozefczyk I'll post it today in review
15:52:01 maciejjozefczyk mriedem: basically: each time self._provider_tree.has_inventory_changed() returns False here:
15:52:02 maciejjozefczyk https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L696
15:52:10 maciejjozefczyk without my change
15:53:00 maciejjozefczyk with my change: It tries to update inventory_data to placement and then placement raises this strange Exception
15:54:09 maciejjozefczyk so for now in upstream we update only DB (once with faulty values, second time with proper ones)
15:54:53 mriedem so it's failing this check? https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L231
15:55:22 maciejjozefczyk mriedem: exactly
15:56:48 mriedem do you see this debug message with the PUT inventory request body in it when this fails? https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L765
15:57:43 maciejjozefczyk mriedem: yes
15:57:49 maciejjozefczyk WARNING nova.scheduler.client.report [None req-89e951da-7283-473f-9de0-53854741839a None None] [req-b6f884cd-883a-4964-8e99-a8f0 f754c0df] Failed to update inventory for resource provider 52559824-5fb1-424b-a4cf-79da9199447d: 400 {"errors": [{"status": 400, "request_id": "req-b6f884cd-883a-4964-8e99-a8f0f754c0df", "detail": "The server could not comply with the request since it is e ither
15:57:51 maciejjozefczyk malformed or otherwise incorrect.\n\n Unable to update inventory for resource provider 52559824-5fb1-424b-a4cf-79da9199447 d: Invalid inventory for 'VCPU' on resource provider '52559824-5fb1-424b-a4cf-79da9199447d'. The reserved value is greater than or equal to total. ", "title": "Bad Request"}]}
15:58:46 mriedem that's the warning, do you have this debug log? https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L765
15:59:12 mriedem i want to see if the request body has "The reserved value is greater than or equal to total." in it
15:59:26 cdent greater than empty value?
15:59:33 maciejjozefczyk mriedem: checking
16:02:23 mriedem cdent: so, we can likely at least turn down the 5 calls per periodic to 2 if we nix the 2 aggregate calls and turn the 2 inventory calls to 1; and then i think the GET /allocations in here is only if you have ocata computes or are using the ironic driver
16:02:43 cdent that makes sense
16:05:26 melwitt lyarwood: thanks for the stable/pike reviews, could you please hit these stable/ocata versions too? https://review.openstack.org/531422 and https://review.openstack.org/#/c/523911
16:06:53 lyarwood melwitt: ack np, I'll get to them tonight
16:07:16 melwitt thanks
16:09:29 maciejjozefczyk mriedem: debug log: DEBUG nova.scheduler.client.report [None req-ad586aa8-27d1-494d-9c4d-bb8f15439fca None None] [req-a4ea518f-1da0-43fd-8348-64704210cb49] Failed inventory update request for resource provider 52559824-5fb1-424b-a4cf-79da9199447d with body: {'resource_provider_generation': 4, 'inventories': {'VCPU': {'allocation_ratio': 0.0, 'total': 2, 'reserved': 0, 'step_size': 1, 'min_unit': 1,
16:09:31 maciejjozefczyk 'max_unit': 2}, 'MEMORY_MB': {'allocation_ratio': 0.0, 'total': 29449, 'reserved': 512, 'ste p_size': 1, 'min_unit': 1, 'max_unit': 29449}, 'DISK_GB': {'allocation_ratio': 0.0, 'total': 193, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 193}}} {{(pid=19609) _update_inventory_attempt /opt/stack/nova/nova/scheduler/cli ent/report.py:765}}
16:09:55 cdent allocation_ratio being 0 is not supposed to happen
16:10:02 cdent we've had bug fixes for that since then
16:10:18 mriedem reserved < total in all of those
16:11:04 cdent but capacity is a calculation that involved allocation_ratio as a multiplier
16:11:07 cdent if it is 0
16:11:10 cdent ...
16:11:34 mriedem yeah
16:11:38 mriedem recheck gerrit restart
16:11:40 cdent I suspect we've got bad exception trapping happening
16:11:40 mriedem ooops
16:11:45 mriedem return int((self.total - self.reserved) * self.allocation_ratio)
16:12:07 mriedem maciejjozefczyk: is that from master with your patch? or pike/ocata?
16:12:35 maciejjozefczyk master, I think from 20 DEC 2017~ when I worked on that
16:12:43 maciejjozefczyk should I pull?
16:13:03 mriedem cdent: BASE_INVENTORY_SCHEMA shows only a max for allocation_ratio, not a min
16:13:07 mriedem in pike anyway
16:13:11 maciejjozefczyk Im on 04c8fa469109098a0ba8e8774f6176c43b7ed19a
16:13:13 cdent mriedem: that's still true, I checked
16:13:26 cdent where things got changed was on the resource tracker side
16:13:44 cdent where it was possible to default to 0, but that was changed, but I'm not sure when/where
16:14:21 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098
16:14:22 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246
16:14:22 openstackgerrit Eric Fried proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187
16:14:23 openstackgerrit Eric Fried proposed openstack/nova master: Raise conflict exception on RP create 409 https://review.openstack.org/532563
16:14:23 openstackgerrit Eric Fried proposed openstack/nova master: Fix nits in update_provider_tree series https://review.openstack.org/531260
16:14:24 openstackgerrit Eric Fried proposed openstack/nova master: WIP: SchedulerReportClient.set_traits_for_provider https://review.openstack.org/532564
16:14:30 mriedem cdent: _normalize_inventory_from_cn_obj ?
16:14:50 cdent not sure, but sounds likely
16:14:59 efried mgoddard WIP: SchedulerReportClient.set_traits_for_provider https://review.openstack.org/532564 is where I was planning to go.
16:15:14 mriedem that's the thing in the RT that sets the allocation_ratio in the inventory payload if the driver.get_inventory() method didn't include allocation ratios
16:15:38 mriedem and all 3 of those default to 0.0 in config :)
16:15:54 mriedem so yeah...we're always sending an inventory update with a capacity that's not going to be accepted
16:16:12 efried ouch
16:17:08 maciejjozefczyk mriedem: yes
16:18:29 mriedem left notes in https://review.openstack.org/#/c/520024/
16:18:33 mriedem so how does this work today?
16:19:03 maciejjozefczyk today, you mean without my patch?

Earlier   Later