Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-10
15:57:51 maciejjozefczyk malformed or otherwise incorrect.\n\n Unable to update inventory for resource provider 52559824-5fb1-424b-a4cf-79da9199447 d: Invalid inventory for 'VCPU' on resource provider '52559824-5fb1-424b-a4cf-79da9199447d'. The reserved value is greater than or equal to total. ", "title": "Bad Request"}]}
15:58:46 mriedem that's the warning, do you have this debug log? https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L765
15:59:12 mriedem i want to see if the request body has "The reserved value is greater than or equal to total." in it
15:59:26 cdent greater than empty value?
15:59:33 maciejjozefczyk mriedem: checking
16:02:23 mriedem cdent: so, we can likely at least turn down the 5 calls per periodic to 2 if we nix the 2 aggregate calls and turn the 2 inventory calls to 1; and then i think the GET /allocations in here is only if you have ocata computes or are using the ironic driver
16:02:43 cdent that makes sense
16:05:26 melwitt lyarwood: thanks for the stable/pike reviews, could you please hit these stable/ocata versions too? https://review.openstack.org/531422 and https://review.openstack.org/#/c/523911
16:06:53 lyarwood melwitt: ack np, I'll get to them tonight
16:07:16 melwitt thanks
16:09:29 maciejjozefczyk mriedem: debug log: DEBUG nova.scheduler.client.report [None req-ad586aa8-27d1-494d-9c4d-bb8f15439fca None None] [req-a4ea518f-1da0-43fd-8348-64704210cb49] Failed inventory update request for resource provider 52559824-5fb1-424b-a4cf-79da9199447d with body: {'resource_provider_generation': 4, 'inventories': {'VCPU': {'allocation_ratio': 0.0, 'total': 2, 'reserved': 0, 'step_size': 1, 'min_unit': 1,
16:09:31 maciejjozefczyk 'max_unit': 2}, 'MEMORY_MB': {'allocation_ratio': 0.0, 'total': 29449, 'reserved': 512, 'ste p_size': 1, 'min_unit': 1, 'max_unit': 29449}, 'DISK_GB': {'allocation_ratio': 0.0, 'total': 193, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 193}}} {{(pid=19609) _update_inventory_attempt /opt/stack/nova/nova/scheduler/cli ent/report.py:765}}
16:09:55 cdent allocation_ratio being 0 is not supposed to happen
16:10:02 cdent we've had bug fixes for that since then
16:10:18 mriedem reserved < total in all of those
16:11:04 cdent but capacity is a calculation that involved allocation_ratio as a multiplier
16:11:07 cdent if it is 0
16:11:10 cdent ...
16:11:34 mriedem yeah
16:11:38 mriedem recheck gerrit restart
16:11:40 mriedem ooops
16:11:40 cdent I suspect we've got bad exception trapping happening
16:11:45 mriedem return int((self.total - self.reserved) * self.allocation_ratio)
16:12:07 mriedem maciejjozefczyk: is that from master with your patch? or pike/ocata?
16:12:35 maciejjozefczyk master, I think from 20 DEC 2017~ when I worked on that
16:12:43 maciejjozefczyk should I pull?
16:13:03 mriedem cdent: BASE_INVENTORY_SCHEMA shows only a max for allocation_ratio, not a min
16:13:07 mriedem in pike anyway
16:13:11 maciejjozefczyk Im on 04c8fa469109098a0ba8e8774f6176c43b7ed19a
16:13:13 cdent mriedem: that's still true, I checked
16:13:26 cdent where things got changed was on the resource tracker side
16:13:44 cdent where it was possible to default to 0, but that was changed, but I'm not sure when/where
16:14:21 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098
16:14:22 openstackgerrit Eric Fried proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187
16:14:22 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246
16:14:23 openstackgerrit Eric Fried proposed openstack/nova master: Fix nits in update_provider_tree series https://review.openstack.org/531260
16:14:23 openstackgerrit Eric Fried proposed openstack/nova master: Raise conflict exception on RP create 409 https://review.openstack.org/532563
16:14:24 openstackgerrit Eric Fried proposed openstack/nova master: WIP: SchedulerReportClient.set_traits_for_provider https://review.openstack.org/532564
16:14:30 mriedem cdent: _normalize_inventory_from_cn_obj ?
16:14:50 cdent not sure, but sounds likely
16:14:59 efried mgoddard WIP: SchedulerReportClient.set_traits_for_provider https://review.openstack.org/532564 is where I was planning to go.
16:15:14 mriedem that's the thing in the RT that sets the allocation_ratio in the inventory payload if the driver.get_inventory() method didn't include allocation ratios
16:15:38 mriedem and all 3 of those default to 0.0 in config :)
16:15:54 mriedem so yeah...we're always sending an inventory update with a capacity that's not going to be accepted
16:16:12 efried ouch
16:17:08 maciejjozefczyk mriedem: yes
16:18:29 mriedem left notes in https://review.openstack.org/#/c/520024/
16:18:33 mriedem so how does this work today?
16:19:03 maciejjozefczyk today, you mean without my patch?
16:19:07 cdent i swear we've seen this before and changed it
16:19:11 mriedem maciejjozefczyk: yeah
16:19:37 maciejjozefczyk mriedem: https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L696 always pass
16:19:51 maciejjozefczyk so we dont send anything to scheduler
16:19:54 maciejjozefczyk pff, placement*
16:19:58 maciejjozefczyk only DB is updated
16:20:18 ameeda Hello novaers, when I try to deploy overcloud on baremetal using undercloud "installed at vm" I got this error "No compute node record for host undercloud: ComputeHostNotFound_Remote: Compute host undercloud could not be found." from nova-compute.log file
16:20:25 cfriesen mriedem: reading your "working towards feature freeze" email...given reviewer resources is there any point in trying to refresh the live-migration resource tracking patches for Q or is that now basically an R thing?
16:20:45 mriedem live migration resource tracking patches?
16:21:02 mriedem maciejjozefczyk: but at some point we have to set the initial inventory for the provider
16:21:14 cfriesen mriedem: yeah, the ones that have been around forever to fix pinned cpus, hugepages, etc. on live migration
16:21:41 mriedem cfriesen: i didn't realize you were trying to get those into Q
16:21:51 cfriesen mriedem: I haven't actively
16:22:04 cfriesen mriedem: sfinucan refreshed one of them in November
16:22:55 mriedem maciejjozefczyk: so i'm trying to figure out why the initial PUT for inventories for that compute node doesn't fail
16:23:05 mriedem like, the first time the compute is created
16:23:28 maciejjozefczyk mriedem: Im curious too
16:23:36 maciejjozefczyk mriedem: also looking
16:24:06 mriedem on the first PUT we'd get here https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L371
16:24:23 mriedem which also checks capacity https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L200
16:25:03 cdent at create time is the inventory constructed by something else, which does not set allocation_ratio in the request? If so, the server side will default to 1.0
16:25:34 mriedem i don't think so, should be the same code,
16:25:37 mriedem calls driver.get_inventory()
16:25:42 mriedem which for libvirt doesn't return allocation_ratio
16:30:20 cdent mriedem: I think maybe it just busted but in our functional tests and tempest we always set those config values
16:30:29 cdent tempest has allocation_ratio conf defaults
16:30:42 cdent and the gabbi test fixture for function does too
16:31:25 mriedem tempest? you mean devstack?
16:31:30 mriedem i don't see allocation_ratio set in nova.conf http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/etc/nova/nova-cpu.conf.txt.gz
16:31:30 cdent yeah, sorry
16:31:58 mriedem http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_27_560214
16:32:00 cdent (I'm simultaneously working on some tempest stuff)
16:32:27 melwitt I've noticed the test_live_migration_actions functional test has been intermittently failing in the gate again http://logs.openstack.org/90/333990/22/check/openstack-tox-functional/24eeb2b/testr_results.html.gz
16:33:00 mriedem cdent: in this test run, this is the first inventory update after the compute node RP is created http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_29_162934
16:33:25 melwitt gibi ^
16:33:54 cdent mriedem: presumably the set to 0.0 here shouldn't be? http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_27_560214
16:34:29 mriedem cdent: 0.0 is the default in the configs
16:34:44 cdent I get that.
16:35:01 cdent But one wouldn't expect that work, would one?
16:35:08 cdent unless 0 is meant to mean 1
16:35:22 cdent and that's the conversation I seem to remember having, but can't find the thread of
16:35:52 mriedem i don't see how it works with the capacity checks on the placement side for updating inventory
16:36:00 mriedem for this ci run, this is the first inventory update http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-placement-api.txt.gz#_Dec_15_15_20_29_138893
16:36:16 mriedem but doesn't tell us much
16:36:28 gibi melwitt: thanks for the heads up, I will look into it
16:36:56 melwitt thanks gibi
16:37:15 cdent mriedem: ah, it is the compute node object that does some jiggery pokery if it is doesn't like the values of the allocation ratios: see _from_db_object in class ComputeNode
16:37:44 mriedem yup i just found that
16:37:54 mriedem maciejjozefczyk:
16:37:54 cdent some presumably somewhere in the first step that pokery is happening, but then getting ruined in the second
16:38:41 mriedem maciejjozefczyk: https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L198

Earlier   Later