Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-10
16:07:16 melwitt thanks
16:09:29 maciejjozefczyk mriedem: debug log: DEBUG nova.scheduler.client.report [None req-ad586aa8-27d1-494d-9c4d-bb8f15439fca None None] [req-a4ea518f-1da0-43fd-8348-64704210cb49] Failed inventory update request for resource provider 52559824-5fb1-424b-a4cf-79da9199447d with body: {'resource_provider_generation': 4, 'inventories': {'VCPU': {'allocation_ratio': 0.0, 'total': 2, 'reserved': 0, 'step_size': 1, 'min_unit': 1,
16:09:31 maciejjozefczyk 'max_unit': 2}, 'MEMORY_MB': {'allocation_ratio': 0.0, 'total': 29449, 'reserved': 512, 'ste p_size': 1, 'min_unit': 1, 'max_unit': 29449}, 'DISK_GB': {'allocation_ratio': 0.0, 'total': 193, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 193}}} {{(pid=19609) _update_inventory_attempt /opt/stack/nova/nova/scheduler/cli ent/report.py:765}}
16:09:55 cdent allocation_ratio being 0 is not supposed to happen
16:10:02 cdent we've had bug fixes for that since then
16:10:18 mriedem reserved < total in all of those
16:11:04 cdent but capacity is a calculation that involved allocation_ratio as a multiplier
16:11:07 cdent if it is 0
16:11:10 cdent ...
16:11:34 mriedem yeah
16:11:38 mriedem recheck gerrit restart
16:11:40 mriedem ooops
16:11:40 cdent I suspect we've got bad exception trapping happening
16:11:45 mriedem return int((self.total - self.reserved) * self.allocation_ratio)
16:12:07 mriedem maciejjozefczyk: is that from master with your patch? or pike/ocata?
16:12:35 maciejjozefczyk master, I think from 20 DEC 2017~ when I worked on that
16:12:43 maciejjozefczyk should I pull?
16:13:03 mriedem cdent: BASE_INVENTORY_SCHEMA shows only a max for allocation_ratio, not a min
16:13:07 mriedem in pike anyway
16:13:11 maciejjozefczyk Im on 04c8fa469109098a0ba8e8774f6176c43b7ed19a
16:13:13 cdent mriedem: that's still true, I checked
16:13:26 cdent where things got changed was on the resource tracker side
16:13:44 cdent where it was possible to default to 0, but that was changed, but I'm not sure when/where
16:14:21 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098
16:14:22 openstackgerrit Eric Fried proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187
16:14:22 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246
16:14:23 openstackgerrit Eric Fried proposed openstack/nova master: Fix nits in update_provider_tree series https://review.openstack.org/531260
16:14:23 openstackgerrit Eric Fried proposed openstack/nova master: Raise conflict exception on RP create 409 https://review.openstack.org/532563
16:14:24 openstackgerrit Eric Fried proposed openstack/nova master: WIP: SchedulerReportClient.set_traits_for_provider https://review.openstack.org/532564
16:14:30 mriedem cdent: _normalize_inventory_from_cn_obj ?
16:14:50 cdent not sure, but sounds likely
16:14:59 efried mgoddard WIP: SchedulerReportClient.set_traits_for_provider https://review.openstack.org/532564 is where I was planning to go.
16:15:14 mriedem that's the thing in the RT that sets the allocation_ratio in the inventory payload if the driver.get_inventory() method didn't include allocation ratios
16:15:38 mriedem and all 3 of those default to 0.0 in config :)
16:15:54 mriedem so yeah...we're always sending an inventory update with a capacity that's not going to be accepted
16:16:12 efried ouch
16:17:08 maciejjozefczyk mriedem: yes
16:18:29 mriedem left notes in https://review.openstack.org/#/c/520024/
16:18:33 mriedem so how does this work today?
16:19:03 maciejjozefczyk today, you mean without my patch?
16:19:07 cdent i swear we've seen this before and changed it
16:19:11 mriedem maciejjozefczyk: yeah
16:19:37 maciejjozefczyk mriedem: https://github.com/openstack/nova/blob/cf33de28b15bb445d34bbdda1897130812e3b5c5/nova/scheduler/client/report.py#L696 always pass
16:19:51 maciejjozefczyk so we dont send anything to scheduler
16:19:54 maciejjozefczyk pff, placement*
16:19:58 maciejjozefczyk only DB is updated
16:20:18 ameeda Hello novaers, when I try to deploy overcloud on baremetal using undercloud "installed at vm" I got this error "No compute node record for host undercloud: ComputeHostNotFound_Remote: Compute host undercloud could not be found." from nova-compute.log file
16:20:25 cfriesen mriedem: reading your "working towards feature freeze" email...given reviewer resources is there any point in trying to refresh the live-migration resource tracking patches for Q or is that now basically an R thing?
16:20:45 mriedem live migration resource tracking patches?
16:21:02 mriedem maciejjozefczyk: but at some point we have to set the initial inventory for the provider
16:21:14 cfriesen mriedem: yeah, the ones that have been around forever to fix pinned cpus, hugepages, etc. on live migration
16:21:41 mriedem cfriesen: i didn't realize you were trying to get those into Q
16:21:51 cfriesen mriedem: I haven't actively
16:22:04 cfriesen mriedem: sfinucan refreshed one of them in November
16:22:55 mriedem maciejjozefczyk: so i'm trying to figure out why the initial PUT for inventories for that compute node doesn't fail
16:23:05 mriedem like, the first time the compute is created
16:23:28 maciejjozefczyk mriedem: Im curious too
16:23:36 maciejjozefczyk mriedem: also looking
16:24:06 mriedem on the first PUT we'd get here https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L371
16:24:23 mriedem which also checks capacity https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L200
16:25:03 cdent at create time is the inventory constructed by something else, which does not set allocation_ratio in the request? If so, the server side will default to 1.0
16:25:34 mriedem i don't think so, should be the same code,
16:25:37 mriedem calls driver.get_inventory()
16:25:42 mriedem which for libvirt doesn't return allocation_ratio
16:30:20 cdent mriedem: I think maybe it just busted but in our functional tests and tempest we always set those config values
16:30:29 cdent tempest has allocation_ratio conf defaults
16:30:42 cdent and the gabbi test fixture for function does too
16:31:25 mriedem tempest? you mean devstack?
16:31:30 mriedem i don't see allocation_ratio set in nova.conf http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/etc/nova/nova-cpu.conf.txt.gz
16:31:30 cdent yeah, sorry
16:31:58 mriedem http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_27_560214
16:32:00 cdent (I'm simultaneously working on some tempest stuff)
16:32:27 melwitt I've noticed the test_live_migration_actions functional test has been intermittently failing in the gate again http://logs.openstack.org/90/333990/22/check/openstack-tox-functional/24eeb2b/testr_results.html.gz
16:33:00 mriedem cdent: in this test run, this is the first inventory update after the compute node RP is created http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_29_162934
16:33:25 melwitt gibi ^
16:33:54 cdent mriedem: presumably the set to 0.0 here shouldn't be? http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_27_560214
16:34:29 mriedem cdent: 0.0 is the default in the configs
16:34:44 cdent I get that.
16:35:01 cdent But one wouldn't expect that work, would one?
16:35:08 cdent unless 0 is meant to mean 1
16:35:22 cdent and that's the conversation I seem to remember having, but can't find the thread of
16:35:52 mriedem i don't see how it works with the capacity checks on the placement side for updating inventory
16:36:00 mriedem for this ci run, this is the first inventory update http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-placement-api.txt.gz#_Dec_15_15_20_29_138893
16:36:16 mriedem but doesn't tell us much
16:36:28 gibi melwitt: thanks for the heads up, I will look into it
16:36:56 melwitt thanks gibi
16:37:15 cdent mriedem: ah, it is the compute node object that does some jiggery pokery if it is doesn't like the values of the allocation ratios: see _from_db_object in class ComputeNode
16:37:44 mriedem yup i just found that
16:37:54 mriedem maciejjozefczyk:
16:37:54 cdent some presumably somewhere in the first step that pokery is happening, but then getting ruined in the second
16:38:41 mriedem maciejjozefczyk: https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L198
16:40:51 maciejjozefczyk mriedem: mmmmm
16:41:49 maciejjozefczyk mriedem: and yes, I dont have it specified in config at all
16:43:29 mriedem i don't really understand why the inventory request wouldn't have the allocation_ratio values from the compute node object though
16:43:39 mriedem because in your patch, _update is called after the compute node record is created,
16:43:56 mriedem and ComputeNode.create() eventually calls _from_db_object to set the 'default' allocation ratios on the object itself
16:43:57 cfriesen just stumbled over something odd...if the os_type isn't set, then in some scenarios the ephemeral disks are formated vfat. if the ephemeral size is too big for vfat, this chokes. Should we default to something else in the code or just not format it at all if a default isn't specified in the config file?
16:44:03 mriedem which will be used in _normalize_inventory_from_cn_obj
16:44:51 cfriesen mdbooth: ^
16:48:08 mriedem maciejjozefczyk: if you have a recreate of that failure, try logging the compute node record in _update() before calling reportclient.set_inventory_for_provider

Earlier   Later