Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-30
10:24:18 Dinesh_Bhor sean-k-mooney: yes, thank you so much
11:00:28 openstackgerrit Rong Han proposed openstack/nova master: Reset global variable after unit test is completed. https://review.openstack.org/598088
11:17:43 openstackgerrit Merged openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599
11:36:55 openstackgerrit Rong Han proposed openstack/nova master: Reset global variable after unit test is completed. https://review.openstack.org/598088
12:04:27 bauzas stephenfin: https://review.openstack.org/#/c/530924/6 needs a new oslo.config version, right?
12:07:57 bauzas stephenfin: nevermind, I can see it's from oslo.config 5.2.0
12:07:58 bauzas https://docs.openstack.org/releasenotes/oslo.config/queens.html#relnotes-5-2-0-stable-queens
12:08:18 bauzas and nova uses the latest https://github.com/openstack/nova/blob/master/requirements.txt#L40
12:12:41 zigo bauzas: Don't worry, Rocky doesn't even build with 5.2.0 anywway ... :)
12:12:50 zigo Once more, requirements are just plain wrong.
12:12:56 zigo As usual, I'd say...
12:15:13 zigo Some packages are using oslo_config.sphinxconfiggen which isn't available in oslo.config 5.2.0.
12:15:28 zigo networking-bagpipe for example.
12:16:47 bauzas stephenfin: heh, I found you a new Friday nick <finucannitbacktick> :p
12:17:31 bauzas zigo: that's a project related issue
12:17:51 zigo bauzas: Ok, you need another example ...
12:17:55 bauzas zigo: the reviewers should look at the needed oslo version when they merge a new feature
12:18:07 zigo bauzas: Neutron has 1700+ unit test failures with current lower bounds ! :)
12:41:02 openstackgerrit Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170
12:48:32 stephenfin bauzas: :D
12:48:51 stephenfin bauzas: People will eventually learn :)
12:49:08 stephenfin zigo: What do you mean, it doesn't build?
12:49:50 stephenfin bauzas: Also, we have oslo.config 6.1.0 in lower-constraints so I think we're all good there
12:50:02 bauzas yep
12:56:15 mriedem cdent: can you remind someone internally to re-propose the spec for this for stein? https://blueprints.launchpad.net/nova/+spec/vmware-live-migration
12:56:56 mriedem or you can if you want, it's just procedural
12:57:12 mriedem https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications
12:58:26 cdent mriedem: yeah, I think rado's gonna take care of it, he was on pto for a while
12:59:45 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Restart scheduler in TestNovaManagePlacementHealAllocations https://review.openstack.org/598152
12:59:48 brinzhang mriedem: Could you please review this specs, https://review.openstack.org/#/c/591976/? If you have time :)
13:00:31 mriedem sure
13:00:42 brinzhang thanks ^^
13:05:50 cdent mriedem: is there any new insight on the allocation thing yet, or has that not come round on the radar yet?
13:08:00 mriedem i think i got a failing xen ci result last night after i knocked off for the day, was going to investigate this morning
13:08:04 mriedem 1 sip into coffee ...
13:09:18 openstackgerrit Chen proposed openstack/nova master: Fix filter server list by multiple vm or task states https://review.openstack.org/598154
13:16:04 mriedem alex_xu: ^ looks like a behavior change
13:17:43 mriedem cdent: oh i remember now, i had to re-run the xen ci patch last night b/c it wasn't picking up my dependency in depends-on: <url> form b/c it's still using zuul v2
13:18:08 cdent mriedem: fun!
13:18:37 mriedem the logs are there in the latest failed run though
13:18:40 mriedem http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/13/597613/2/check/dsvm-tempest-neutron-network/cc81140/logs/screen-n-cpu.txt.gz
13:18:48 mriedem looking at req-99d9d496-6720-4837-a2ee-560605fd1afe
13:18:56 mriedem naichuans: efried: ^
13:19:28 mriedem Aug 29 16:56:06.926641 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Using cpu_allocation_ratio 16.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07
13:19:41 mriedem Aug 29 16:56:06.926926 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] RT: Sending compute node inventory changes back toplacement for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07
13:19:55 mriedem Aug 29 16:56:06.965945 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Inventory has not changed in ProviderTree for provider: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07
13:19:55 mriedem WAH WAH
13:20:23 mriedem hmm, but then it says it does update inventory
13:20:24 mriedem n_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 1.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}}
13:20:24 mriedem ia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {'VCPU': {'allocation_ratio': 16.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 1.5, 'total': 12795, 'reserved': 512, 'step_size': 1,
13:20:24 mriedem Aug 29 16:56:07.057208 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: DEBUG nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updating resource provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 generation from 0 to 1 during operation: update_inventory {{(pid=24436) _update_generation /opt/stack/new/nova/nova/compute/provider_tree.py:161}} Aug 29 16:56:07.057499 dsvm-devstack-citr
13:20:48 mriedem there the allocation ratios are all correct
13:21:08 mriedem 12, 'step_size': 1, 'min_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 1.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}} {{(pid=24436) _update_inventory_attempt /opt/stack/new/nova/nova/scheduler/client/report.py:965}}
13:21:08 mriedem Aug 29 16:56:07.058213 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: DEBUG nova.scheduler.client.report [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updated inventory for 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 at generation 1: {'VCPU': {'allocation_ratio': 16.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 1.5, 'total': 12795, 'reserved
13:21:42 cdent goes to zero at 16:58:05.613741
13:23:03 cdent right after an "Inventory has not changed in ProviderTree for provider"
13:23:14 openstackgerrit Radoslav Gerganov proposed openstack/nova-specs master: VMware: add support for live migration https://review.openstack.org/598163
13:24:40 mriedem Aug 29 16:58:05.483508 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Using cpu_allocation_ratio 0.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07
13:24:41 mriedem yeah wtf
13:25:09 mriedem that's in the _normalize_inventory_from_cn_obj method
13:25:21 cdent I'm gonna go with "something is being side-effecty"
13:25:33 mriedem somehow the ComputeNode.cpu_allocation_ratio is getting persisted as 0.0 maybe?
13:25:54 cdent you added a log for that didn't you?
13:26:00 mriedem yes and i don't see either of them
13:26:03 mriedem https://review.openstack.org/#/c/597560/3/nova/objects/compute_node.py.
13:26:50 cdent w & the t & the actual f
13:31:34 cdent write before the correct inventory is sent we have this line "Using cpu_allocation_ratio 0.0 for node [...]". that value, I would guess, is somehow being used for the _next_ inventory
13:32:22 mriedem i noticed that also,
13:32:26 cdent we're getting update inventories within 2 ms of one another. first one right, second one wrong
13:32:27 mriedem Aug 29 16:58:05.483508 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Using cpu_allocation_ratio 0.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07
13:32:27 mriedem we have this with the wrong value
13:32:33 mriedem and we have a good update here:
13:32:38 mriedem , u'total': 12795, u'reserved': 512, u'step_size': 1, u'min_unit': 1, u'max_unit': 12795}, u'DISK_GB': {u'allocation_ratio': 1.0, u'total': 47, u'reserved': 0, u'step_size': 1, u'min_unit': 1, u'max_unit': 47}}
13:32:38 mriedem Aug 29 16:58:05.525151 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {u'VCPU': {u'allocation_ratio': 16.0, u'total': 8, u'reserved': 0, u'step_size': 1, u'min_unit': 1, u'max_unit': 8}, u'MEMORY_MB': {u'allocation_ratio':
13:32:47 mriedem 12795, 'reserved': 512, 'step_size': 1, 'min_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 0.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}}
13:32:47 mriedem Aug 29 16:58:05.613741 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {'VCPU': {'allocation_ratio': 0.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 0.0, 'tot
13:32:47 mriedem and then the bad update:
13:42:51 mriedem hmm https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L622
13:43:03 mriedem ^ we set the ComputeNode.cpu_allocation_ratio based on the config, which is 0.0
13:43:45 sean-k-mooney mriedem: so it had the correct allocation ratio then it was chaged to 0.0
13:43:46 mriedem i bet that is the problem
13:45:10 mriedem _copy_resources is called from _init_compute_node,
13:45:29 cdent mriedem: but none of that stuff is new is it?
13:45:37 mriedem and on initial create of the compute node record, the ComputeNode.create() method will call _from_db_object at the end and fix the 0.0 allocation ratio to the hard-coded one,
13:45:57 openstackgerrit Takashi NATSUME proposed openstack/nova master: Transform compute_task notifications https://review.openstack.org/482629
13:46:09 mriedem then in a periodic run, the cn already exists, we'll copy over the busted 0.0 allocations from config, but b/c we removed the _update calls there, we don't fix the allocation ratios
13:46:30 cdent ah, there's the rub
13:46:31 mriedem which is https://review.openstack.org/#/c/520024/
13:46:40 mriedem but that doesn't explain how zigo was hitting this on rocky
13:46:50 mriedem or how we're *not* hitting this in the normal gate
13:47:17 openstackgerrit Merged openstack/nova master: (Re)start caching scheduler after starting computes in tests https://review.openstack.org/597606
13:47:26 cdent does the normal gate set conf?
13:47:31 mriedem no
13:47:34 sean-k-mooney mriedem: is there a reason we do not set the defults here https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L413-L416
13:47:49 mriedem http://logs.openstack.org/24/520024/9/check/tempest-full/cbd025d/controller/logs/etc/nova/nova-cpu_conf.txt.gz
13:48:00 mriedem sean-k-mooney: yes read the help text
13:48:27 alex_xu mriedem: yea, sounds like
13:49:48 sean-k-mooney mriedem: hum right... so we can single to use the schduler nodes value
13:50:21 mriedem this was the change to make the defaults 0.0 https://github.com/openstack/nova/commit/4a9e14a7a73832b6b878160ba4a45f259d078d27
13:51:30 sean-k-mooney mriedem: "That compat mode (having ratios defaulted to 0.0) is only planned to be kept for

Earlier   Later