| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-30 | |||
| 11:36:55 | openstackgerrit | Rong Han proposed openstack/nova master: Reset global variable after unit test is completed. https://review.openstack.org/598088 | |
| 12:04:27 | bauzas | stephenfin: https://review.openstack.org/#/c/530924/6 needs a new oslo.config version, right? | |
| 12:07:57 | bauzas | stephenfin: nevermind, I can see it's from oslo.config 5.2.0 | |
| 12:07:58 | bauzas | https://docs.openstack.org/releasenotes/oslo.config/queens.html#relnotes-5-2-0-stable-queens | |
| 12:08:18 | bauzas | and nova uses the latest https://github.com/openstack/nova/blob/master/requirements.txt#L40 | |
| 12:12:41 | zigo | bauzas: Don't worry, Rocky doesn't even build with 5.2.0 anywway ... :) | |
| 12:12:50 | zigo | Once more, requirements are just plain wrong. | |
| 12:12:56 | zigo | As usual, I'd say... | |
| 12:15:13 | zigo | Some packages are using oslo_config.sphinxconfiggen which isn't available in oslo.config 5.2.0. | |
| 12:15:28 | zigo | networking-bagpipe for example. | |
| 12:16:47 | bauzas | stephenfin: heh, I found you a new Friday nick <finucannitbacktick> :p | |
| 12:17:31 | bauzas | zigo: that's a project related issue | |
| 12:17:51 | zigo | bauzas: Ok, you need another example ... | |
| 12:17:55 | bauzas | zigo: the reviewers should look at the needed oslo version when they merge a new feature | |
| 12:18:07 | zigo | bauzas: Neutron has 1700+ unit test failures with current lower bounds ! :) | |
| 12:41:02 | openstackgerrit | Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170 | |
| 12:48:32 | stephenfin | bauzas: :D | |
| 12:48:51 | stephenfin | bauzas: People will eventually learn :) | |
| 12:49:08 | stephenfin | zigo: What do you mean, it doesn't build? | |
| 12:49:50 | stephenfin | bauzas: Also, we have oslo.config 6.1.0 in lower-constraints so I think we're all good there | |
| 12:50:02 | bauzas | yep | |
| 12:56:15 | mriedem | cdent: can you remind someone internally to re-propose the spec for this for stein? https://blueprints.launchpad.net/nova/+spec/vmware-live-migration | |
| 12:56:56 | mriedem | or you can if you want, it's just procedural | |
| 12:57:12 | mriedem | https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications | |
| 12:58:26 | cdent | mriedem: yeah, I think rado's gonna take care of it, he was on pto for a while | |
| 12:59:45 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Restart scheduler in TestNovaManagePlacementHealAllocations https://review.openstack.org/598152 | |
| 12:59:48 | brinzhang | mriedem: Could you please review this specs, https://review.openstack.org/#/c/591976/? If you have time :) | |
| 13:00:31 | mriedem | sure | |
| 13:00:42 | brinzhang | thanks ^^ | |
| 13:05:50 | cdent | mriedem: is there any new insight on the allocation thing yet, or has that not come round on the radar yet? | |
| 13:08:00 | mriedem | i think i got a failing xen ci result last night after i knocked off for the day, was going to investigate this morning | |
| 13:08:04 | mriedem | 1 sip into coffee ... | |
| 13:09:18 | openstackgerrit | Chen proposed openstack/nova master: Fix filter server list by multiple vm or task states https://review.openstack.org/598154 | |
| 13:16:04 | mriedem | alex_xu: ^ looks like a behavior change | |
| 13:17:43 | mriedem | cdent: oh i remember now, i had to re-run the xen ci patch last night b/c it wasn't picking up my dependency in depends-on: <url> form b/c it's still using zuul v2 | |
| 13:18:08 | cdent | mriedem: fun! | |
| 13:18:37 | mriedem | the logs are there in the latest failed run though | |
| 13:18:40 | mriedem | http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/13/597613/2/check/dsvm-tempest-neutron-network/cc81140/logs/screen-n-cpu.txt.gz | |
| 13:18:48 | mriedem | looking at req-99d9d496-6720-4837-a2ee-560605fd1afe | |
| 13:18:56 | mriedem | naichuans: efried: ^ | |
| 13:19:28 | mriedem | Aug 29 16:56:06.926641 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Using cpu_allocation_ratio 16.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:19:41 | mriedem | Aug 29 16:56:06.926926 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] RT: Sending compute node inventory changes back toplacement for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:19:55 | mriedem | Aug 29 16:56:06.965945 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Inventory has not changed in ProviderTree for provider: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:19:55 | mriedem | WAH WAH | |
| 13:20:23 | mriedem | hmm, but then it says it does update inventory | |
| 13:20:24 | mriedem | n_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 1.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}} | |
| 13:20:24 | mriedem | ia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {'VCPU': {'allocation_ratio': 16.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 1.5, 'total': 12795, 'reserved': 512, 'step_size': 1, | |
| 13:20:24 | mriedem | Aug 29 16:56:07.057208 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: DEBUG nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updating resource provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 generation from 0 to 1 during operation: update_inventory {{(pid=24436) _update_generation /opt/stack/new/nova/nova/compute/provider_tree.py:161}} Aug 29 16:56:07.057499 dsvm-devstack-citr | |
| 13:20:48 | mriedem | there the allocation ratios are all correct | |
| 13:21:08 | mriedem | 12, 'step_size': 1, 'min_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 1.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}} {{(pid=24436) _update_inventory_attempt /opt/stack/new/nova/nova/scheduler/client/report.py:965}} | |
| 13:21:08 | mriedem | Aug 29 16:56:07.058213 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: DEBUG nova.scheduler.client.report [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updated inventory for 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 at generation 1: {'VCPU': {'allocation_ratio': 16.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 1.5, 'total': 12795, 'reserved | |
| 13:21:42 | cdent | goes to zero at 16:58:05.613741 | |
| 13:23:03 | cdent | right after an "Inventory has not changed in ProviderTree for provider" | |
| 13:23:14 | openstackgerrit | Radoslav Gerganov proposed openstack/nova-specs master: VMware: add support for live migration https://review.openstack.org/598163 | |
| 13:24:40 | mriedem | Aug 29 16:58:05.483508 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Using cpu_allocation_ratio 0.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:24:41 | mriedem | yeah wtf | |
| 13:25:09 | mriedem | that's in the _normalize_inventory_from_cn_obj method | |
| 13:25:21 | cdent | I'm gonna go with "something is being side-effecty" | |
| 13:25:33 | mriedem | somehow the ComputeNode.cpu_allocation_ratio is getting persisted as 0.0 maybe? | |
| 13:25:54 | cdent | you added a log for that didn't you? | |
| 13:26:00 | mriedem | yes and i don't see either of them | |
| 13:26:03 | mriedem | https://review.openstack.org/#/c/597560/3/nova/objects/compute_node.py. | |
| 13:26:50 | cdent | w & the t & the actual f | |
| 13:31:34 | cdent | write before the correct inventory is sent we have this line "Using cpu_allocation_ratio 0.0 for node [...]". that value, I would guess, is somehow being used for the _next_ inventory | |
| 13:32:22 | mriedem | i noticed that also, | |
| 13:32:26 | cdent | we're getting update inventories within 2 ms of one another. first one right, second one wrong | |
| 13:32:27 | mriedem | Aug 29 16:58:05.483508 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Using cpu_allocation_ratio 0.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:32:27 | mriedem | we have this with the wrong value | |
| 13:32:33 | mriedem | and we have a good update here: | |
| 13:32:38 | mriedem | , u'total': 12795, u'reserved': 512, u'step_size': 1, u'min_unit': 1, u'max_unit': 12795}, u'DISK_GB': {u'allocation_ratio': 1.0, u'total': 47, u'reserved': 0, u'step_size': 1, u'min_unit': 1, u'max_unit': 47}} | |
| 13:32:38 | mriedem | Aug 29 16:58:05.525151 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {u'VCPU': {u'allocation_ratio': 16.0, u'total': 8, u'reserved': 0, u'step_size': 1, u'min_unit': 1, u'max_unit': 8}, u'MEMORY_MB': {u'allocation_ratio': | |
| 13:32:47 | mriedem | 12795, 'reserved': 512, 'step_size': 1, 'min_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 0.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}} | |
| 13:32:47 | mriedem | Aug 29 16:58:05.613741 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {'VCPU': {'allocation_ratio': 0.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 0.0, 'tot | |
| 13:32:47 | mriedem | and then the bad update: | |
| 13:42:51 | mriedem | hmm https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L622 | |
| 13:43:03 | mriedem | ^ we set the ComputeNode.cpu_allocation_ratio based on the config, which is 0.0 | |
| 13:43:45 | sean-k-mooney | mriedem: so it had the correct allocation ratio then it was chaged to 0.0 | |
| 13:43:46 | mriedem | i bet that is the problem | |
| 13:45:10 | mriedem | _copy_resources is called from _init_compute_node, | |
| 13:45:29 | cdent | mriedem: but none of that stuff is new is it? | |
| 13:45:37 | mriedem | and on initial create of the compute node record, the ComputeNode.create() method will call _from_db_object at the end and fix the 0.0 allocation ratio to the hard-coded one, | |
| 13:45:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform compute_task notifications https://review.openstack.org/482629 | |
| 13:46:09 | mriedem | then in a periodic run, the cn already exists, we'll copy over the busted 0.0 allocations from config, but b/c we removed the _update calls there, we don't fix the allocation ratios | |
| 13:46:30 | cdent | ah, there's the rub | |
| 13:46:31 | mriedem | which is https://review.openstack.org/#/c/520024/ | |
| 13:46:40 | mriedem | but that doesn't explain how zigo was hitting this on rocky | |
| 13:46:50 | mriedem | or how we're *not* hitting this in the normal gate | |
| 13:47:17 | openstackgerrit | Merged openstack/nova master: (Re)start caching scheduler after starting computes in tests https://review.openstack.org/597606 | |
| 13:47:26 | cdent | does the normal gate set conf? | |
| 13:47:31 | mriedem | no | |
| 13:47:34 | sean-k-mooney | mriedem: is there a reason we do not set the defults here https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L413-L416 | |
| 13:47:49 | mriedem | http://logs.openstack.org/24/520024/9/check/tempest-full/cbd025d/controller/logs/etc/nova/nova-cpu_conf.txt.gz | |
| 13:48:00 | mriedem | sean-k-mooney: yes read the help text | |
| 13:48:27 | alex_xu | mriedem: yea, sounds like | |
| 13:49:48 | sean-k-mooney | mriedem: hum right... so we can single to use the schduler nodes value | |
| 13:50:21 | mriedem | this was the change to make the defaults 0.0 https://github.com/openstack/nova/commit/4a9e14a7a73832b6b878160ba4a45f259d078d27 | |
| 13:51:30 | sean-k-mooney | mriedem: "That compat mode (having ratios defaulted to 0.0) is only planned to be kept for | |
| 13:51:31 | sean-k-mooney | Liberty and will be removed in the next release (Mitaka) | |
| 13:52:03 | sean-k-mooney | well that never happened | |
| 13:52:46 | mriedem | talk to bauzas | |