| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-30 | |||
| 10:07:37 | Dinesh_Bhor | Or something else may be | |
| 10:08:01 | sean-k-mooney | Dinesh_Bhor: this is similar to premptiable instances. in that case we also had the idea of two classes of instance that may have different sla's | |
| 10:09:10 | sean-k-mooney | Dinesh_Bhor: so if you were to submit a blueprint for this it may be worth trying to adress that usecase also so its a more general solution. | |
| 10:09:20 | sean-k-mooney | Dinesh_Bhor: that said the rich vms | |
| 10:09:42 | sean-k-mooney | is there richness determined by the host they land on or is it an aspect of the flavor | |
| 10:10:07 | sean-k-mooney | e.g. certin flavor you would like to have a seperate quota for. | |
| 10:11:27 | sean-k-mooney | Dinesh_Bhor: submitting a blueprint that clearly states what your usecase is and the constraits is never a bad idea but that highlevel usecase is what we would like it to capture rather then i think it should be done X way | |
| 10:12:36 | Dinesh_Bhor | sean-k-mooney: we have dedicated rich flavors which land on predefined hypervisors. Its like giving bare metal kind of experience with Rich-VMs | |
| 10:14:23 | Dinesh_Bhor | sean-k-mooney: hosts are high in memory, cpus to give bare metal kind of performance. So we want Rich-Flavors to be deployed on those hosts and want to manage quota for them. | |
| 10:14:41 | Dinesh_Bhor | sean-k-mooney: similar to normal vms per project | |
| 10:14:42 | sean-k-mooney | Dinesh_Bhor: right in that case rather then an instance quota the abblity to set a flavor qouta for a speific flavor would be enough to suit your partcalar usecase correct? | |
| 10:15:17 | sean-k-mooney | you could then use flavor to host affinity to ensure those flavor got shceduled to the correct hosts | |
| 10:15:33 | Dinesh_Bhor | A project can have normal as well as rich vms | |
| 10:16:03 | Dinesh_Bhor | sean-k-mooney: yes, we are managing that with AggregateInstanceExtraSpecFilter | |
| 10:16:26 | Dinesh_Bhor | sean-k-mooney: we are on Mitaka so can not use placement. | |
| 10:16:45 | sean-k-mooney | Dinesh_Bhor: yes when i said flavor quota i ment can have 10 instances of flavor X and any number of flavors without a qota provided they dont exceed other qoutas such as cpus | |
| 10:18:38 | sean-k-mooney | Dinesh_Bhor: the only non invasive way i can consive of to achive this i mitaka would be to write a schduler filter that would check you usage of the rich vms and fail all hosts if you exceed a qouta | |
| 10:21:38 | Dinesh_Bhor | sean-k-mooney: yes, but for that I thought of storing "no of rich-vms allowed" metadata in host-aggregate for per project but that will again degrade the performance of scheduler I think If we have 1000+ projects. | |
| 10:22:06 | Dinesh_Bhor | sean-k-mooney: okay, let me check the flavor quota thing. | |
| 10:22:11 | Dinesh_Bhor | first | |
| 10:22:49 | sean-k-mooney | Dinesh_Bhor: yes it would degrade the performance. | |
| 10:23:54 | sean-k-mooney | Dinesh_Bhor: i would speak to melwitt about this also. i belive she will be looking at how we can start to use the new keystone limmits api in nova going forward. that might help with this usecase | |
| 10:24:18 | Dinesh_Bhor | sean-k-mooney: yes, thank you so much | |
| 11:00:28 | openstackgerrit | Rong Han proposed openstack/nova master: Reset global variable after unit test is completed. https://review.openstack.org/598088 | |
| 11:17:43 | openstackgerrit | Merged openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 11:36:55 | openstackgerrit | Rong Han proposed openstack/nova master: Reset global variable after unit test is completed. https://review.openstack.org/598088 | |
| 12:04:27 | bauzas | stephenfin: https://review.openstack.org/#/c/530924/6 needs a new oslo.config version, right? | |
| 12:07:57 | bauzas | stephenfin: nevermind, I can see it's from oslo.config 5.2.0 | |
| 12:07:58 | bauzas | https://docs.openstack.org/releasenotes/oslo.config/queens.html#relnotes-5-2-0-stable-queens | |
| 12:08:18 | bauzas | and nova uses the latest https://github.com/openstack/nova/blob/master/requirements.txt#L40 | |
| 12:12:41 | zigo | bauzas: Don't worry, Rocky doesn't even build with 5.2.0 anywway ... :) | |
| 12:12:50 | zigo | Once more, requirements are just plain wrong. | |
| 12:12:56 | zigo | As usual, I'd say... | |
| 12:15:13 | zigo | Some packages are using oslo_config.sphinxconfiggen which isn't available in oslo.config 5.2.0. | |
| 12:15:28 | zigo | networking-bagpipe for example. | |
| 12:16:47 | bauzas | stephenfin: heh, I found you a new Friday nick <finucannitbacktick> :p | |
| 12:17:31 | bauzas | zigo: that's a project related issue | |
| 12:17:51 | zigo | bauzas: Ok, you need another example ... | |
| 12:17:55 | bauzas | zigo: the reviewers should look at the needed oslo version when they merge a new feature | |
| 12:18:07 | zigo | bauzas: Neutron has 1700+ unit test failures with current lower bounds ! :) | |
| 12:41:02 | openstackgerrit | Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170 | |
| 12:48:32 | stephenfin | bauzas: :D | |
| 12:48:51 | stephenfin | bauzas: People will eventually learn :) | |
| 12:49:08 | stephenfin | zigo: What do you mean, it doesn't build? | |
| 12:49:50 | stephenfin | bauzas: Also, we have oslo.config 6.1.0 in lower-constraints so I think we're all good there | |
| 12:50:02 | bauzas | yep | |
| 12:56:15 | mriedem | cdent: can you remind someone internally to re-propose the spec for this for stein? https://blueprints.launchpad.net/nova/+spec/vmware-live-migration | |
| 12:56:56 | mriedem | or you can if you want, it's just procedural | |
| 12:57:12 | mriedem | https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications | |
| 12:58:26 | cdent | mriedem: yeah, I think rado's gonna take care of it, he was on pto for a while | |
| 12:59:45 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Restart scheduler in TestNovaManagePlacementHealAllocations https://review.openstack.org/598152 | |
| 12:59:48 | brinzhang | mriedem: Could you please review this specs, https://review.openstack.org/#/c/591976/? If you have time :) | |
| 13:00:31 | mriedem | sure | |
| 13:00:42 | brinzhang | thanks ^^ | |
| 13:05:50 | cdent | mriedem: is there any new insight on the allocation thing yet, or has that not come round on the radar yet? | |
| 13:08:00 | mriedem | i think i got a failing xen ci result last night after i knocked off for the day, was going to investigate this morning | |
| 13:08:04 | mriedem | 1 sip into coffee ... | |
| 13:09:18 | openstackgerrit | Chen proposed openstack/nova master: Fix filter server list by multiple vm or task states https://review.openstack.org/598154 | |
| 13:16:04 | mriedem | alex_xu: ^ looks like a behavior change | |
| 13:17:43 | mriedem | cdent: oh i remember now, i had to re-run the xen ci patch last night b/c it wasn't picking up my dependency in depends-on: <url> form b/c it's still using zuul v2 | |
| 13:18:08 | cdent | mriedem: fun! | |
| 13:18:37 | mriedem | the logs are there in the latest failed run though | |
| 13:18:40 | mriedem | http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/13/597613/2/check/dsvm-tempest-neutron-network/cc81140/logs/screen-n-cpu.txt.gz | |
| 13:18:48 | mriedem | looking at req-99d9d496-6720-4837-a2ee-560605fd1afe | |
| 13:18:56 | mriedem | naichuans: efried: ^ | |
| 13:19:28 | mriedem | Aug 29 16:56:06.926641 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Using cpu_allocation_ratio 16.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:19:41 | mriedem | Aug 29 16:56:06.926926 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] RT: Sending compute node inventory changes back toplacement for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:19:55 | mriedem | Aug 29 16:56:06.965945 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Inventory has not changed in ProviderTree for provider: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:19:55 | mriedem | WAH WAH | |
| 13:20:23 | mriedem | hmm, but then it says it does update inventory | |
| 13:20:24 | mriedem | n_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 1.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}} | |
| 13:20:24 | mriedem | ia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {'VCPU': {'allocation_ratio': 16.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 1.5, 'total': 12795, 'reserved': 512, 'step_size': 1, | |
| 13:20:24 | mriedem | Aug 29 16:56:07.057208 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: DEBUG nova.compute.provider_tree [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updating resource provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 generation from 0 to 1 during operation: update_inventory {{(pid=24436) _update_generation /opt/stack/new/nova/nova/compute/provider_tree.py:161}} Aug 29 16:56:07.057499 dsvm-devstack-citr | |
| 13:20:48 | mriedem | there the allocation ratios are all correct | |
| 13:21:08 | mriedem | 12, 'step_size': 1, 'min_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 1.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}} {{(pid=24436) _update_inventory_attempt /opt/stack/new/nova/nova/scheduler/client/report.py:965}} | |
| 13:21:08 | mriedem | Aug 29 16:56:07.058213 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: DEBUG nova.scheduler.client.report [None req-99d9d496-6720-4837-a2ee-560605fd1afe None None] Updated inventory for 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 at generation 1: {'VCPU': {'allocation_ratio': 16.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 1.5, 'total': 12795, 'reserved | |
| 13:21:42 | cdent | goes to zero at 16:58:05.613741 | |
| 13:23:03 | cdent | right after an "Inventory has not changed in ProviderTree for provider" | |
| 13:23:14 | openstackgerrit | Radoslav Gerganov proposed openstack/nova-specs master: VMware: add support for live migration https://review.openstack.org/598163 | |
| 13:24:40 | mriedem | Aug 29 16:58:05.483508 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Using cpu_allocation_ratio 0.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:24:41 | mriedem | yeah wtf | |
| 13:25:09 | mriedem | that's in the _normalize_inventory_from_cn_obj method | |
| 13:25:21 | cdent | I'm gonna go with "something is being side-effecty" | |
| 13:25:33 | mriedem | somehow the ComputeNode.cpu_allocation_ratio is getting persisted as 0.0 maybe? | |
| 13:25:54 | cdent | you added a log for that didn't you? | |
| 13:26:00 | mriedem | yes and i don't see either of them | |
| 13:26:03 | mriedem | https://review.openstack.org/#/c/597560/3/nova/objects/compute_node.py. | |
| 13:26:50 | cdent | w & the t & the actual f | |
| 13:31:34 | cdent | write before the correct inventory is sent we have this line "Using cpu_allocation_ratio 0.0 for node [...]". that value, I would guess, is somehow being used for the _next_ inventory | |
| 13:32:22 | mriedem | i noticed that also, | |
| 13:32:26 | cdent | we're getting update inventories within 2 ms of one another. first one right, second one wrong | |
| 13:32:27 | mriedem | Aug 29 16:58:05.483508 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.resource_tracker [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Using cpu_allocation_ratio 0.0 for node: 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 | |
| 13:32:27 | mriedem | we have this with the wrong value | |
| 13:32:33 | mriedem | and we have a good update here: | |
| 13:32:38 | mriedem | , u'total': 12795, u'reserved': 512, u'step_size': 1, u'min_unit': 1, u'max_unit': 12795}, u'DISK_GB': {u'allocation_ratio': 1.0, u'total': 47, u'reserved': 0, u'step_size': 1, u'min_unit': 1, u'max_unit': 47}} | |
| 13:32:38 | mriedem | Aug 29 16:58:05.525151 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {u'VCPU': {u'allocation_ratio': 16.0, u'total': 8, u'reserved': 0, u'step_size': 1, u'min_unit': 1, u'max_unit': 8}, u'MEMORY_MB': {u'allocation_ratio': | |
| 13:32:47 | mriedem | 12795, 'reserved': 512, 'step_size': 1, 'min_unit': 1, 'max_unit': 12795}, 'DISK_GB': {'allocation_ratio': 0.0, 'total': 47, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 47}} | |
| 13:32:47 | mriedem | Aug 29 16:58:05.613741 dsvm-devstack-citrix-mia-nodepool-1379368 nova-compute[24436]: INFO nova.compute.provider_tree [None req-a869fa19-aa9d-4335-9816-42ff29b64d48 None None] Updating inventory in ProviderTree for provider 2f5a2e04-1b61-4437-ab6e-8dbbf797dc07 with inventory: {'VCPU': {'allocation_ratio': 0.0, 'total': 8, 'reserved': 0, 'step_size': 1, 'min_unit': 1, 'max_unit': 8}, 'MEMORY_MB': {'allocation_ratio': 0.0, 'tot | |
| 13:32:47 | mriedem | and then the bad update: | |
| 13:42:51 | mriedem | hmm https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L622 | |