| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-17 | |||
| 14:24:49 | tssurya | yea we hit that too the marker stuff :( | |
| 14:24:56 | mriedem | ^ isn't marker stuff | |
| 14:25:02 | bauzas | oh this one | |
| 14:25:07 | bauzas | the requested_destination field | |
| 14:25:21 | mriedem | tssurya: https://review.openstack.org/#/c/610974/ | |
| 14:25:51 | tssurya | ah hehe yea | |
| 14:25:53 | tssurya | sorry | |
| 14:26:20 | mriedem | but yeah would be good to get that in also since it affects upgrades | |
| 14:26:24 | mriedem | johnthetubaguy: are you around? | |
| 14:26:58 | openstackgerrit | Surya Seetharaman proposed openstack/nova stable/rocky: Ignore uuid if already set in ComputeNode.update_from_virt_driver https://review.openstack.org/611337 | |
| 14:27:09 | artom | mriedem, yeah, like I said, downstream pressure :) | |
| 14:27:25 | tssurya | bauzas, efried: ^ the backport to rocky for after the master merges | |
| 14:28:11 | mriedem | +1 | |
| 14:34:26 | stephenfin | Can someone help me understand what's going on here? paste.openstack.org/show/732338/ | |
| 14:34:56 | stephenfin | Was discussing it with bauzas earlier today. We've configured 'cpu_allocation=1' and yet it still seems like overallocation is happening | |
| 14:36:04 | bauzas | stephenfin: like I said to you, we should check the instances number | |
| 14:36:49 | stephenfin | bauzas: Yeah, there's one instance on that host before we do the host-evacuate. Once we do that, two instances land on it | |
| 14:37:09 | stephenfin | One instance has 2 vcpu, the other has 1 vcpu | |
| 14:37:12 | bauzas | given vcpus_used is generated by looking up the instances | |
| 14:37:24 | bauzas | host-evacuate is racy | |
| 14:37:44 | bauzas | you should really avoid it, and do two evacuations concurrently | |
| 14:37:57 | bauzas | I thought we commented that btw. | |
| 14:38:21 | stephenfin | Yup, vcpus_used appears to be correct. Compare lines 24-29 to 90-95 | |
| 14:38:42 | bauzas | https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/contrib/host_evacuate.py | |
| 14:39:04 | stephenfin | The second hypervisor (dcfb3fb7) goes from 5/6 used to 7/6 | |
| 14:41:02 | stephenfin | bauzas: But its done client side. It seems odd that the scheduler wouldn't honour overcommit ratios just because the requests came in at the same time. Is this known behavior? | |
| 14:42:13 | mriedem | is the target host in an aggregate with an allocation ratio metadata? | |
| 14:42:38 | stephenfin | sec - checking | |
| 14:42:56 | mriedem | compute-0.localdomain was forced down but was the service actually stopped? | |
| 14:44:31 | bauzas | otp but looking | |
| 14:44:52 | mriedem | because vcpus_used didn't change on compute-0 after the evacuation | |
| 14:45:22 | mriedem | though i guess i wouldn't expect them to until compute-0 is restarted and removes locally evacuated instances | |
| 14:45:42 | mriedem | but not really safe to have the guest running on the source and dest at the same time... | |
| 14:45:48 | stephenfin | I don't see vcpus_used reported for compute-0 in those logs. Mixing it up with compute-1? | |
| 14:46:05 | mriedem | oh yeah you're right | |
| 14:46:29 | mriedem | pi | |
| 14:46:41 | mriedem | you're just wondering why they are evacuated to compute-2 rather than compute-0? | |
| 14:46:47 | mriedem | or not spread between them? | |
| 14:46:58 | stephenfin | spread between compute-1 and compute-2 | |
| 14:47:07 | stephenfin | Well | |
| 14:47:26 | mriedem | right you expect compute_0_instance_1_vcpu to go to compute-1 and compute_0_instance_2_vcpu to go to compute-2 | |
| 14:47:34 | mriedem | which would fill up each of those computes | |
| 14:47:37 | stephenfin | More that the second migration wouldn't fail because it would bump compute-2 over our overallocation ratio | |
| 14:47:45 | stephenfin | Sure, though the opposite could also happen | |
| 14:47:58 | mriedem | well, which release is this? is the CoreFilter enabled? | |
| 14:48:00 | awaugama | I'm the one who tried it, the hosts were not in aggregates | |
| 14:48:14 | stephenfin | so compute_0_instance_1_vcpu goes to compute-2 and then compute_0_instance_2_vcpu gets rejected | |
| 14:48:14 | awaugama | I can retry with that | |
| 14:48:48 | mriedem | if the CoreFilter is not enabled, there will be no cpu limits sent to the compute to claim in the RT | |
| 14:48:54 | mriedem | which means you can overcommit | |
| 14:49:21 | mriedem | if this is pike+, the filter scheduler should be claiming allocations in placement and we don't need to care about vcpu claims in the RT | |
| 14:49:34 | mriedem | so i'd take a look at your resource provider inventory/allocations | |
| 14:49:45 | mriedem | using the osc-placement commands | |
| 14:50:05 | mriedem | https://docs.openstack.org/osc-placement/latest/ | |
| 14:50:21 | awaugama | ok thanks | |
| 14:52:15 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Handle missing marker during online data migration https://review.openstack.org/611343 | |
| 14:56:57 | awaugama | mriedem: sorry I'm confused about something. you're saying placement has it's own ratios that will cause the configured compute node ratios to be ignored? | |
| 14:57:25 | dansmith | awaugama: nova's ratios are mostly irrelevant now, yes | |
| 14:58:13 | awaugama | so that's where the corefilter comes in? | |
| 14:58:51 | mriedem | well, | |
| 14:58:57 | mriedem | is the CoreFilter enabled? it shouldn't be. | |
| 14:59:07 | mriedem | if you're >=pike and using the filter scheduler | |
| 14:59:14 | mriedem | no one told me what release this is | |
| 14:59:27 | awaugama | Sorry, this is Rocky | |
| 14:59:49 | mriedem | the compute service will create a resource provider in the placement service and report it's inventory (vcpu/ram/disk) and allocation ratios for each of those resource classes | |
| 15:00:08 | mriedem | the scheduler will atomically allocate resources from each provider during the evacuate | |
| 15:00:25 | mriedem | so it should only overcommit if the VCPU inventory allocation_ratio in placement is > 1.0 | |
| 15:00:58 | mriedem | so, find the uuid for the compute-2 resource provider using "openstack resource provider list" | |
| 15:01:14 | mriedem | then using that uuid, check the inventory for that provider: openstack resource provider inventory show $uuid | |
| 15:01:28 | mriedem | sorry, openstack resource provider inventory list $uuid | |
| 15:02:20 | mriedem | if the allocation_ratio for the VCPU resource class is 1.0 and we still overcommitted, then something else is going on | |
| 15:03:53 | awaugama | is there a microversion I need to use for the openstack cli? openstack resource provider list is giving me no results | |
| 15:04:29 | mriedem | hmm, shouldn't need to | |
| 15:04:31 | mriedem | are you using admin creds? | |
| 15:04:36 | mriedem | you'd get a 403 otherwise | |
| 15:04:59 | mriedem | if there are no resource providers....you shouldn't have been able to evacuate, you'd get a NoValidHost | |
| 15:05:36 | awaugama | Sorry, I'm getting that openstack resource provider list is not an openstack command | |
| 15:05:44 | awaugama | wasn't clear with no results | |
| 15:06:11 | mriedem | you probably don't have the osc-placement plugin installed | |
| 15:06:21 | mriedem | i'm assuming you guys have an rpm for that | |
| 15:06:46 | awaugama | yeah, I've done that before, it's not enabled by default. I'll have to find my logs for that | |
| 15:07:27 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 15:15:02 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: VMware: OVA and StrOpt images as VM templates https://review.openstack.org/609736 | |
| 15:26:22 | mriedem | efried: you can drop the -2 on https://review.openstack.org/#/c/521041/ | |
| 15:26:29 | efried | ... | |
| 15:27:13 | efried | mriedem: Is this going to be filed under reshaper or vgpu? | |
| 15:27:22 | mriedem | i left a comment that it should be reshaper i think | |
| 15:29:11 | efried | If that's the case, then the -2 should be dropped once the commit message references that bp? | |
| 15:29:22 | awaugama | mriedem: The allocation ratio for VCPU is 16. that's where the issue is coming from | |
| 15:29:28 | mriedem | awaugama: aha | |
| 15:30:00 | mriedem | awaugama: that's the compute_nodes.cpu_allocation_ratio for compute-2 in the nova cell db? | |
| 15:30:05 | mriedem | efried: just -1 | |
| 15:30:14 | mriedem | awaugama: *what's | |
| 15:31:17 | awaugama | checking | |
| 15:31:22 | mriedem | you might be missing this fix https://github.com/openstack/nova/commit/01265b98c4cd2b1377e891a06ce748fc6f8f3425 | |
| 15:31:43 | mriedem | that went into 18.0.1 | |
| 15:32:39 | mriedem | if cpu_allocation_ratio is 0.0 in the compute_nodes table, it will default to 16.0 in code and that's what gets put into the resource provider VCPU inventory in placement | |
| 15:53:40 | mnaser | any stable-maint folks around to push https://review.openstack.org/#/c/611314/1 and https://review.openstack.org/#/c/611315/1 out? :) | |
| 15:58:13 | artom | Dammit what do I need to mock so that can_send_version from compute rpcapi returns what I want >_< | |
| 16:01:34 | dansmith | mnaser: yes | |