Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-17
14:37:57 bauzas I thought we commented that btw.
14:38:21 stephenfin Yup, vcpus_used appears to be correct. Compare lines 24-29 to 90-95
14:38:42 bauzas https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/contrib/host_evacuate.py
14:39:04 stephenfin The second hypervisor (dcfb3fb7) goes from 5/6 used to 7/6
14:41:02 stephenfin bauzas: But its done client side. It seems odd that the scheduler wouldn't honour overcommit ratios just because the requests came in at the same time. Is this known behavior?
14:42:13 mriedem is the target host in an aggregate with an allocation ratio metadata?
14:42:38 stephenfin sec - checking
14:42:56 mriedem compute-0.localdomain was forced down but was the service actually stopped?
14:44:31 bauzas otp but looking
14:44:52 mriedem because vcpus_used didn't change on compute-0 after the evacuation
14:45:22 mriedem though i guess i wouldn't expect them to until compute-0 is restarted and removes locally evacuated instances
14:45:42 mriedem but not really safe to have the guest running on the source and dest at the same time...
14:45:48 stephenfin I don't see vcpus_used reported for compute-0 in those logs. Mixing it up with compute-1?
14:46:05 mriedem oh yeah you're right
14:46:29 mriedem pi
14:46:41 mriedem you're just wondering why they are evacuated to compute-2 rather than compute-0?
14:46:47 mriedem or not spread between them?
14:46:58 stephenfin spread between compute-1 and compute-2
14:47:07 stephenfin Well
14:47:26 mriedem right you expect compute_0_instance_1_vcpu to go to compute-1 and compute_0_instance_2_vcpu to go to compute-2
14:47:34 mriedem which would fill up each of those computes
14:47:37 stephenfin More that the second migration wouldn't fail because it would bump compute-2 over our overallocation ratio
14:47:45 stephenfin Sure, though the opposite could also happen
14:47:58 mriedem well, which release is this? is the CoreFilter enabled?
14:48:00 awaugama I'm the one who tried it, the hosts were not in aggregates
14:48:14 awaugama I can retry with that
14:48:14 stephenfin so compute_0_instance_1_vcpu goes to compute-2 and then compute_0_instance_2_vcpu gets rejected
14:48:48 mriedem if the CoreFilter is not enabled, there will be no cpu limits sent to the compute to claim in the RT
14:48:54 mriedem which means you can overcommit
14:49:21 mriedem if this is pike+, the filter scheduler should be claiming allocations in placement and we don't need to care about vcpu claims in the RT
14:49:34 mriedem so i'd take a look at your resource provider inventory/allocations
14:49:45 mriedem using the osc-placement commands
14:50:05 mriedem https://docs.openstack.org/osc-placement/latest/
14:50:21 awaugama ok thanks
14:52:15 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Handle missing marker during online data migration https://review.openstack.org/611343
14:56:57 awaugama mriedem: sorry I'm confused about something. you're saying placement has it's own ratios that will cause the configured compute node ratios to be ignored?
14:57:25 dansmith awaugama: nova's ratios are mostly irrelevant now, yes
14:58:13 awaugama so that's where the corefilter comes in?
14:58:51 mriedem well,
14:58:57 mriedem is the CoreFilter enabled? it shouldn't be.
14:59:07 mriedem if you're >=pike and using the filter scheduler
14:59:14 mriedem no one told me what release this is
14:59:27 awaugama Sorry, this is Rocky
14:59:49 mriedem the compute service will create a resource provider in the placement service and report it's inventory (vcpu/ram/disk) and allocation ratios for each of those resource classes
15:00:08 mriedem the scheduler will atomically allocate resources from each provider during the evacuate
15:00:25 mriedem so it should only overcommit if the VCPU inventory allocation_ratio in placement is > 1.0
15:00:58 mriedem so, find the uuid for the compute-2 resource provider using "openstack resource provider list"
15:01:14 mriedem then using that uuid, check the inventory for that provider: openstack resource provider inventory show $uuid
15:01:28 mriedem sorry, openstack resource provider inventory list $uuid
15:02:20 mriedem if the allocation_ratio for the VCPU resource class is 1.0 and we still overcommitted, then something else is going on
15:03:53 awaugama is there a microversion I need to use for the openstack cli? openstack resource provider list is giving me no results
15:04:29 mriedem hmm, shouldn't need to
15:04:31 mriedem are you using admin creds?
15:04:36 mriedem you'd get a 403 otherwise
15:04:59 mriedem if there are no resource providers....you shouldn't have been able to evacuate, you'd get a NoValidHost
15:05:36 awaugama Sorry, I'm getting that openstack resource provider list is not an openstack command
15:05:44 awaugama wasn't clear with no results
15:06:11 mriedem you probably don't have the osc-placement plugin installed
15:06:21 mriedem i'm assuming you guys have an rpm for that
15:06:46 awaugama yeah, I've done that before, it's not enabled by default. I'll have to find my logs for that
15:07:27 openstackgerrit Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174
15:15:02 openstackgerrit Ivaylo Mitev proposed openstack/nova master: VMware: OVA and StrOpt images as VM templates https://review.openstack.org/609736
15:26:22 mriedem efried: you can drop the -2 on https://review.openstack.org/#/c/521041/
15:26:29 efried ...
15:27:13 efried mriedem: Is this going to be filed under reshaper or vgpu?
15:27:22 mriedem i left a comment that it should be reshaper i think
15:29:11 efried If that's the case, then the -2 should be dropped once the commit message references that bp?
15:29:22 awaugama mriedem: The allocation ratio for VCPU is 16. that's where the issue is coming from
15:29:28 mriedem awaugama: aha
15:30:00 mriedem awaugama: that's the compute_nodes.cpu_allocation_ratio for compute-2 in the nova cell db?
15:30:05 mriedem efried: just -1
15:30:14 mriedem awaugama: *what's
15:31:17 awaugama checking
15:31:22 mriedem you might be missing this fix https://github.com/openstack/nova/commit/01265b98c4cd2b1377e891a06ce748fc6f8f3425
15:31:43 mriedem that went into 18.0.1
15:32:39 mriedem if cpu_allocation_ratio is 0.0 in the compute_nodes table, it will default to 16.0 in code and that's what gets put into the resource provider VCPU inventory in placement
15:53:40 mnaser any stable-maint folks around to push https://review.openstack.org/#/c/611314/1 and https://review.openstack.org/#/c/611315/1 out? :)
15:58:13 artom Dammit what do I need to mock so that can_send_version from compute rpcapi returns what I want >_<
16:01:34 dansmith mnaser: yes
16:01:59 mnaser dansmith: thank you, didn't want to keep carrying a patch :)
16:06:41 efried artom: Have you tried mock.patch('oslo_messaging.rpc.client.RPCClient.can_send_version') ?
16:08:05 efried artom: Or you could mock nova.rpc.get_client and then set its return_value.can_send_version.return_value
16:08:13 efried depending how far back you want to go.
16:08:50 artom efried, so, I was listing the mock_calls incorrectly, turns out mocking oslo_messaging.RPClient.can_send_version is what I need
16:09:01 mriedem another clean rocky backport ready to go https://review.openstack.org/#/c/611327/
16:09:14 efried artom: cool.
16:09:14 artom I can see the call I care about in here. New problem: it seems to be having no effect :(
16:09:27 efried with that, I must leave you.
16:09:33 artom efried, hehe, thanks :) enjoy
16:10:03 awaugama mriedem: cpu_allocation_ratio is set to 1 in the db
16:10:10 awaugama sorry got yanked into a meeting
16:10:16 mriedem awaugama: hmm
16:10:22 awaugama for compute node 2
16:11:04 mriedem well this is the code that sets the cpu_allocation_ratio in the inventory record we send to placement https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L106
16:11:31 mriedem unless we have some code in rocky that is using a cache and thinks we don't have any updates...
16:11:51 mriedem i.e. the cache might say we already have that allocation ratio even though placement doesn't
16:12:08 awaugama stephenfin: is that the code you saw earlier?
16:12:23 stephenfin awaugama: Yup, that's the one
16:12:31 openstackgerrit Stephen Finucane proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014
16:12:31 openstackgerrit Stephen Finucane proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/587013

Earlier   Later