| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-17 | |||
| 14:50:21 | awaugama | ok thanks | |
| 14:52:15 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Handle missing marker during online data migration https://review.openstack.org/611343 | |
| 14:56:57 | awaugama | mriedem: sorry I'm confused about something. you're saying placement has it's own ratios that will cause the configured compute node ratios to be ignored? | |
| 14:57:25 | dansmith | awaugama: nova's ratios are mostly irrelevant now, yes | |
| 14:58:13 | awaugama | so that's where the corefilter comes in? | |
| 14:58:51 | mriedem | well, | |
| 14:58:57 | mriedem | is the CoreFilter enabled? it shouldn't be. | |
| 14:59:07 | mriedem | if you're >=pike and using the filter scheduler | |
| 14:59:14 | mriedem | no one told me what release this is | |
| 14:59:27 | awaugama | Sorry, this is Rocky | |
| 14:59:49 | mriedem | the compute service will create a resource provider in the placement service and report it's inventory (vcpu/ram/disk) and allocation ratios for each of those resource classes | |
| 15:00:08 | mriedem | the scheduler will atomically allocate resources from each provider during the evacuate | |
| 15:00:25 | mriedem | so it should only overcommit if the VCPU inventory allocation_ratio in placement is > 1.0 | |
| 15:00:58 | mriedem | so, find the uuid for the compute-2 resource provider using "openstack resource provider list" | |
| 15:01:14 | mriedem | then using that uuid, check the inventory for that provider: openstack resource provider inventory show $uuid | |
| 15:01:28 | mriedem | sorry, openstack resource provider inventory list $uuid | |
| 15:02:20 | mriedem | if the allocation_ratio for the VCPU resource class is 1.0 and we still overcommitted, then something else is going on | |
| 15:03:53 | awaugama | is there a microversion I need to use for the openstack cli? openstack resource provider list is giving me no results | |
| 15:04:29 | mriedem | hmm, shouldn't need to | |
| 15:04:31 | mriedem | are you using admin creds? | |
| 15:04:36 | mriedem | you'd get a 403 otherwise | |
| 15:04:59 | mriedem | if there are no resource providers....you shouldn't have been able to evacuate, you'd get a NoValidHost | |
| 15:05:36 | awaugama | Sorry, I'm getting that openstack resource provider list is not an openstack command | |
| 15:05:44 | awaugama | wasn't clear with no results | |
| 15:06:11 | mriedem | you probably don't have the osc-placement plugin installed | |
| 15:06:21 | mriedem | i'm assuming you guys have an rpm for that | |
| 15:06:46 | awaugama | yeah, I've done that before, it's not enabled by default. I'll have to find my logs for that | |
| 15:07:27 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 15:15:02 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: VMware: OVA and StrOpt images as VM templates https://review.openstack.org/609736 | |
| 15:26:22 | mriedem | efried: you can drop the -2 on https://review.openstack.org/#/c/521041/ | |
| 15:26:29 | efried | ... | |
| 15:27:13 | efried | mriedem: Is this going to be filed under reshaper or vgpu? | |
| 15:27:22 | mriedem | i left a comment that it should be reshaper i think | |
| 15:29:11 | efried | If that's the case, then the -2 should be dropped once the commit message references that bp? | |
| 15:29:22 | awaugama | mriedem: The allocation ratio for VCPU is 16. that's where the issue is coming from | |
| 15:29:28 | mriedem | awaugama: aha | |
| 15:30:00 | mriedem | awaugama: that's the compute_nodes.cpu_allocation_ratio for compute-2 in the nova cell db? | |
| 15:30:05 | mriedem | efried: just -1 | |
| 15:30:14 | mriedem | awaugama: *what's | |
| 15:31:17 | awaugama | checking | |
| 15:31:22 | mriedem | you might be missing this fix https://github.com/openstack/nova/commit/01265b98c4cd2b1377e891a06ce748fc6f8f3425 | |
| 15:31:43 | mriedem | that went into 18.0.1 | |
| 15:32:39 | mriedem | if cpu_allocation_ratio is 0.0 in the compute_nodes table, it will default to 16.0 in code and that's what gets put into the resource provider VCPU inventory in placement | |
| 15:53:40 | mnaser | any stable-maint folks around to push https://review.openstack.org/#/c/611314/1 and https://review.openstack.org/#/c/611315/1 out? :) | |
| 15:58:13 | artom | Dammit what do I need to mock so that can_send_version from compute rpcapi returns what I want >_< | |
| 16:01:34 | dansmith | mnaser: yes | |
| 16:01:59 | mnaser | dansmith: thank you, didn't want to keep carrying a patch :) | |
| 16:06:41 | efried | artom: Have you tried mock.patch('oslo_messaging.rpc.client.RPCClient.can_send_version') ? | |
| 16:08:05 | efried | artom: Or you could mock nova.rpc.get_client and then set its return_value.can_send_version.return_value | |
| 16:08:13 | efried | depending how far back you want to go. | |
| 16:08:50 | artom | efried, so, I was listing the mock_calls incorrectly, turns out mocking oslo_messaging.RPClient.can_send_version is what I need | |
| 16:09:01 | mriedem | another clean rocky backport ready to go https://review.openstack.org/#/c/611327/ | |
| 16:09:14 | artom | I can see the call I care about in here. New problem: it seems to be having no effect :( | |
| 16:09:14 | efried | artom: cool. | |
| 16:09:27 | efried | with that, I must leave you. | |
| 16:09:33 | artom | efried, hehe, thanks :) enjoy | |
| 16:10:03 | awaugama | mriedem: cpu_allocation_ratio is set to 1 in the db | |
| 16:10:10 | awaugama | sorry got yanked into a meeting | |
| 16:10:16 | mriedem | awaugama: hmm | |
| 16:10:22 | awaugama | for compute node 2 | |
| 16:11:04 | mriedem | well this is the code that sets the cpu_allocation_ratio in the inventory record we send to placement https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L106 | |
| 16:11:31 | mriedem | unless we have some code in rocky that is using a cache and thinks we don't have any updates... | |
| 16:11:51 | mriedem | i.e. the cache might say we already have that allocation ratio even though placement doesn't | |
| 16:12:08 | awaugama | stephenfin: is that the code you saw earlier? | |
| 16:12:23 | stephenfin | awaugama: Yup, that's the one | |
| 16:12:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/587013 | |
| 16:12:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014 | |
| 16:12:32 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conductor: Recreate volume attachments during a reschedule https://review.openstack.org/587071 | |
| 16:13:15 | mriedem | have you tried restarting the compute-2 nova-compute service to see if that flushes the cpu_allocation_ratio 1.0 change to placement? | |
| 16:13:25 | stephenfin | mriedem: If I'm reading that right, if we send that once, we'll never be able to update it, right? | |
| 16:13:43 | mriedem | stephenfin: we should be able to update it any time we change config and restart nova-compute | |
| 16:13:43 | stephenfin | Because placement will always return 'allocation_ratio' once it's been set https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L105 | |
| 16:14:09 | mriedem | the allocation ratio config is read and set on the compute node object here https://github.com/openstack/nova/blob/18.0.0/nova/compute/resource_tracker.py#L624 | |
| 16:14:17 | mriedem | no, | |
| 16:14:22 | mriedem | that cpu_inv is coming from the virt driver, | |
| 16:14:33 | mriedem | and none of the virt driver's have an allocation_ratio in their inventory dicts | |
| 16:14:44 | stephenfin | Correct. My mistake | |
| 16:14:45 | mriedem | :param inv_data: Dict, keyed by resource class, of inventory information returned from virt driver's get_inventory() method | |
| 16:15:00 | mriedem | so try restarting compute-2's nova-compute service | |
| 16:15:16 | mriedem | if you changed the config but never restarted the service, the change wouldn't get to placement, | |
| 16:15:33 | mriedem | if you did, but something is f'ed in the local provider tree cache, that should refresh the cache | |
| 16:15:47 | mriedem | the provider tree cache stuff is all black magic to me | |
| 16:17:32 | awaugama | let's give it a shot | |
| 16:19:58 | awaugama | No luck. I'm still seeing 16 set for placement for compute node 2 | |
| 16:20:44 | stephenfin | awaugama: Dumb question but this config option is configured on the compute node 2 and not the controller, right? | |
| 16:20:53 | awaugama | yeah | |
| 16:21:19 | awaugama | [root@compute-2 ~]# cat /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep cpu_allocation | |
| 16:21:19 | awaugama | #cpu_allocation_ratio=0.0 | |
| 16:21:19 | awaugama | cpu_allocation_ratio=1 | |
| 16:21:36 | stephenfin | Indeed. Cool, just checking | |
| 16:23:45 | bauzas | wait, just catching the discussion | |
| 16:24:04 | bauzas | awaugama: so you're saying that placement ratio != nova conf option ? | |
| 16:24:12 | awaugama | yeah | |
| 16:26:10 | bauzas | awaugama: PM me the compute logs | |
| 16:26:21 | bauzas | that's... weirdo | |
| 16:26:33 | bauzas | the only case where we bypass the ratios is when we have 0.0 | |
| 16:26:40 | bauzas | but mriedem fixed it recently | |
| 16:27:27 | mriedem | i mentioned https://github.com/openstack/nova/commit/01265b98c4cd2b1377e891a06ce748fc6f8f3425 but that's kind of a different issue | |
| 16:27:45 | mriedem | awaugama: when nova-compute starts up, it will dump the config options at debug level | |
| 16:28:00 | mriedem | are you able to see cpu_allocation_ratio in the logs after you restarted compute-2? | |