| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-17 | |||
| 15:31:22 | mriedem | you might be missing this fix https://github.com/openstack/nova/commit/01265b98c4cd2b1377e891a06ce748fc6f8f3425 | |
| 15:31:43 | mriedem | that went into 18.0.1 | |
| 15:32:39 | mriedem | if cpu_allocation_ratio is 0.0 in the compute_nodes table, it will default to 16.0 in code and that's what gets put into the resource provider VCPU inventory in placement | |
| 15:53:40 | mnaser | any stable-maint folks around to push https://review.openstack.org/#/c/611314/1 and https://review.openstack.org/#/c/611315/1 out? :) | |
| 15:58:13 | artom | Dammit what do I need to mock so that can_send_version from compute rpcapi returns what I want >_< | |
| 16:01:34 | dansmith | mnaser: yes | |
| 16:01:59 | mnaser | dansmith: thank you, didn't want to keep carrying a patch :) | |
| 16:06:41 | efried | artom: Have you tried mock.patch('oslo_messaging.rpc.client.RPCClient.can_send_version') ? | |
| 16:08:05 | efried | artom: Or you could mock nova.rpc.get_client and then set its return_value.can_send_version.return_value | |
| 16:08:13 | efried | depending how far back you want to go. | |
| 16:08:50 | artom | efried, so, I was listing the mock_calls incorrectly, turns out mocking oslo_messaging.RPClient.can_send_version is what I need | |
| 16:09:01 | mriedem | another clean rocky backport ready to go https://review.openstack.org/#/c/611327/ | |
| 16:09:14 | artom | I can see the call I care about in here. New problem: it seems to be having no effect :( | |
| 16:09:14 | efried | artom: cool. | |
| 16:09:27 | efried | with that, I must leave you. | |
| 16:09:33 | artom | efried, hehe, thanks :) enjoy | |
| 16:10:03 | awaugama | mriedem: cpu_allocation_ratio is set to 1 in the db | |
| 16:10:10 | awaugama | sorry got yanked into a meeting | |
| 16:10:16 | mriedem | awaugama: hmm | |
| 16:10:22 | awaugama | for compute node 2 | |
| 16:11:04 | mriedem | well this is the code that sets the cpu_allocation_ratio in the inventory record we send to placement https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L106 | |
| 16:11:31 | mriedem | unless we have some code in rocky that is using a cache and thinks we don't have any updates... | |
| 16:11:51 | mriedem | i.e. the cache might say we already have that allocation ratio even though placement doesn't | |
| 16:12:08 | awaugama | stephenfin: is that the code you saw earlier? | |
| 16:12:23 | stephenfin | awaugama: Yup, that's the one | |
| 16:12:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/587013 | |
| 16:12:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014 | |
| 16:12:32 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conductor: Recreate volume attachments during a reschedule https://review.openstack.org/587071 | |
| 16:13:15 | mriedem | have you tried restarting the compute-2 nova-compute service to see if that flushes the cpu_allocation_ratio 1.0 change to placement? | |
| 16:13:25 | stephenfin | mriedem: If I'm reading that right, if we send that once, we'll never be able to update it, right? | |
| 16:13:43 | mriedem | stephenfin: we should be able to update it any time we change config and restart nova-compute | |
| 16:13:43 | stephenfin | Because placement will always return 'allocation_ratio' once it's been set https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L105 | |
| 16:14:09 | mriedem | the allocation ratio config is read and set on the compute node object here https://github.com/openstack/nova/blob/18.0.0/nova/compute/resource_tracker.py#L624 | |
| 16:14:17 | mriedem | no, | |
| 16:14:22 | mriedem | that cpu_inv is coming from the virt driver, | |
| 16:14:33 | mriedem | and none of the virt driver's have an allocation_ratio in their inventory dicts | |
| 16:14:44 | stephenfin | Correct. My mistake | |
| 16:14:45 | mriedem | :param inv_data: Dict, keyed by resource class, of inventory information returned from virt driver's get_inventory() method | |
| 16:15:00 | mriedem | so try restarting compute-2's nova-compute service | |
| 16:15:16 | mriedem | if you changed the config but never restarted the service, the change wouldn't get to placement, | |
| 16:15:33 | mriedem | if you did, but something is f'ed in the local provider tree cache, that should refresh the cache | |
| 16:15:47 | mriedem | the provider tree cache stuff is all black magic to me | |
| 16:17:32 | awaugama | let's give it a shot | |
| 16:19:58 | awaugama | No luck. I'm still seeing 16 set for placement for compute node 2 | |
| 16:20:44 | stephenfin | awaugama: Dumb question but this config option is configured on the compute node 2 and not the controller, right? | |
| 16:20:53 | awaugama | yeah | |
| 16:21:19 | awaugama | [root@compute-2 ~]# cat /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep cpu_allocation | |
| 16:21:19 | awaugama | #cpu_allocation_ratio=0.0 | |
| 16:21:19 | awaugama | cpu_allocation_ratio=1 | |
| 16:21:36 | stephenfin | Indeed. Cool, just checking | |
| 16:23:45 | bauzas | wait, just catching the discussion | |
| 16:24:04 | bauzas | awaugama: so you're saying that placement ratio != nova conf option ? | |
| 16:24:12 | awaugama | yeah | |
| 16:26:10 | bauzas | awaugama: PM me the compute logs | |
| 16:26:21 | bauzas | that's... weirdo | |
| 16:26:33 | bauzas | the only case where we bypass the ratios is when we have 0.0 | |
| 16:26:40 | bauzas | but mriedem fixed it recently | |
| 16:27:27 | mriedem | i mentioned https://github.com/openstack/nova/commit/01265b98c4cd2b1377e891a06ce748fc6f8f3425 but that's kind of a different issue | |
| 16:27:45 | mriedem | awaugama: when nova-compute starts up, it will dump the config options at debug level | |
| 16:28:00 | mriedem | are you able to see cpu_allocation_ratio in the logs after you restarted compute-2? | |
| 16:28:09 | bauzas | yup, but we also audit them IIRC | |
| 16:28:21 | bauzas | lemme double check on the gate | |
| 16:28:27 | awaugama | will check in a little bit, I have an appointment I can't miss that I have to leave for. will let you know | |
| 16:35:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fail to live migration if instance has a NUMA topology https://review.openstack.org/611088 | |
| 16:45:26 | melwitt | ||
| 16:52:18 | mriedem | what's going on with the handling a down cell stuff? seems that is blocked or languishing or other? tssurya is already gone for the day though | |
| 16:52:53 | dansmith | last I saw I think it was getting nailed by zuul failures | |
| 16:53:00 | mriedem | it also looks like people are putting more specs in runways https://etherpad.openstack.org/p/nova-runways-stein | |
| 16:53:08 | dansmith | I don't think I've seen it pop back up recently | |
| 16:53:33 | mriedem | melwitt: you might want to send a reminder to the ML that runways aren't for spec reviews | |
| 16:53:50 | mriedem | dansmith: was the full series even passing? | |
| 16:54:00 | mriedem | like last i looked i thought the change with the microversion wasn't passing tests | |
| 16:54:22 | dansmith | mriedem: I think there is one patch you're holding a -2 on that you can drop, as she moved the microversion stuff to the end and isn't doing the check-for-unmerged-microversion It hink | |
| 16:54:29 | melwitt | mriedem: ack | |
| 16:54:38 | mriedem | dansmith: yeah i can do that | |
| 16:54:47 | dansmith | mriedem: yeah the top one looks sick, but the others are fallout from zuul fails I think | |
| 16:54:52 | mriedem | since it's one of our "themes" i'm worried it's just sitting in a pile of poo | |
| 16:55:00 | mriedem | https://review.openstack.org/#/c/609807/ | |
| 16:55:31 | dansmith | oh some functional fails on one of them too | |
| 16:55:35 | mriedem | was there agreement on how exceptions would be returned when a scatter/gather fails? | |
| 16:55:44 | mriedem | https://review.openstack.org/#/c/607934/ | |
| 16:56:21 | dansmith | yeah I think so | |
| 16:56:36 | dansmith | lemme pull this down and see if I can fix up her functional failures | |
| 16:56:39 | mriedem | so rather than the sentinel, we just return an instance of the thing that failed | |
| 16:57:36 | dansmith | an instance of the exception you mean right? | |
| 16:59:24 | mriedem | right | |
| 17:00:19 | dansmith | that's what we agreed yeah | |
| 17:10:58 | mriedem | ok -1 on https://review.openstack.org/#/c/567785/ which is the api plumbing patch, | |
| 17:11:10 | mriedem | mostly b/c of tests and lack of docs and not explaining it's plumbing | |
| 17:11:18 | mriedem | but i also think it might be doing the queries wrong | |
| 17:11:27 | mriedem | but i've dropped the -2 | |
| 17:12:23 | mriedem | i'd have to look at the spec again, but it's querying instances from down cells for the user's request context project_id, | |
| 17:12:36 | mriedem | which if it's an admin listing instances for all_tenants, or a specific project_id, would be wrong | |
| 17:13:13 | dansmith | it changes that behavior? | |
| 17:13:26 | dansmith | that shoudl only be enabled if it's enabled in config and not doing all tenants | |
| 17:13:34 | dansmith | unless I'm misunderstanding what you mean | |
| 17:14:30 | macza | hello, ive run into an issue on ocata using nova with ceph, a compute hypervisor crashed and now all the instances wont connect to their drives after bringing the hypervisor back up | |
| 17:16:40 | macza | hoping i could get some help | |
| 17:19:45 | spatel | I am seeing very strange issue | |
| 17:19:58 | spatel | In /etc/default/grub hugepagesz=2M hugepages=12288 | |