| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-17 | |||
| 16:11:31 | mriedem | unless we have some code in rocky that is using a cache and thinks we don't have any updates... | |
| 16:11:51 | mriedem | i.e. the cache might say we already have that allocation ratio even though placement doesn't | |
| 16:12:08 | awaugama | stephenfin: is that the code you saw earlier? | |
| 16:12:23 | stephenfin | awaugama: Yup, that's the one | |
| 16:12:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/587013 | |
| 16:12:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014 | |
| 16:12:32 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conductor: Recreate volume attachments during a reschedule https://review.openstack.org/587071 | |
| 16:13:15 | mriedem | have you tried restarting the compute-2 nova-compute service to see if that flushes the cpu_allocation_ratio 1.0 change to placement? | |
| 16:13:25 | stephenfin | mriedem: If I'm reading that right, if we send that once, we'll never be able to update it, right? | |
| 16:13:43 | mriedem | stephenfin: we should be able to update it any time we change config and restart nova-compute | |
| 16:13:43 | stephenfin | Because placement will always return 'allocation_ratio' once it's been set https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L105 | |
| 16:14:09 | mriedem | the allocation ratio config is read and set on the compute node object here https://github.com/openstack/nova/blob/18.0.0/nova/compute/resource_tracker.py#L624 | |
| 16:14:17 | mriedem | no, | |
| 16:14:22 | mriedem | that cpu_inv is coming from the virt driver, | |
| 16:14:33 | mriedem | and none of the virt driver's have an allocation_ratio in their inventory dicts | |
| 16:14:44 | stephenfin | Correct. My mistake | |
| 16:14:45 | mriedem | :param inv_data: Dict, keyed by resource class, of inventory information returned from virt driver's get_inventory() method | |
| 16:15:00 | mriedem | so try restarting compute-2's nova-compute service | |
| 16:15:16 | mriedem | if you changed the config but never restarted the service, the change wouldn't get to placement, | |
| 16:15:33 | mriedem | if you did, but something is f'ed in the local provider tree cache, that should refresh the cache | |
| 16:15:47 | mriedem | the provider tree cache stuff is all black magic to me | |
| 16:17:32 | awaugama | let's give it a shot | |
| 16:19:58 | awaugama | No luck. I'm still seeing 16 set for placement for compute node 2 | |
| 16:20:44 | stephenfin | awaugama: Dumb question but this config option is configured on the compute node 2 and not the controller, right? | |
| 16:20:53 | awaugama | yeah | |
| 16:21:19 | awaugama | [root@compute-2 ~]# cat /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep cpu_allocation | |
| 16:21:19 | awaugama | #cpu_allocation_ratio=0.0 | |
| 16:21:19 | awaugama | cpu_allocation_ratio=1 | |
| 16:21:36 | stephenfin | Indeed. Cool, just checking | |
| 16:23:45 | bauzas | wait, just catching the discussion | |
| 16:24:04 | bauzas | awaugama: so you're saying that placement ratio != nova conf option ? | |
| 16:24:12 | awaugama | yeah | |
| 16:26:10 | bauzas | awaugama: PM me the compute logs | |
| 16:26:21 | bauzas | that's... weirdo | |
| 16:26:33 | bauzas | the only case where we bypass the ratios is when we have 0.0 | |
| 16:26:40 | bauzas | but mriedem fixed it recently | |
| 16:27:27 | mriedem | i mentioned https://github.com/openstack/nova/commit/01265b98c4cd2b1377e891a06ce748fc6f8f3425 but that's kind of a different issue | |
| 16:27:45 | mriedem | awaugama: when nova-compute starts up, it will dump the config options at debug level | |
| 16:28:00 | mriedem | are you able to see cpu_allocation_ratio in the logs after you restarted compute-2? | |
| 16:28:09 | bauzas | yup, but we also audit them IIRC | |
| 16:28:21 | bauzas | lemme double check on the gate | |
| 16:28:27 | awaugama | will check in a little bit, I have an appointment I can't miss that I have to leave for. will let you know | |
| 16:35:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fail to live migration if instance has a NUMA topology https://review.openstack.org/611088 | |
| 16:45:26 | melwitt | ||
| 16:52:18 | mriedem | what's going on with the handling a down cell stuff? seems that is blocked or languishing or other? tssurya is already gone for the day though | |
| 16:52:53 | dansmith | last I saw I think it was getting nailed by zuul failures | |
| 16:53:00 | mriedem | it also looks like people are putting more specs in runways https://etherpad.openstack.org/p/nova-runways-stein | |
| 16:53:08 | dansmith | I don't think I've seen it pop back up recently | |
| 16:53:33 | mriedem | melwitt: you might want to send a reminder to the ML that runways aren't for spec reviews | |
| 16:53:50 | mriedem | dansmith: was the full series even passing? | |
| 16:54:00 | mriedem | like last i looked i thought the change with the microversion wasn't passing tests | |
| 16:54:22 | dansmith | mriedem: I think there is one patch you're holding a -2 on that you can drop, as she moved the microversion stuff to the end and isn't doing the check-for-unmerged-microversion It hink | |
| 16:54:29 | melwitt | mriedem: ack | |
| 16:54:38 | mriedem | dansmith: yeah i can do that | |
| 16:54:47 | dansmith | mriedem: yeah the top one looks sick, but the others are fallout from zuul fails I think | |
| 16:54:52 | mriedem | since it's one of our "themes" i'm worried it's just sitting in a pile of poo | |
| 16:55:00 | mriedem | https://review.openstack.org/#/c/609807/ | |
| 16:55:31 | dansmith | oh some functional fails on one of them too | |
| 16:55:35 | mriedem | was there agreement on how exceptions would be returned when a scatter/gather fails? | |
| 16:55:44 | mriedem | https://review.openstack.org/#/c/607934/ | |
| 16:56:21 | dansmith | yeah I think so | |
| 16:56:36 | dansmith | lemme pull this down and see if I can fix up her functional failures | |
| 16:56:39 | mriedem | so rather than the sentinel, we just return an instance of the thing that failed | |
| 16:57:36 | dansmith | an instance of the exception you mean right? | |
| 16:59:24 | mriedem | right | |
| 17:00:19 | dansmith | that's what we agreed yeah | |
| 17:10:58 | mriedem | ok -1 on https://review.openstack.org/#/c/567785/ which is the api plumbing patch, | |
| 17:11:10 | mriedem | mostly b/c of tests and lack of docs and not explaining it's plumbing | |
| 17:11:18 | mriedem | but i also think it might be doing the queries wrong | |
| 17:11:27 | mriedem | but i've dropped the -2 | |
| 17:12:23 | mriedem | i'd have to look at the spec again, but it's querying instances from down cells for the user's request context project_id, | |
| 17:12:36 | mriedem | which if it's an admin listing instances for all_tenants, or a specific project_id, would be wrong | |
| 17:13:13 | dansmith | it changes that behavior? | |
| 17:13:26 | dansmith | that shoudl only be enabled if it's enabled in config and not doing all tenants | |
| 17:13:34 | dansmith | unless I'm misunderstanding what you mean | |
| 17:14:30 | macza | hello, ive run into an issue on ocata using nova with ceph, a compute hypervisor crashed and now all the instances wont connect to their drives after bringing the hypervisor back up | |
| 17:16:40 | macza | hoping i could get some help | |
| 17:19:45 | spatel | I am seeing very strange issue | |
| 17:19:58 | spatel | In /etc/default/grub hugepagesz=2M hugepages=12288 | |
| 17:20:13 | spatel | 1 instance running on it with 8G | |
| 17:20:41 | spatel | trying to spin up one more instance but getting error qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory | |
| 17:20:51 | spatel | I do have 32G memory on compute node | |
| 17:20:51 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 17:23:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 17:33:10 | spatel | nevermind, my bad i forgot to add hugepage properties to flavor | |
| 17:34:27 | cfriesen | spatel: that'll do it. :) | |
| 17:39:00 | mriedem | dansmith: i mean this https://review.openstack.org/#/c/567785/20/nova/compute/api.py@2295 | |
| 17:39:22 | mriedem | while listing instances, if there are down cells, it lists instances via the mappings from the down cells but filters on the user's request context | |
| 17:39:40 | dansmith | oh for the minimal fill ins | |
| 17:39:47 | mriedem | and if i'm an admin listing instances for another project via the project_id filter, that woudl be wrong | |
| 17:39:49 | mriedem | right | |
| 17:39:56 | dansmith | yeah I guess | |
| 17:40:09 | dansmith | I was thinking we talked about that and said "kinda meh" but I might be misremembering | |
| 17:40:20 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/handling-down-cell_new.html#edge-cases | |
| 17:40:27 | mriedem | yeah i remember we talked about it too, glad it's in the spec | |
| 17:41:10 | dansmith | um, is it? | |
| 17:41:16 | dansmith | the --deleted thing is mentioned there | |
| 17:41:21 | dansmith | --all-tenants is a little different | |
| 17:41:47 | mriedem | "Filters: If the user is listing servers using filters the results from the down cell will be skipped and no minimalistic construct will be provided since there is no way of validating the filtered results from the down cell if the value of the filter key itself is missing. " | |
| 17:41:57 | mriedem | so like if i'm filtering on status or something | |