| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-17 | |||
| 16:20:53 | awaugama | yeah | |
| 16:21:19 | awaugama | [root@compute-2 ~]# cat /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep cpu_allocation | |
| 16:21:19 | awaugama | #cpu_allocation_ratio=0.0 | |
| 16:21:19 | awaugama | cpu_allocation_ratio=1 | |
| 16:21:36 | stephenfin | Indeed. Cool, just checking | |
| 16:23:45 | bauzas | wait, just catching the discussion | |
| 16:24:04 | bauzas | awaugama: so you're saying that placement ratio != nova conf option ? | |
| 16:24:12 | awaugama | yeah | |
| 16:26:10 | bauzas | awaugama: PM me the compute logs | |
| 16:26:21 | bauzas | that's... weirdo | |
| 16:26:33 | bauzas | the only case where we bypass the ratios is when we have 0.0 | |
| 16:26:40 | bauzas | but mriedem fixed it recently | |
| 16:27:27 | mriedem | i mentioned https://github.com/openstack/nova/commit/01265b98c4cd2b1377e891a06ce748fc6f8f3425 but that's kind of a different issue | |
| 16:27:45 | mriedem | awaugama: when nova-compute starts up, it will dump the config options at debug level | |
| 16:28:00 | mriedem | are you able to see cpu_allocation_ratio in the logs after you restarted compute-2? | |
| 16:28:09 | bauzas | yup, but we also audit them IIRC | |
| 16:28:21 | bauzas | lemme double check on the gate | |
| 16:28:27 | awaugama | will check in a little bit, I have an appointment I can't miss that I have to leave for. will let you know | |
| 16:35:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fail to live migration if instance has a NUMA topology https://review.openstack.org/611088 | |
| 16:45:26 | melwitt | ||
| 16:52:18 | mriedem | what's going on with the handling a down cell stuff? seems that is blocked or languishing or other? tssurya is already gone for the day though | |
| 16:52:53 | dansmith | last I saw I think it was getting nailed by zuul failures | |
| 16:53:00 | mriedem | it also looks like people are putting more specs in runways https://etherpad.openstack.org/p/nova-runways-stein | |
| 16:53:08 | dansmith | I don't think I've seen it pop back up recently | |
| 16:53:33 | mriedem | melwitt: you might want to send a reminder to the ML that runways aren't for spec reviews | |
| 16:53:50 | mriedem | dansmith: was the full series even passing? | |
| 16:54:00 | mriedem | like last i looked i thought the change with the microversion wasn't passing tests | |
| 16:54:22 | dansmith | mriedem: I think there is one patch you're holding a -2 on that you can drop, as she moved the microversion stuff to the end and isn't doing the check-for-unmerged-microversion It hink | |
| 16:54:29 | melwitt | mriedem: ack | |
| 16:54:38 | mriedem | dansmith: yeah i can do that | |
| 16:54:47 | dansmith | mriedem: yeah the top one looks sick, but the others are fallout from zuul fails I think | |
| 16:54:52 | mriedem | since it's one of our "themes" i'm worried it's just sitting in a pile of poo | |
| 16:55:00 | mriedem | https://review.openstack.org/#/c/609807/ | |
| 16:55:31 | dansmith | oh some functional fails on one of them too | |
| 16:55:35 | mriedem | was there agreement on how exceptions would be returned when a scatter/gather fails? | |
| 16:55:44 | mriedem | https://review.openstack.org/#/c/607934/ | |
| 16:56:21 | dansmith | yeah I think so | |
| 16:56:36 | dansmith | lemme pull this down and see if I can fix up her functional failures | |
| 16:56:39 | mriedem | so rather than the sentinel, we just return an instance of the thing that failed | |
| 16:57:36 | dansmith | an instance of the exception you mean right? | |
| 16:59:24 | mriedem | right | |
| 17:00:19 | dansmith | that's what we agreed yeah | |
| 17:10:58 | mriedem | ok -1 on https://review.openstack.org/#/c/567785/ which is the api plumbing patch, | |
| 17:11:10 | mriedem | mostly b/c of tests and lack of docs and not explaining it's plumbing | |
| 17:11:18 | mriedem | but i also think it might be doing the queries wrong | |
| 17:11:27 | mriedem | but i've dropped the -2 | |
| 17:12:23 | mriedem | i'd have to look at the spec again, but it's querying instances from down cells for the user's request context project_id, | |
| 17:12:36 | mriedem | which if it's an admin listing instances for all_tenants, or a specific project_id, would be wrong | |
| 17:13:13 | dansmith | it changes that behavior? | |
| 17:13:26 | dansmith | that shoudl only be enabled if it's enabled in config and not doing all tenants | |
| 17:13:34 | dansmith | unless I'm misunderstanding what you mean | |
| 17:14:30 | macza | hello, ive run into an issue on ocata using nova with ceph, a compute hypervisor crashed and now all the instances wont connect to their drives after bringing the hypervisor back up | |
| 17:16:40 | macza | hoping i could get some help | |
| 17:19:45 | spatel | I am seeing very strange issue | |
| 17:19:58 | spatel | In /etc/default/grub hugepagesz=2M hugepages=12288 | |
| 17:20:13 | spatel | 1 instance running on it with 8G | |
| 17:20:41 | spatel | trying to spin up one more instance but getting error qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory | |
| 17:20:51 | spatel | I do have 32G memory on compute node | |
| 17:20:51 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 17:23:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 17:33:10 | spatel | nevermind, my bad i forgot to add hugepage properties to flavor | |
| 17:34:27 | cfriesen | spatel: that'll do it. :) | |
| 17:39:00 | mriedem | dansmith: i mean this https://review.openstack.org/#/c/567785/20/nova/compute/api.py@2295 | |
| 17:39:22 | mriedem | while listing instances, if there are down cells, it lists instances via the mappings from the down cells but filters on the user's request context | |
| 17:39:40 | dansmith | oh for the minimal fill ins | |
| 17:39:47 | mriedem | and if i'm an admin listing instances for another project via the project_id filter, that woudl be wrong | |
| 17:39:49 | mriedem | right | |
| 17:39:56 | dansmith | yeah I guess | |
| 17:40:09 | dansmith | I was thinking we talked about that and said "kinda meh" but I might be misremembering | |
| 17:40:20 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/handling-down-cell_new.html#edge-cases | |
| 17:40:27 | mriedem | yeah i remember we talked about it too, glad it's in the spec | |
| 17:41:10 | dansmith | um, is it? | |
| 17:41:16 | dansmith | the --deleted thing is mentioned there | |
| 17:41:21 | dansmith | --all-tenants is a little different | |
| 17:41:47 | mriedem | "Filters: If the user is listing servers using filters the results from the down cell will be skipped and no minimalistic construct will be provided since there is no way of validating the filtered results from the down cell if the value of the filter key itself is missing. " | |
| 17:41:57 | mriedem | so like if i'm filtering on status or something | |
| 17:42:09 | mriedem | we said in the spec we'd just ignore what's in down cells since we can't know | |
| 17:42:22 | dansmith | right | |
| 17:42:23 | dansmith | for filters | |
| 17:42:24 | dansmith | but your point was --all-tenants right? | |
| 17:42:31 | mriedem | that's one of them | |
| 17:42:37 | mriedem | and not called out specifically | |
| 17:42:44 | mriedem | for --all-tenants, | |
| 17:42:51 | mriedem | i'd think we could just list all instance mappings in down cells, | |
| 17:42:54 | mriedem | not filtered on project_id | |
| 17:43:03 | mriedem | up to whatever limit | |
| 17:49:31 | mriedem | so apparently filtering on project_id only works if you're also using all_tenants | |
| 17:49:37 | mriedem | otherwise we just filter on the context.project_id | |
| 17:50:42 | dansmith | um, what? | |
| 17:51:15 | dansmith | meaning --all_tenants is required in order to say --but-only-this-one ? | |
| 17:51:26 | mriedem | https://github.com/openstack/nova/blob/9c5d4eb2008df02837985779d87fedb17b4f70bf/nova/api/openstack/compute/servers.py#L206L248 | |
| 17:51:29 | mriedem | yes... | |
| 17:52:32 | dansmith | hmm | |
| 18:24:58 | openstackgerrit | Jack Ding proposed openstack/nova master: Add I/O Semaphore to limit concurrent disk ops https://review.openstack.org/609180 | |
| 18:32:31 | awaugama | mriedem: when you have a chance, it looks like the value is set on the compute node. after a reboot I saw this in the nova-compute log | |
| 18:32:34 | awaugama | 2018-10-17 17:44:00.277 1 DEBUG oslo_service.service [req-c9c4f04c-cfc6-4fe3-868d-206f9329419d - - - - -] cpu_allocation_ratio = 1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3023 | |
| 18:33:15 | mriedem | jroll: where in the ironic API reference would one find anything about conductor_groups? https://developer.openstack.org/api-ref/baremetal/ | |
| 18:33:33 | mriedem | i mean i see https://docs.openstack.org/ironic/latest/contributor/webapi-version-history.html#rocky-11-1-0 | |
| 18:33:37 | mriedem | but nothing in the API reference | |
| 18:34:21 | jroll | mriedem: apparently I forgot to update that when I added it :( | |