| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-17 | |||
| 17:12:23 | mriedem | i'd have to look at the spec again, but it's querying instances from down cells for the user's request context project_id, | |
| 17:12:36 | mriedem | which if it's an admin listing instances for all_tenants, or a specific project_id, would be wrong | |
| 17:13:13 | dansmith | it changes that behavior? | |
| 17:13:26 | dansmith | that shoudl only be enabled if it's enabled in config and not doing all tenants | |
| 17:13:34 | dansmith | unless I'm misunderstanding what you mean | |
| 17:14:30 | macza | hello, ive run into an issue on ocata using nova with ceph, a compute hypervisor crashed and now all the instances wont connect to their drives after bringing the hypervisor back up | |
| 17:16:40 | macza | hoping i could get some help | |
| 17:19:45 | spatel | I am seeing very strange issue | |
| 17:19:58 | spatel | In /etc/default/grub hugepagesz=2M hugepages=12288 | |
| 17:20:13 | spatel | 1 instance running on it with 8G | |
| 17:20:41 | spatel | trying to spin up one more instance but getting error qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory | |
| 17:20:51 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 17:20:51 | spatel | I do have 32G memory on compute node | |
| 17:23:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 17:33:10 | spatel | nevermind, my bad i forgot to add hugepage properties to flavor | |
| 17:34:27 | cfriesen | spatel: that'll do it. :) | |
| 17:39:00 | mriedem | dansmith: i mean this https://review.openstack.org/#/c/567785/20/nova/compute/api.py@2295 | |
| 17:39:22 | mriedem | while listing instances, if there are down cells, it lists instances via the mappings from the down cells but filters on the user's request context | |
| 17:39:40 | dansmith | oh for the minimal fill ins | |
| 17:39:47 | mriedem | and if i'm an admin listing instances for another project via the project_id filter, that woudl be wrong | |
| 17:39:49 | mriedem | right | |
| 17:39:56 | dansmith | yeah I guess | |
| 17:40:09 | dansmith | I was thinking we talked about that and said "kinda meh" but I might be misremembering | |
| 17:40:20 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/handling-down-cell_new.html#edge-cases | |
| 17:40:27 | mriedem | yeah i remember we talked about it too, glad it's in the spec | |
| 17:41:10 | dansmith | um, is it? | |
| 17:41:16 | dansmith | the --deleted thing is mentioned there | |
| 17:41:21 | dansmith | --all-tenants is a little different | |
| 17:41:47 | mriedem | "Filters: If the user is listing servers using filters the results from the down cell will be skipped and no minimalistic construct will be provided since there is no way of validating the filtered results from the down cell if the value of the filter key itself is missing. " | |
| 17:41:57 | mriedem | so like if i'm filtering on status or something | |
| 17:42:09 | mriedem | we said in the spec we'd just ignore what's in down cells since we can't know | |
| 17:42:22 | dansmith | right | |
| 17:42:23 | dansmith | for filters | |
| 17:42:24 | dansmith | but your point was --all-tenants right? | |
| 17:42:31 | mriedem | that's one of them | |
| 17:42:37 | mriedem | and not called out specifically | |
| 17:42:44 | mriedem | for --all-tenants, | |
| 17:42:51 | mriedem | i'd think we could just list all instance mappings in down cells, | |
| 17:42:54 | mriedem | not filtered on project_id | |
| 17:43:03 | mriedem | up to whatever limit | |
| 17:49:31 | mriedem | so apparently filtering on project_id only works if you're also using all_tenants | |
| 17:49:37 | mriedem | otherwise we just filter on the context.project_id | |
| 17:50:42 | dansmith | um, what? | |
| 17:51:15 | dansmith | meaning --all_tenants is required in order to say --but-only-this-one ? | |
| 17:51:26 | mriedem | https://github.com/openstack/nova/blob/9c5d4eb2008df02837985779d87fedb17b4f70bf/nova/api/openstack/compute/servers.py#L206L248 | |
| 17:51:29 | mriedem | yes... | |
| 17:52:32 | dansmith | hmm | |
| 18:24:58 | openstackgerrit | Jack Ding proposed openstack/nova master: Add I/O Semaphore to limit concurrent disk ops https://review.openstack.org/609180 | |
| 18:32:31 | awaugama | mriedem: when you have a chance, it looks like the value is set on the compute node. after a reboot I saw this in the nova-compute log | |
| 18:32:34 | awaugama | 2018-10-17 17:44:00.277 1 DEBUG oslo_service.service [req-c9c4f04c-cfc6-4fe3-868d-206f9329419d - - - - -] cpu_allocation_ratio = 1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3023 | |
| 18:33:15 | mriedem | jroll: where in the ironic API reference would one find anything about conductor_groups? https://developer.openstack.org/api-ref/baremetal/ | |
| 18:33:33 | mriedem | i mean i see https://docs.openstack.org/ironic/latest/contributor/webapi-version-history.html#rocky-11-1-0 | |
| 18:33:37 | mriedem | but nothing in the API reference | |
| 18:34:21 | jroll | mriedem: apparently I forgot to update that when I added it :( | |
| 18:34:26 | mriedem | awaugama: ok so it's 1.0 in config, it's 1.0 in the compute_nodes.cpu_allocation_ratio column in the db, but it's 16.0 in the associated resource provider VCPU inventory in placement | |
| 18:34:36 | awaugama | yeah | |
| 18:34:46 | mriedem | well wtf | |
| 18:34:49 | jroll | mriedem: it would be in node CRUD as its own field, I'll get that done real quick | |
| 18:34:56 | mriedem | jroll: a uuid or what? | |
| 18:35:33 | mriedem | jroll: and you can update a node's conductor_group? | |
| 18:35:34 | jroll | mriedem: a string, up to 255 characters IIRC: https://github.com/openstack/ironic/blob/b8ffcc0f0298fca5b4b36ad016e2c3b2f0e81710/ironic/common/utils.py#L530 | |
| 18:35:45 | mriedem | ok so it's just some tag | |
| 18:35:50 | awaugama | I'll sit down with sylvain tomorrow and we can do some debugging, will let you know if we find anything | |
| 18:35:51 | mriedem | special tag | |
| 18:35:52 | jroll | yeah | |
| 18:36:10 | jroll | alphanumeric, plus - _ . | |
| 18:42:25 | artom | - _ . is what I look like after a few drinks | |
| 18:42:52 | jroll | hah | |
| 18:53:13 | jroll | mriedem: api-ref for you https://review.openstack.org/611415 | |
| 18:53:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: Re-propose numa-aware-live-migration spec https://review.openstack.org/599587 | |
| 18:58:05 | mriedem | jroll: i've brought the wrath | |
| 18:58:47 | dansmith | hrm, pretty sure this functional timeout on the down cell series is a real deadlock on our cell cache | |
| 18:59:11 | jroll | mriedem: thanks, valid points | |
| 18:59:46 | jroll | for context I haven't touched our API ref in a long time :P | |
| 19:00:26 | mriedem | ugh | |
| 19:00:39 | mriedem | so GET /v1/nodes/detail is deprecated for GET /v1/nodes?detail=True, | |
| 19:01:00 | mriedem | but the request filter and response parameters for the latter don't mention anything possible in the former | |
| 19:08:37 | jroll | I'm not sure it's even properly deprecated | |
| 19:08:58 | jroll | added a note to the detail=True parameter | |
| 19:29:28 | mriedem | so we're deprecating the force flag from the evacuate and live migration apis, | |
| 19:29:40 | mriedem | wouldn't it behoove us to deprecate that as an option from nova commands as well? | |
| 19:29:49 | mriedem | or at least doc it up real good that you shouldn't use it? | |
| 19:32:50 | artom | mriedem, wait, deprecate or remove? | |
| 19:33:01 | melwitt | dansmith: would appreciate your review on mah backport https://review.openstack.org/610673 | |
| 19:33:05 | artom | Because for removal the usual novaclient microversion stuff applies, no? | |
| 19:34:08 | mriedem | artom: if we don't want people using the force flag to live migrate or evacuate an instance, | |
| 19:34:17 | mriedem | so much so that we're deprecating the api parameter, | |
| 19:34:26 | mriedem | you could argue that we should not have it in the CLI either | |
| 19:34:31 | mriedem | like, at all | |
| 19:34:34 | mriedem | even for older microversions | |
| 19:34:43 | artom | mriedem, but it still exists for old microversions | |
| 19:34:45 | artom | In the API | |
| 19:34:47 | mriedem | yes i know | |
| 19:34:52 | artom | So, we have to keep client support | |
| 19:35:02 | mriedem | not really | |
| 19:35:03 | artom | So if they specifically --os-compute-version <old>, they have it | |
| 19:35:06 | artom | Otherwise, it's gone | |
| 19:35:36 | mriedem | once all allocations are nested, you won't be able to force at all | |
| 19:35:43 | mriedem | regardless of microversion | |
| 19:35:52 | mriedem | anyway, it was just a thought | |