| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-17 | |||
| 16:53:33 | mriedem | melwitt: you might want to send a reminder to the ML that runways aren't for spec reviews | |
| 16:53:50 | mriedem | dansmith: was the full series even passing? | |
| 16:54:00 | mriedem | like last i looked i thought the change with the microversion wasn't passing tests | |
| 16:54:22 | dansmith | mriedem: I think there is one patch you're holding a -2 on that you can drop, as she moved the microversion stuff to the end and isn't doing the check-for-unmerged-microversion It hink | |
| 16:54:29 | melwitt | mriedem: ack | |
| 16:54:38 | mriedem | dansmith: yeah i can do that | |
| 16:54:47 | dansmith | mriedem: yeah the top one looks sick, but the others are fallout from zuul fails I think | |
| 16:54:52 | mriedem | since it's one of our "themes" i'm worried it's just sitting in a pile of poo | |
| 16:55:00 | mriedem | https://review.openstack.org/#/c/609807/ | |
| 16:55:31 | dansmith | oh some functional fails on one of them too | |
| 16:55:35 | mriedem | was there agreement on how exceptions would be returned when a scatter/gather fails? | |
| 16:55:44 | mriedem | https://review.openstack.org/#/c/607934/ | |
| 16:56:21 | dansmith | yeah I think so | |
| 16:56:36 | dansmith | lemme pull this down and see if I can fix up her functional failures | |
| 16:56:39 | mriedem | so rather than the sentinel, we just return an instance of the thing that failed | |
| 16:57:36 | dansmith | an instance of the exception you mean right? | |
| 16:59:24 | mriedem | right | |
| 17:00:19 | dansmith | that's what we agreed yeah | |
| 17:10:58 | mriedem | ok -1 on https://review.openstack.org/#/c/567785/ which is the api plumbing patch, | |
| 17:11:10 | mriedem | mostly b/c of tests and lack of docs and not explaining it's plumbing | |
| 17:11:18 | mriedem | but i also think it might be doing the queries wrong | |
| 17:11:27 | mriedem | but i've dropped the -2 | |
| 17:12:23 | mriedem | i'd have to look at the spec again, but it's querying instances from down cells for the user's request context project_id, | |
| 17:12:36 | mriedem | which if it's an admin listing instances for all_tenants, or a specific project_id, would be wrong | |
| 17:13:13 | dansmith | it changes that behavior? | |
| 17:13:26 | dansmith | that shoudl only be enabled if it's enabled in config and not doing all tenants | |
| 17:13:34 | dansmith | unless I'm misunderstanding what you mean | |
| 17:14:30 | macza | hello, ive run into an issue on ocata using nova with ceph, a compute hypervisor crashed and now all the instances wont connect to their drives after bringing the hypervisor back up | |
| 17:16:40 | macza | hoping i could get some help | |
| 17:19:45 | spatel | I am seeing very strange issue | |
| 17:19:58 | spatel | In /etc/default/grub hugepagesz=2M hugepages=12288 | |
| 17:20:13 | spatel | 1 instance running on it with 8G | |
| 17:20:41 | spatel | trying to spin up one more instance but getting error qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory | |
| 17:20:51 | spatel | I do have 32G memory on compute node | |
| 17:20:51 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 17:23:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 17:33:10 | spatel | nevermind, my bad i forgot to add hugepage properties to flavor | |
| 17:34:27 | cfriesen | spatel: that'll do it. :) | |
| 17:39:00 | mriedem | dansmith: i mean this https://review.openstack.org/#/c/567785/20/nova/compute/api.py@2295 | |
| 17:39:22 | mriedem | while listing instances, if there are down cells, it lists instances via the mappings from the down cells but filters on the user's request context | |
| 17:39:40 | dansmith | oh for the minimal fill ins | |
| 17:39:47 | mriedem | and if i'm an admin listing instances for another project via the project_id filter, that woudl be wrong | |
| 17:39:49 | mriedem | right | |
| 17:39:56 | dansmith | yeah I guess | |
| 17:40:09 | dansmith | I was thinking we talked about that and said "kinda meh" but I might be misremembering | |
| 17:40:20 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/handling-down-cell_new.html#edge-cases | |
| 17:40:27 | mriedem | yeah i remember we talked about it too, glad it's in the spec | |
| 17:41:10 | dansmith | um, is it? | |
| 17:41:16 | dansmith | the --deleted thing is mentioned there | |
| 17:41:21 | dansmith | --all-tenants is a little different | |
| 17:41:47 | mriedem | "Filters: If the user is listing servers using filters the results from the down cell will be skipped and no minimalistic construct will be provided since there is no way of validating the filtered results from the down cell if the value of the filter key itself is missing. " | |
| 17:41:57 | mriedem | so like if i'm filtering on status or something | |
| 17:42:09 | mriedem | we said in the spec we'd just ignore what's in down cells since we can't know | |
| 17:42:22 | dansmith | right | |
| 17:42:23 | dansmith | for filters | |
| 17:42:24 | dansmith | but your point was --all-tenants right? | |
| 17:42:31 | mriedem | that's one of them | |
| 17:42:37 | mriedem | and not called out specifically | |
| 17:42:44 | mriedem | for --all-tenants, | |
| 17:42:51 | mriedem | i'd think we could just list all instance mappings in down cells, | |
| 17:42:54 | mriedem | not filtered on project_id | |
| 17:43:03 | mriedem | up to whatever limit | |
| 17:49:31 | mriedem | so apparently filtering on project_id only works if you're also using all_tenants | |
| 17:49:37 | mriedem | otherwise we just filter on the context.project_id | |
| 17:50:42 | dansmith | um, what? | |
| 17:51:15 | dansmith | meaning --all_tenants is required in order to say --but-only-this-one ? | |
| 17:51:26 | mriedem | https://github.com/openstack/nova/blob/9c5d4eb2008df02837985779d87fedb17b4f70bf/nova/api/openstack/compute/servers.py#L206L248 | |
| 17:51:29 | mriedem | yes... | |
| 17:52:32 | dansmith | hmm | |
| 18:24:58 | openstackgerrit | Jack Ding proposed openstack/nova master: Add I/O Semaphore to limit concurrent disk ops https://review.openstack.org/609180 | |
| 18:32:31 | awaugama | mriedem: when you have a chance, it looks like the value is set on the compute node. after a reboot I saw this in the nova-compute log | |
| 18:32:34 | awaugama | 2018-10-17 17:44:00.277 1 DEBUG oslo_service.service [req-c9c4f04c-cfc6-4fe3-868d-206f9329419d - - - - -] cpu_allocation_ratio = 1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3023 | |
| 18:33:15 | mriedem | jroll: where in the ironic API reference would one find anything about conductor_groups? https://developer.openstack.org/api-ref/baremetal/ | |
| 18:33:33 | mriedem | i mean i see https://docs.openstack.org/ironic/latest/contributor/webapi-version-history.html#rocky-11-1-0 | |
| 18:33:37 | mriedem | but nothing in the API reference | |
| 18:34:21 | jroll | mriedem: apparently I forgot to update that when I added it :( | |
| 18:34:26 | mriedem | awaugama: ok so it's 1.0 in config, it's 1.0 in the compute_nodes.cpu_allocation_ratio column in the db, but it's 16.0 in the associated resource provider VCPU inventory in placement | |
| 18:34:36 | awaugama | yeah | |
| 18:34:46 | mriedem | well wtf | |
| 18:34:49 | jroll | mriedem: it would be in node CRUD as its own field, I'll get that done real quick | |
| 18:34:56 | mriedem | jroll: a uuid or what? | |
| 18:35:33 | mriedem | jroll: and you can update a node's conductor_group? | |
| 18:35:34 | jroll | mriedem: a string, up to 255 characters IIRC: https://github.com/openstack/ironic/blob/b8ffcc0f0298fca5b4b36ad016e2c3b2f0e81710/ironic/common/utils.py#L530 | |
| 18:35:45 | mriedem | ok so it's just some tag | |
| 18:35:50 | awaugama | I'll sit down with sylvain tomorrow and we can do some debugging, will let you know if we find anything | |
| 18:35:51 | mriedem | special tag | |
| 18:35:52 | jroll | yeah | |
| 18:36:10 | jroll | alphanumeric, plus - _ . | |
| 18:42:25 | artom | - _ . is what I look like after a few drinks | |
| 18:42:52 | jroll | hah | |
| 18:53:13 | jroll | mriedem: api-ref for you https://review.openstack.org/611415 | |
| 18:53:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: Re-propose numa-aware-live-migration spec https://review.openstack.org/599587 | |
| 18:58:05 | mriedem | jroll: i've brought the wrath | |
| 18:58:47 | dansmith | hrm, pretty sure this functional timeout on the down cell series is a real deadlock on our cell cache | |
| 18:59:11 | jroll | mriedem: thanks, valid points | |
| 18:59:46 | jroll | for context I haven't touched our API ref in a long time :P | |
| 19:00:26 | mriedem | ugh | |
| 19:00:39 | mriedem | so GET /v1/nodes/detail is deprecated for GET /v1/nodes?detail=True, | |
| 19:01:00 | mriedem | but the request filter and response parameters for the latter don't mention anything possible in the former | |
| 19:08:37 | jroll | I'm not sure it's even properly deprecated | |