| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-14 | |||
| 15:48:09 | mriedem | mnaser: do you see anything in the cinder logs for that volume id before nova tries to reserve it? | |
| 15:48:19 | dansmith | belmoreira: let me think on it a bit | |
| 15:48:20 | mnaser | mriedem: good question, i should have thought of that | |
| 15:48:52 | belmoreira | dansmith but for Pike the only option will be the have a "normal" aggregate per cell and let the filter scheduler do the job. concerned about the performance though | |
| 15:49:09 | mriedem | mnaser: also this is likely something you want in pike https://github.com/openstack/nova/commit/6d1877bf1da098b3be17fbd2dd5e53eecd2f048c | |
| 15:49:17 | dansmith | belmoreira: yeah, although you can have more schedulers and parallelize that load | |
| 15:49:17 | mriedem | should be in 16.1.0 | |
| 15:50:04 | mnaser | oh that's interesting, that might be related | |
| 15:50:25 | belmoreira | yes, that's the plan. but in the worst case scenario each scheduler will need to go through 9k nodes | |
| 15:50:51 | belmoreira | that's why in ocata i decided to have local placements | |
| 15:51:00 | mriedem | mnaser: yeah you might be hiting this https://github.com/openstack/nova/blob/stable/pike/nova/compute/api.py#L1406 | |
| 15:51:12 | mriedem | so nova isn't actually trying to call cinder to reserve the volume | |
| 15:51:12 | dansmith | belmoreira: aye, well, might need a hack for that case in pike until we think of something else | |
| 15:51:16 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add check for redundant import aliases https://review.openstack.org/543995 | |
| 15:51:19 | mriedem | it's doing it's own conditional state check and failure | |
| 15:51:21 | mriedem | *failing | |
| 15:51:33 | mriedem | although i don't know why the volume status would be 'attaching' | |
| 15:52:17 | mnaser | interestingly enough i see a POST to /action | |
| 15:52:27 | dansmith | belmoreira: what's the reasoning for that split? was it for scheduling initially, or security or hardware ownership? | |
| 15:53:52 | belmoreira | dansmith manly specific hardware for a use-case | |
| 15:54:11 | dansmith | belmoreira: okay well that should really be a trait thing (although that's not done either :) | |
| 15:54:18 | mnaser | ok so i see a volume create in cinder-volume | |
| 15:54:40 | mriedem | dansmith: well, | |
| 15:54:41 | belmoreira | dansmith until now we isolate the hardware type per cell and allocate the right project to it | |
| 15:54:51 | mriedem | we do have traits-based scheduling, | |
| 15:55:01 | mriedem | so you could have per-project flavors that have a trait for the hw you want for those projects | |
| 15:55:06 | dansmith | mriedem: we're not passing it to placement yet right? | |
| 15:55:09 | mriedem | we are | |
| 15:55:11 | mriedem | in queens | |
| 15:55:12 | dansmith | in pike? | |
| 15:55:14 | dansmith | right okay | |
| 15:55:15 | mriedem | oh | |
| 15:55:22 | mriedem | yeah not pike | |
| 15:55:29 | mriedem | queens solves all problems, i swear | |
| 15:55:32 | dansmith | maybe they could backport | |
| 15:55:53 | mriedem | the flavor plumbing stuff isn't versioned in anyway, the placement apis to use it are, | |
| 15:55:54 | cdent | dansmith: traits in non-neste scenarios is working in queens last I checked | |
| 15:56:02 | mriedem | but you could upgrade placement on the side, and backport the nova plumbing | |
| 15:56:03 | belmoreira | dansmith i will have a look into that | |
| 15:56:13 | mriedem | cdent: cern is on ocata | |
| 15:56:29 | mriedem | queens placement + ocata nova (+patches) should be ok | |
| 15:56:33 | mnaser | ok so | |
| 15:56:40 | mnaser | it looks nova reserved the volume twice | |
| 15:56:41 | mriedem | *should* == test in pre-prod please | |
| 15:57:02 | mnaser | reserved it the first time succesfully, then shortly after it got reserved again | |
| 15:57:16 | belmoreira | mriedem :) | |
| 15:57:27 | cdent | mriedem: yeah, I know, it just sounded like dan was saying it's _still_ not done | |
| 15:57:42 | cdent | and for some limited definitions of "done" it is, a bit | |
| 15:58:00 | dansmith | cdent: well, tbh I didn't think we were even passing it in allocation_candidates :) | |
| 15:58:08 | dansmith | I just hadn't been paying enough attention | |
| 15:58:10 | mriedem | i know https://github.com/openstack/nova/commit/477a14e81e8c69b91a39395d78d845c8603e10fd#diff-b4a2926790df668d47bf16d43128c4cf | |
| 15:58:13 | mnaser | so 12 seconds later, cinder got a second reserve request for the same volume and thats why it failed (i guess a volume goes to attaching state when its being reserved) | |
| 15:58:51 | mriedem | mnaser: yeah it does, | |
| 15:59:00 | mriedem | so i'm guessing the 2nd reserve request was made by nova-compute... | |
| 15:59:14 | mriedem | looking | |
| 15:59:21 | mnaser | im gonna investigate and see if that theory adds up | |
| 15:59:29 | mriedem | this is where compute starts doing the attach work https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L1595 | |
| 16:00:01 | belmoreira | thanks all | |
| 16:00:27 | mriedem | mnaser: so what's probably happening is it's hitting an ocata compute | |
| 16:00:33 | mriedem | which calls check_attach again, and fails | |
| 16:00:46 | mriedem | https://github.com/openstack/nova/blob/stable/ocata/nova/virt/block_device.py#L250 | |
| 16:01:05 | mriedem | mnaser: that's why the API has that min compute service version check, but that was busted with the multi-cell cells v2 stuff, unless you have melwitt's patch | |
| 16:01:10 | mriedem | mnaser: so i think that's your issue | |
| 16:01:16 | mriedem | can you verify the compute it failed on is running ocata? | |
| 16:01:28 | mnaser | mriedem: that's what i'm double checing | |
| 16:01:32 | mnaser | checking* | |
| 16:05:18 | mnaser | mriedem: i wonder if that was the wsgi registration bug we ran into a while back | |
| 16:05:38 | mnaser | "MariaDB [nova]> select * from services where deleted_at is null order by version;" shows everything at 22 except for 3 control plane services | |
| 16:05:45 | mnaser | "nova-osapi_compute" | |
| 16:05:56 | mnaser | which is at version 16 | |
| 16:05:59 | mriedem | mnaser: the problem is w/o that patch, the version comes back at 0 | |
| 16:06:17 | mriedem | mnaser: your nova-api config is probably pointing [database] at cell0? | |
| 16:06:49 | mnaser | mriedem: oh you're right that's an old record, updated_at '2017-09-02 15:34:18' | |
| 16:07:10 | mriedem | if your api config is pointing the [database] entry at nova_cell0, there are no nova-compute services in cell0, so it will return 0 as the version | |
| 16:07:17 | mriedem | and hit that else block in the api code | |
| 16:07:38 | mriedem | in other words, the api wasn't properly iterating non-cell0 cells for nova-compute service versions | |
| 16:08:35 | mnaser | makes a lot of sense | |
| 16:09:01 | mriedem | so that api patch will make your stuff to the right thing while you still have ocata computes | |
| 16:09:17 | mnaser | we shouldn't have ocata computes so im still tracking down that issue right now | |
| 16:19:02 | mnaser | mriedem: i think i might have another theory.. heat creates a volume, tries to create an instance, instance fails creation for some reason (but after reserving volume), it retries, and nova fails | |
| 16:19:20 | mnaser | because im seeing a POST to /os-volumes_boot returning 400 | |
| 16:20:11 | mriedem | hmm | |
| 16:20:21 | mnaser | ill have to dig up heat logs | |
| 16:21:23 | mriedem | mnaser: yeah we apparently don't unreserve the volume in the api if something fails later in the api | |
| 16:22:26 | mriedem | or in compute | |
| 16:23:32 | mnaser | interesting.. i see heat doing the create twice in logs 10 seconds apart, let me see if i can find an error there | |
| 16:23:38 | mnaser | but yeah, it's masking a failure | |
| 16:23:41 | mriedem | does heat delete the failed instance (which reserved the volume) before trying to create the 2nd instance? | |
| 16:23:51 | mnaser | digging into that now | |
| 16:24:07 | mriedem | because that would likely be related to https://review.openstack.org/#/c/340614/ | |
| 16:24:50 | mnaser | ok there was a scheduler error (not enough hosts), i'll dig into that | |
| 16:25:00 | mnaser | "Task pause timed out" | |
| 16:25:06 | mnaser | and then it tried to create again | |
| 16:25:16 | mriedem | i don't know what that is, | |
| 16:25:21 | mriedem | but for the scheduler novalidhost, | |
| 16:25:42 | mriedem | the api would reserve the volume, rpc cast to conductor which rpc calls the scheduler for a host, that raises NoValidHost, and the instance is put into ERROR state, and the volume would still be reserved | |
| 16:25:43 | mnaser | the "Task pause timed out" is a heat thing so i'll look into why it didn't attempt to delete before retry | |
| 16:26:15 | mriedem | so if heat deletes the failed instance, nova can clean up the volume, but that's probably where we need https://review.openstack.org/#/c/340614/ i think | |
| 16:27:07 | mnaser | mriedem: i think heat's normal behaviour is to just set the stack to failed | |
| 16:27:11 | mnaser | and not retry | |