Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-14
15:55:22 mriedem yeah not pike
15:55:29 mriedem queens solves all problems, i swear
15:55:32 dansmith maybe they could backport
15:55:53 mriedem the flavor plumbing stuff isn't versioned in anyway, the placement apis to use it are,
15:55:54 cdent dansmith: traits in non-neste scenarios is working in queens last I checked
15:56:02 mriedem but you could upgrade placement on the side, and backport the nova plumbing
15:56:03 belmoreira dansmith i will have a look into that
15:56:13 mriedem cdent: cern is on ocata
15:56:29 mriedem queens placement + ocata nova (+patches) should be ok
15:56:33 mnaser ok so
15:56:40 mnaser it looks nova reserved the volume twice
15:56:41 mriedem *should* == test in pre-prod please
15:57:02 mnaser reserved it the first time succesfully, then shortly after it got reserved again
15:57:16 belmoreira mriedem :)
15:57:27 cdent mriedem: yeah, I know, it just sounded like dan was saying it's _still_ not done
15:57:42 cdent and for some limited definitions of "done" it is, a bit
15:58:00 dansmith cdent: well, tbh I didn't think we were even passing it in allocation_candidates :)
15:58:08 dansmith I just hadn't been paying enough attention
15:58:10 mriedem i know https://github.com/openstack/nova/commit/477a14e81e8c69b91a39395d78d845c8603e10fd#diff-b4a2926790df668d47bf16d43128c4cf
15:58:13 mnaser so 12 seconds later, cinder got a second reserve request for the same volume and thats why it failed (i guess a volume goes to attaching state when its being reserved)
15:58:51 mriedem mnaser: yeah it does,
15:59:00 mriedem so i'm guessing the 2nd reserve request was made by nova-compute...
15:59:14 mriedem looking
15:59:21 mnaser im gonna investigate and see if that theory adds up
15:59:29 mriedem this is where compute starts doing the attach work https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L1595
16:00:01 belmoreira thanks all
16:00:27 mriedem mnaser: so what's probably happening is it's hitting an ocata compute
16:00:33 mriedem which calls check_attach again, and fails
16:00:46 mriedem https://github.com/openstack/nova/blob/stable/ocata/nova/virt/block_device.py#L250
16:01:05 mriedem mnaser: that's why the API has that min compute service version check, but that was busted with the multi-cell cells v2 stuff, unless you have melwitt's patch
16:01:10 mriedem mnaser: so i think that's your issue
16:01:16 mriedem can you verify the compute it failed on is running ocata?
16:01:28 mnaser mriedem: that's what i'm double checing
16:01:32 mnaser checking*
16:05:18 mnaser mriedem: i wonder if that was the wsgi registration bug we ran into a while back
16:05:38 mnaser "MariaDB [nova]> select * from services where deleted_at is null order by version;" shows everything at 22 except for 3 control plane services
16:05:45 mnaser "nova-osapi_compute"
16:05:56 mnaser which is at version 16
16:05:59 mriedem mnaser: the problem is w/o that patch, the version comes back at 0
16:06:17 mriedem mnaser: your nova-api config is probably pointing [database] at cell0?
16:06:49 mnaser mriedem: oh you're right that's an old record, updated_at '2017-09-02 15:34:18'
16:07:10 mriedem if your api config is pointing the [database] entry at nova_cell0, there are no nova-compute services in cell0, so it will return 0 as the version
16:07:17 mriedem and hit that else block in the api code
16:07:38 mriedem in other words, the api wasn't properly iterating non-cell0 cells for nova-compute service versions
16:08:35 mnaser makes a lot of sense
16:09:01 mriedem so that api patch will make your stuff to the right thing while you still have ocata computes
16:09:17 mnaser we shouldn't have ocata computes so im still tracking down that issue right now
16:19:02 mnaser mriedem: i think i might have another theory.. heat creates a volume, tries to create an instance, instance fails creation for some reason (but after reserving volume), it retries, and nova fails
16:19:20 mnaser because im seeing a POST to /os-volumes_boot returning 400
16:20:11 mriedem hmm
16:20:21 mnaser ill have to dig up heat logs
16:21:23 mriedem mnaser: yeah we apparently don't unreserve the volume in the api if something fails later in the api
16:22:26 mriedem or in compute
16:23:32 mnaser interesting.. i see heat doing the create twice in logs 10 seconds apart, let me see if i can find an error there
16:23:38 mnaser but yeah, it's masking a failure
16:23:41 mriedem does heat delete the failed instance (which reserved the volume) before trying to create the 2nd instance?
16:23:51 mnaser digging into that now
16:24:07 mriedem because that would likely be related to https://review.openstack.org/#/c/340614/
16:24:50 mnaser ok there was a scheduler error (not enough hosts), i'll dig into that
16:25:00 mnaser "Task pause timed out"
16:25:06 mnaser and then it tried to create again
16:25:16 mriedem i don't know what that is,
16:25:21 mriedem but for the scheduler novalidhost,
16:25:42 mriedem the api would reserve the volume, rpc cast to conductor which rpc calls the scheduler for a host, that raises NoValidHost, and the instance is put into ERROR state, and the volume would still be reserved
16:25:43 mnaser the "Task pause timed out" is a heat thing so i'll look into why it didn't attempt to delete before retry
16:26:15 mriedem so if heat deletes the failed instance, nova can clean up the volume, but that's probably where we need https://review.openstack.org/#/c/340614/ i think
16:27:07 mnaser mriedem: i think heat's normal behaviour is to just set the stack to failed
16:27:11 mnaser and not retry
16:27:29 mnaser googling "Task pause timed out" yields nothing so that's a nice start, but we're in heat problem land
16:28:12 mriedem ok, but even if heat did try to delete the instance before retrying, the 2nd attempt will likely fail because the volume is already reserved because of the bug that https://review.openstack.org/#/c/340614/ is trying to fix
16:30:42 cfriesen does anyone know off the top of their head which operations would cause an instance with a qcow2-backed rootfs (that normally has the glance image file as a backing file) to get flattened?
16:31:30 mnaser mriedem: yeah that's ideal, looks like melwitt was doing work on that, i can pick it up and work on whats missing if melwitt is busy with other things
16:32:28 mriedem mnaser: it's ready for review, just needs review
16:32:36 mriedem that patch has a long history
16:32:44 mriedem of merging, breaking hte gate, getting reverted, repeat
16:33:07 mnaser mriedem: there was a comment suggesting removing instance.host is None that's why i asked
16:33:34 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_access_get_by_flavor_id db api method https://review.openstack.org/539720
16:33:35 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_destroy db api method https://review.openstack.org/539729
16:33:35 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_get_by_flavor_id db api method https://review.openstack.org/539733
16:33:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_get_by_name db api method https://review.openstack.org/544379
16:47:03 cfriesen mriedem: do you know of any work in progress to deal with https://bugs.launchpad.net/nova/+bug/1583145 or https://bugs.launchpad.net/nova/+bug/1644248 ?
16:47:05 openstack Launchpad bug 1644248 in OpenStack Compute (nova) ocata "duplicate for #1583145 Nova incorrectly tracks live migration progress" [High,In progress] - Assigned to Matt Riedemann (mriedem)
16:47:06 openstack Launchpad bug 1644248 in OpenStack Compute (nova) ocata "Nova incorrectly tracks live migration progress" [High,In progress] - Assigned to Matt Riedemann (mriedem)
16:50:39 jaypipes mriedem_afk, dansmith: https://review.openstack.org/#/c/539712/ is pretty straightforward.
16:59:41 dansmith melwitt: mriedem_afk isn't around for the cells meeting, we talked to tssurya and belmiro earlier and neither are online now, so I assume we're canceling
17:13:14 cdent jaypipes: if you're chasing easy wins: https://review.openstack.org/#/c/536784/
17:52:29 openstackgerrit Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/544282
18:09:23 mnaser so it looks like this bug wasn't visible because we had old disabled but inactive pre-pike compute nodes
18:09:35 mnaser removing them made the minimum pike and landed us in this situation
18:15:00 openstackgerrit Dan Smith proposed openstack/nova-specs master: Add placement-req-filter spec https://review.openstack.org/544585
18:16:48 mnaser "The missing 'reserve_volume' call is added to the BFV flow. In case of build failure the volume will be locked in 'attaching' state until the instance in ERROR state is cleaned up." -- i guess the problem here is that it doesn't account for scheduling failures :x
18:31:25 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/544277
18:32:31 openstackgerrit Dan Smith proposed openstack/nova-specs master: Add placement-req-filter spec https://review.openstack.org/544585
18:32:51 mriedem_afk mnaser: i want to say that "so it looks like this bug wasn't visible because we had old disabled but inactive pre-pike compute nodes" was also a bug fix
18:33:57 mriedem mnaser: oh nvm we excluded forced_down services, but we do include disabled services in that min version check
18:34:13 mriedem i was thinking of this so it looks like this bug wasn't visible because we had old disabled but inactive pre-pike compute nodes
18:34:16 mriedem oops
18:34:19 mriedem https://github.com/openstack/nova/commit/c79770e615799cd4457ac603dcad4fb3452fe2bc
18:35:52 mnaser mriedem: yeah so right now i'm not too sure why this occurs because it looks like if it fails to schedule, it means when nova attempts to delete it, it does a local delete, which calls '_local_cleanup_bdm_volumes'
18:36:06 mnaser or maybe im wrong?

Earlier   Later