Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-14
15:42:35 mdbooth stephenfin: Generally good advice.
15:42:47 belmoreira dansmith placement aggregates?
15:42:50 stephenfin mdbooth: https://review.openstack.org/#/c/538217/3/specs/rocky/approved/integrate-mypy-type-checking.rst
15:42:51 dansmith belmoreira: aye
15:43:00 mdbooth stephenfin: Haha
15:43:04 mnaser mriedem: i believe the control plane is stable/pike, i think computes might be a bit older i'm not sure if they're 16.0.4
15:43:06 jaypipes dansmith: aggregate filtering isn't supported yet in allocation candidates...
15:43:14 jaypipes dansmith: or at least, I don't think it is..
15:43:23 dansmith jaypipes: I'm saying that's the plan
15:43:26 mnaser we're running into a lot of weird conditions with heat
15:43:27 jaypipes ah
15:43:29 belmoreira dansmith can they the used in Pike?
15:43:30 dansmith jaypipes: we're not asking for it now anyway
15:43:33 mdbooth stephenfin: Spit shine that turd ;)
15:43:33 dansmith belmoreira: no
15:43:40 mnaser rather heavy heat usage of create volume + attach to instance
15:43:44 mriedem mnaser: so this is boot from volume - is the volume provided to nova, or is nova creating the volume?
15:43:50 mriedem i.e. does the failure happen in nova-api or nova-compute?
15:44:10 mnaser mriedem: boot from volume, volume created by heat and then given to bdm_v2
15:44:53 belmoreira dansmith are there any docs that describes how to use them? can find a lot of info about them
15:45:15 mnaser http://paste.openstack.org/show/672350/ <-- pretty much this heat template
15:45:54 jaypipes I prefer a cold press for pain.
15:46:01 dansmith belmoreira: it's not going to solve your immediate problem, I'm just saying that aggregates, resources, and traits are the ways you can slice the result from placement (not cells)
15:46:01 mnaser :P
15:46:05 jaypipes :)
15:46:25 mriedem mnaser: ok likely getting here https://github.com/openstack/nova/blob/stable/pike/nova/compute/api.py#L1347
15:46:26 mnaser technically speaking heat is creating a volume and then giving it to nova (so boot from volume with provided volume id)
15:46:28 jaypipes mnaser: well, that heat template isn't exactly complicated...
15:46:43 dansmith belmoreira: tbh, I'm not sure how to address your case of wanting to pre-filter or slice hosts by tenant
15:47:00 openstackgerrit Aditya Vaja proposed openstack/nova master: remove IVS plug/unplug as they're moved to separate plugin https://review.openstack.org/534371
15:47:29 mnaser mriedem: yeah that's as far as i got, but it seems that once it gets there, the volume is already in attaching state.. so something kicked off the attach somewhere
15:47:48 belmoreira dansmith thanks, we will continue to dig into this
15:47:49 mnaser i can see the volume stuck in 'attaching' state right now in cinder still
15:48:09 mriedem mnaser: do you see anything in the cinder logs for that volume id before nova tries to reserve it?
15:48:19 dansmith belmoreira: let me think on it a bit
15:48:20 mnaser mriedem: good question, i should have thought of that
15:48:52 belmoreira dansmith but for Pike the only option will be the have a "normal" aggregate per cell and let the filter scheduler do the job. concerned about the performance though
15:49:09 mriedem mnaser: also this is likely something you want in pike https://github.com/openstack/nova/commit/6d1877bf1da098b3be17fbd2dd5e53eecd2f048c
15:49:17 mriedem should be in 16.1.0
15:49:17 dansmith belmoreira: yeah, although you can have more schedulers and parallelize that load
15:50:04 mnaser oh that's interesting, that might be related
15:50:25 belmoreira yes, that's the plan. but in the worst case scenario each scheduler will need to go through 9k nodes
15:50:51 belmoreira that's why in ocata i decided to have local placements
15:51:00 mriedem mnaser: yeah you might be hiting this https://github.com/openstack/nova/blob/stable/pike/nova/compute/api.py#L1406
15:51:12 dansmith belmoreira: aye, well, might need a hack for that case in pike until we think of something else
15:51:12 mriedem so nova isn't actually trying to call cinder to reserve the volume
15:51:16 openstackgerrit Eric Berglund proposed openstack/nova master: Add check for redundant import aliases https://review.openstack.org/543995
15:51:19 mriedem it's doing it's own conditional state check and failure
15:51:21 mriedem *failing
15:51:33 mriedem although i don't know why the volume status would be 'attaching'
15:52:17 mnaser interestingly enough i see a POST to /action
15:52:27 dansmith belmoreira: what's the reasoning for that split? was it for scheduling initially, or security or hardware ownership?
15:53:52 belmoreira dansmith manly specific hardware for a use-case
15:54:11 dansmith belmoreira: okay well that should really be a trait thing (although that's not done either :)
15:54:18 mnaser ok so i see a volume create in cinder-volume
15:54:40 mriedem dansmith: well,
15:54:41 belmoreira dansmith until now we isolate the hardware type per cell and allocate the right project to it
15:54:51 mriedem we do have traits-based scheduling,
15:55:01 mriedem so you could have per-project flavors that have a trait for the hw you want for those projects
15:55:06 dansmith mriedem: we're not passing it to placement yet right?
15:55:09 mriedem we are
15:55:11 mriedem in queens
15:55:12 dansmith in pike?
15:55:14 dansmith right okay
15:55:15 mriedem oh
15:55:22 mriedem yeah not pike
15:55:29 mriedem queens solves all problems, i swear
15:55:32 dansmith maybe they could backport
15:55:53 mriedem the flavor plumbing stuff isn't versioned in anyway, the placement apis to use it are,
15:55:54 cdent dansmith: traits in non-neste scenarios is working in queens last I checked
15:56:02 mriedem but you could upgrade placement on the side, and backport the nova plumbing
15:56:03 belmoreira dansmith i will have a look into that
15:56:13 mriedem cdent: cern is on ocata
15:56:29 mriedem queens placement + ocata nova (+patches) should be ok
15:56:33 mnaser ok so
15:56:40 mnaser it looks nova reserved the volume twice
15:56:41 mriedem *should* == test in pre-prod please
15:57:02 mnaser reserved it the first time succesfully, then shortly after it got reserved again
15:57:16 belmoreira mriedem :)
15:57:27 cdent mriedem: yeah, I know, it just sounded like dan was saying it's _still_ not done
15:57:42 cdent and for some limited definitions of "done" it is, a bit
15:58:00 dansmith cdent: well, tbh I didn't think we were even passing it in allocation_candidates :)
15:58:08 dansmith I just hadn't been paying enough attention
15:58:10 mriedem i know https://github.com/openstack/nova/commit/477a14e81e8c69b91a39395d78d845c8603e10fd#diff-b4a2926790df668d47bf16d43128c4cf
15:58:13 mnaser so 12 seconds later, cinder got a second reserve request for the same volume and thats why it failed (i guess a volume goes to attaching state when its being reserved)
15:58:51 mriedem mnaser: yeah it does,
15:59:00 mriedem so i'm guessing the 2nd reserve request was made by nova-compute...
15:59:14 mriedem looking
15:59:21 mnaser im gonna investigate and see if that theory adds up
15:59:29 mriedem this is where compute starts doing the attach work https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L1595
16:00:01 belmoreira thanks all
16:00:27 mriedem mnaser: so what's probably happening is it's hitting an ocata compute
16:00:33 mriedem which calls check_attach again, and fails
16:00:46 mriedem https://github.com/openstack/nova/blob/stable/ocata/nova/virt/block_device.py#L250
16:01:05 mriedem mnaser: that's why the API has that min compute service version check, but that was busted with the multi-cell cells v2 stuff, unless you have melwitt's patch
16:01:10 mriedem mnaser: so i think that's your issue
16:01:16 mriedem can you verify the compute it failed on is running ocata?
16:01:28 mnaser mriedem: that's what i'm double checing
16:01:32 mnaser checking*
16:05:18 mnaser mriedem: i wonder if that was the wsgi registration bug we ran into a while back

Earlier   Later