| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-14 | |||
| 15:37:08 | tssurya | we under no circumstance delete a cell if there are instances | |
| 15:37:10 | mriedem | i'm saying delete_cell --force will fail today if there are instance mappings in the cell | |
| 15:37:50 | mriedem | ok this patch https://review.openstack.org/#/c/540073/ | |
| 15:38:08 | tssurya | mriedem : yes , my after merging my patch, delete_cell will fail if there any living instances | |
| 15:39:25 | mriedem | ok i see | |
| 15:39:47 | jaypipes | stephenfin: https://review.openstack.org/#/c/538217/3/specs/rocky/approved/integrate-mypy-type-checking.rst <-- well, sheeet, let's just rewrite Nova in Golang (or Rust). :P | |
| 15:40:49 | stephenfin | jaypipes: If you can find me a tool to autoconvert one language to another, I'm all for it ;) | |
| 15:40:59 | mriedem | vish suggested java after joining oracle | |
| 15:41:04 | mriedem | i know java, not go or rust | |
| 15:41:27 | jaypipes | stephenfin: :) | |
| 15:41:34 | belmoreira | a concern that I also have with placement is that it doesn't know anything about cells, that is now an official think in nova | |
| 15:41:39 | mnaser | i've been looking through the nova codebase throughout this morning -- we're running into an issue where nova fails to boot an instance "Invalid volume: Volume status must be available to reserve, but the status is attaching. (HTTP 400)" | |
| 15:41:46 | jaypipes | I'd be up for a rewrite in C++. | |
| 15:41:49 | mriedem | belmoreira: placement shouldn't know about cells | |
| 15:41:52 | belmoreira | we map projects to cells. Meaning that to handle a "small" VM the placement will return "all" nodes and then is the scheduler filtering to find the right cell | |
| 15:41:57 | stephenfin | jaypipes: I do suggest checking out the sample patches I've thrown up through. An afternoons work and it really is minimally invasive | |
| 15:42:04 | belmoreira | is there a way to get a more filtered answer from placement? | |
| 15:42:04 | jaypipes | stephenfin: will do :) | |
| 15:42:09 | mnaser | it seems like *somehow*, the cinder volume is in attaching state, and nova tries to reserve it, and so it fails after | |
| 15:42:09 | stephenfin | jaypipes: and whatever you do, _don't_ listen to mdbooth! :D | |
| 15:42:15 | jaypipes | heh | |
| 15:42:22 | mriedem | mnaser: this is pike 16.0.4? | |
| 15:42:25 | dansmith | belmoreira: aggregates | |
| 15:42:33 | dansmith | belmoreira: that's how you get a more filtered answer | |
| 15:42:35 | mdbooth | stephenfin: Generally good advice. | |
| 15:42:47 | belmoreira | dansmith placement aggregates? | |
| 15:42:50 | stephenfin | mdbooth: https://review.openstack.org/#/c/538217/3/specs/rocky/approved/integrate-mypy-type-checking.rst | |
| 15:42:51 | dansmith | belmoreira: aye | |
| 15:43:00 | mdbooth | stephenfin: Haha | |
| 15:43:04 | mnaser | mriedem: i believe the control plane is stable/pike, i think computes might be a bit older i'm not sure if they're 16.0.4 | |
| 15:43:06 | jaypipes | dansmith: aggregate filtering isn't supported yet in allocation candidates... | |
| 15:43:14 | jaypipes | dansmith: or at least, I don't think it is.. | |
| 15:43:23 | dansmith | jaypipes: I'm saying that's the plan | |
| 15:43:26 | mnaser | we're running into a lot of weird conditions with heat | |
| 15:43:27 | jaypipes | ah | |
| 15:43:29 | belmoreira | dansmith can they the used in Pike? | |
| 15:43:30 | dansmith | jaypipes: we're not asking for it now anyway | |
| 15:43:33 | mdbooth | stephenfin: Spit shine that turd ;) | |
| 15:43:33 | dansmith | belmoreira: no | |
| 15:43:40 | mnaser | rather heavy heat usage of create volume + attach to instance | |
| 15:43:44 | mriedem | mnaser: so this is boot from volume - is the volume provided to nova, or is nova creating the volume? | |
| 15:43:50 | mriedem | i.e. does the failure happen in nova-api or nova-compute? | |
| 15:44:10 | mnaser | mriedem: boot from volume, volume created by heat and then given to bdm_v2 | |
| 15:44:53 | belmoreira | dansmith are there any docs that describes how to use them? can find a lot of info about them | |
| 15:45:15 | mnaser | http://paste.openstack.org/show/672350/ <-- pretty much this heat template | |
| 15:45:54 | jaypipes | I prefer a cold press for pain. | |
| 15:46:01 | dansmith | belmoreira: it's not going to solve your immediate problem, I'm just saying that aggregates, resources, and traits are the ways you can slice the result from placement (not cells) | |
| 15:46:01 | mnaser | :P | |
| 15:46:05 | jaypipes | :) | |
| 15:46:25 | mriedem | mnaser: ok likely getting here https://github.com/openstack/nova/blob/stable/pike/nova/compute/api.py#L1347 | |
| 15:46:26 | mnaser | technically speaking heat is creating a volume and then giving it to nova (so boot from volume with provided volume id) | |
| 15:46:28 | jaypipes | mnaser: well, that heat template isn't exactly complicated... | |
| 15:46:43 | dansmith | belmoreira: tbh, I'm not sure how to address your case of wanting to pre-filter or slice hosts by tenant | |
| 15:47:00 | openstackgerrit | Aditya Vaja proposed openstack/nova master: remove IVS plug/unplug as they're moved to separate plugin https://review.openstack.org/534371 | |
| 15:47:29 | mnaser | mriedem: yeah that's as far as i got, but it seems that once it gets there, the volume is already in attaching state.. so something kicked off the attach somewhere | |
| 15:47:48 | belmoreira | dansmith thanks, we will continue to dig into this | |
| 15:47:49 | mnaser | i can see the volume stuck in 'attaching' state right now in cinder still | |
| 15:48:09 | mriedem | mnaser: do you see anything in the cinder logs for that volume id before nova tries to reserve it? | |
| 15:48:19 | dansmith | belmoreira: let me think on it a bit | |
| 15:48:20 | mnaser | mriedem: good question, i should have thought of that | |
| 15:48:52 | belmoreira | dansmith but for Pike the only option will be the have a "normal" aggregate per cell and let the filter scheduler do the job. concerned about the performance though | |
| 15:49:09 | mriedem | mnaser: also this is likely something you want in pike https://github.com/openstack/nova/commit/6d1877bf1da098b3be17fbd2dd5e53eecd2f048c | |
| 15:49:17 | mriedem | should be in 16.1.0 | |
| 15:49:17 | dansmith | belmoreira: yeah, although you can have more schedulers and parallelize that load | |
| 15:50:04 | mnaser | oh that's interesting, that might be related | |
| 15:50:25 | belmoreira | yes, that's the plan. but in the worst case scenario each scheduler will need to go through 9k nodes | |
| 15:50:51 | belmoreira | that's why in ocata i decided to have local placements | |
| 15:51:00 | mriedem | mnaser: yeah you might be hiting this https://github.com/openstack/nova/blob/stable/pike/nova/compute/api.py#L1406 | |
| 15:51:12 | dansmith | belmoreira: aye, well, might need a hack for that case in pike until we think of something else | |
| 15:51:12 | mriedem | so nova isn't actually trying to call cinder to reserve the volume | |
| 15:51:16 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add check for redundant import aliases https://review.openstack.org/543995 | |
| 15:51:19 | mriedem | it's doing it's own conditional state check and failure | |
| 15:51:21 | mriedem | *failing | |
| 15:51:33 | mriedem | although i don't know why the volume status would be 'attaching' | |
| 15:52:17 | mnaser | interestingly enough i see a POST to /action | |
| 15:52:27 | dansmith | belmoreira: what's the reasoning for that split? was it for scheduling initially, or security or hardware ownership? | |
| 15:53:52 | belmoreira | dansmith manly specific hardware for a use-case | |
| 15:54:11 | dansmith | belmoreira: okay well that should really be a trait thing (although that's not done either :) | |
| 15:54:18 | mnaser | ok so i see a volume create in cinder-volume | |
| 15:54:40 | mriedem | dansmith: well, | |
| 15:54:41 | belmoreira | dansmith until now we isolate the hardware type per cell and allocate the right project to it | |
| 15:54:51 | mriedem | we do have traits-based scheduling, | |
| 15:55:01 | mriedem | so you could have per-project flavors that have a trait for the hw you want for those projects | |
| 15:55:06 | dansmith | mriedem: we're not passing it to placement yet right? | |
| 15:55:09 | mriedem | we are | |
| 15:55:11 | mriedem | in queens | |
| 15:55:12 | dansmith | in pike? | |
| 15:55:14 | dansmith | right okay | |
| 15:55:15 | mriedem | oh | |
| 15:55:22 | mriedem | yeah not pike | |
| 15:55:29 | mriedem | queens solves all problems, i swear | |
| 15:55:32 | dansmith | maybe they could backport | |
| 15:55:53 | mriedem | the flavor plumbing stuff isn't versioned in anyway, the placement apis to use it are, | |
| 15:55:54 | cdent | dansmith: traits in non-neste scenarios is working in queens last I checked | |
| 15:56:02 | mriedem | but you could upgrade placement on the side, and backport the nova plumbing | |
| 15:56:03 | belmoreira | dansmith i will have a look into that | |
| 15:56:13 | mriedem | cdent: cern is on ocata | |
| 15:56:29 | mriedem | queens placement + ocata nova (+patches) should be ok | |
| 15:56:33 | mnaser | ok so | |
| 15:56:40 | mnaser | it looks nova reserved the volume twice | |