| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-24 | |||
| 15:19:38 | mriedem | using yet another option | |
| 15:20:17 | mriedem | savvas: anyway, does that answer your question? | |
| 15:20:37 | savvas | trying it as we speak :) | |
| 15:20:52 | mriedem | maybe we need to update the cross_az_attach config option help docs a bit to make this clear | |
| 15:21:09 | mriedem | i.e. by default nova will not specify an AZ when creating a volume so the volume will be created in the default AZ defined in Cinder | |
| 15:21:20 | mriedem | which may or may not be the same as the AZ that the instance is created in | |
| 15:21:57 | savvas | it still fails on the same error for me though | |
| 15:22:27 | savvas | I set cross_az_attach to False and restarted my scheduler service | |
| 15:22:50 | savvas | but it still tries to create a volume in the nova az, regardless of whether I defined the AZ for the instance to be created in or not | |
| 15:23:58 | smcginnis | savvas: As mriedem explained, the AZ doesn't get passed through to cinder. So that makes sense. | |
| 15:24:52 | mriedem | savvas: that flag is checked in the compute service | |
| 15:24:54 | savvas | smcginnis: yes that's what i noticed as well, but his suggested fix for cross_az_attach would/should help from my understanding | |
| 15:24:56 | mriedem | not scheduler | |
| 15:25:05 | mriedem | nova-compute is what creates the volume | |
| 15:25:08 | mriedem | surprise! | |
| 15:25:12 | savvas | oh I see so it is done on the compute nodes | |
| 15:25:28 | mriedem | yeah - ideally conductor would do this, but... | |
| 15:25:43 | mriedem | or even the api so we could fail fast | |
| 15:25:43 | smcginnis | savvas: That will just allow it to be attached across AZs. But there is no way to have it created in the right AZ in the first place unless the right AZ is the default one. | |
| 15:25:46 | mriedem | i've wanted that for a long time | |
| 15:26:11 | savvas | smcginnis: right that's what i figured, is there any way around that? | |
| 15:26:24 | savvas | mriedem: ye that would make much more sense | |
| 15:26:31 | mriedem | we've also talked about changing the cinder reserve (now attachment_create) api to take an AZ and fail if it doesn't exist, something like that | |
| 15:26:37 | mriedem | so we could fail early in the api | |
| 15:26:40 | mriedem | when reserving the volume | |
| 15:27:08 | mriedem | smcginnis: setting cross_az_attach=False will make nova pass the instance az to cinder when creating the volume | |
| 15:27:14 | mriedem | if that az doesn't exist in cinder, it will fail | |
| 15:27:25 | mriedem | and you'll get an instance in ERROR state and NoValidHost, because we don't do retries during bfv | |
| 15:27:29 | mriedem | bfv == boot from volume | |
| 15:27:38 | smcginnis | mriedem: Ah, missed that part. | |
| 15:27:50 | mriedem | https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L80 | |
| 15:28:27 | mriedem | now don't forget that cinder has allow_availability_zone_fallback | |
| 15:28:37 | mriedem | which is yet another terrible wrinkle in this mess, and should probably be deleted | |
| 15:28:46 | mriedem | as it was a hack workaround for some bug in nova, which i think we fixed | |
| 15:28:49 | smcginnis | That was actually the one I was thinking of. | |
| 15:29:04 | savvas | mriedem: your cross_az suggestion does seem to fix it when explicitly selecting a zone to create the instance in | |
| 15:29:09 | mriedem | allow_availability_zone_fallback=True means (i think) if nova says create the instance in this az but that az doesn't exist, don't fail | |
| 15:29:16 | mriedem | savvas: cool | |
| 15:29:20 | smcginnis | That was actually an end user request where they didn't care about nova AZs matching cinder AZs. | |
| 15:29:35 | mriedem | so if you set allow_availability_zone_fallback=True in cinder and cross_az_attach=False in nova, you're just screwing your users | |
| 15:29:55 | mriedem | smcginnis: then they should have set cross_az_attach=False (or left the default) | |
| 15:30:03 | mriedem | sorry, True | |
| 15:30:13 | mriedem | cross_az_attach=True by default, which means we don't care | |
| 15:30:34 | savvas | mriedem: now from a production perspective, it would make sense to have nova define the best pool to provision though. Any suggestions for that? To create an instance with the "any availability zone" selected or via cli left blank? | |
| 15:30:38 | mriedem | i'm fairly certain that when i wrote https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L80 it was to fix a bug which was hacked around by allow_availability_zone_fallback in cinder | |
| 15:31:07 | mriedem | savvas: if you don't need to be in a specific az, then you don't specify one | |
| 15:31:23 | mriedem | smcginnis: we (and by we i mean you or one of your henchmen) should investigate removing allow_availability_zone_fallback | |
| 15:31:30 | mriedem | via deprecation first of course | |
| 15:31:33 | savvas | mriedem: right, but then creation fails as it will try to put the volume in nova again | |
| 15:31:45 | mriedem | savvas: i don't understand that statement | |
| 15:31:56 | mriedem | you mean if you set cross_az_attach=False? | |
| 15:31:58 | savvas | your cross_az_attach fixed that when I define an AZ, it will grab the right volume zone | |
| 15:32:09 | savvas | but if I don't define/select an AZ to create the instance in | |
| 15:32:11 | savvas | it still fails | |
| 15:32:13 | mriedem | savvas: the default instance az in nova matches the default volume az in cinder | |
| 15:32:20 | mriedem | which is 'nova' | |
| 15:32:33 | smcginnis | mriedem: Well, there's the non-Nova case to account for too. So I think it's still needed. You just can't misconfigure your nova and cinder settings like that. | |
| 15:32:37 | cdent | melwitt, edleafe: in case it wasn't clear from my posting, I totally prefer a new porcelain api over transitioning the existing cruft. Was merely trying to pose an alternative/observation/etc. But as sdague eloquently pointed out in #openstack-dev last night, unless we put some muscle behind it, it is for naught. Got spare muscle? | |
| 15:32:43 | mriedem | smcginnis: psh non-nova case | |
| 15:33:10 | mriedem | smcginnis: allow_availability_zone_fallback is also not interoperable | |
| 15:33:30 | smcginnis | mriedem: Not sure I follow what you mean. | |
| 15:33:36 | savvas | mriedem: right, however leaving that blank is what would leave nova to determine the best node to spawn the instance on right? This would be a desirable feature when automating provisioning tasks | |
| 15:33:40 | mriedem | smcginnis: on one cloud i can create a volume with az=foo even though it doesn't exist, on another cloud i can't create a volume in az=foo because it doesn't exist and it's enforced | |
| 15:33:43 | edleafe | cdent: I know that you were just being hypothetical | |
| 15:33:57 | mriedem | smcginnis: it's per-cloud configurable API behavior | |
| 15:33:59 | edleafe | cdent: And no, no spare muscles at the moment :) | |
| 15:34:02 | mriedem | which is by definition not interoperable | |
| 15:34:10 | cdent | not so much hypothetical as omnithetical | |
| 15:34:24 | sdague | cdent: I also think it's far more work than people imagine. Even the code itself is a lot of work, but in lessons learn from nova v3 and the current api-ref effort, the code is only a small amount of the total work | |
| 15:34:49 | cdent | sdague: no dispute | |
| 15:35:04 | smcginnis | mriedem: If that's the concern, we have much bigger fish to fry than AZ enforcement. | |
| 15:35:08 | mriedem | smcginnis: maybe i should propose a deprecation patch in cinder and people can discuss i there | |
| 15:35:17 | mriedem | smcginnis: it's also complexity | |
| 15:35:23 | mriedem | which the foundation is apparently caring about now | |
| 15:35:30 | mriedem | hell, even thingee might get behind this | |
| 15:35:34 | smcginnis | mriedem: And deprecate the nova option too? | |
| 15:35:34 | mriedem | being foundation + cinder guy | |
| 15:35:38 | mriedem | smcginnis: which one? | |
| 15:35:47 | mriedem | cross_az_attach? | |
| 15:35:56 | smcginnis | mriedem: Yep. | |
| 15:36:12 | smcginnis | mriedem: Same argument, if that is set on one cloud and not another, you get different behavior. | |
| 15:36:12 | mriedem | i'd like to | |
| 15:36:18 | mriedem | true | |
| 15:36:29 | mriedem | you know what, | |
| 15:36:30 | jmlowe | I think I've botched adding cellsv2, I'm on newton | |
| 15:36:35 | sdague | cdent: and, the reference point that everyone keeps referencing is git, which isn't doing those calls over a network, with a token validation in the middle. I'm not actually sure how applicable a porcelain approach is to REST services. It's at least a research project to even figure it out | |
| 15:36:40 | mriedem | we should just allow passing all of the things you need to create a volume in to nova, | |
| 15:36:42 | mriedem | volume type, | |
| 15:36:43 | mriedem | volume az, | |
| 15:36:46 | mriedem | volume qos | |
| 15:36:47 | mriedem | everything | |
| 15:36:53 | jmlowe | nova list shows an instance, nova delete sayas it's not there | |
| 15:36:53 | cdent | jmlowe: did you show it some bourbon? | |
| 15:37:06 | mriedem | jmlowe: sec | |
| 15:37:17 | smcginnis | mriedem: There are some who would be in favor of that. | |
| 15:37:23 | edleafe | sdague: writing an SDK is sort of like writing a porcelain wrapper (but much simpler), and that was a LOT of work | |
| 15:37:28 | mriedem | jmlowe: see #3 https://docs.openstack.org/developer/nova/cells.html#faqs | |
| 15:37:40 | jmlowe | cdent: I'm going to need some bourbon to steady my nerves here in a minute if I don't figure this out | |
| 15:37:48 | mriedem | jmlowe: if that fixes your problem, i can die happy | |
| 15:37:56 | edleafe | creating a full orchestrator would be an order of magnitude more | |