Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-24
14:47:48 sdague yeh
14:48:01 sdague it would be nice if the output wasn't an eye chart with a soda straw
14:48:51 mriedem yeah it's hard to navigate
14:49:05 sdague and non searchable
14:49:22 sdague the svg thing looks all sexy, until you try to do things with it
15:09:15 savvas Hi guys, I've been hopping from channel to channel trying to isolate the cause of a problem I've been having
15:09:24 savvas when I try to boot an instance as following: nova boot newInstance --flavor testpack --security-groups default --nic net-id=7a2a1e61-1d86-4551-b578-dedf7f8443ea --block-device source=image,id=de61646b-8c48-43ac-a8b1-8f8348757d66,dest=volume,size=10,shutdown=remove,bootindex=0 --availability-zone hv01-zone
15:10:25 savvas nova seems to ignore the availability-zone for cinder, it tries to use the default "nova" zone for creating the volume, instead of the AZ defined in the boot command. Am I missing something in my configuration perhaps that tells nova to look in diff volume AZ?
15:12:17 savvas I should add that if I create a volume manually in the right AZ and then boot an instance from that volume, it works just fine. However, combining those actions into a nova boot / server create command fails
15:13:38 mriedem cfriesen: you might be interested in this thread http://lists.openstack.org/pipermail/openstack-dev/2017-May/116595.html
15:14:10 mriedem savvas: which release?
15:14:17 savvas ocata
15:14:20 mriedem and what's the value of [cinder]cross_az_attach in nvoa.conf?
15:14:21 mriedem *nova.conf
15:14:31 mriedem or is it not set?
15:14:41 savvas that's left to default, I found comments on that but I didn't think it was relevant in my case
15:14:44 savvas not set
15:15:41 savvas I figured based on the AZ I define when creating an instance that it would look for the available AZ as defined by the cinder-volume service
15:15:59 mriedem nova doesn't list AZs from cinder
15:16:44 mriedem have you looked at https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L55 ?
15:17:12 mriedem cross_az_attach defaults to True, so that method is going to return None
15:17:38 mriedem which means when nova creates the volume, it's not going to pass an az to cinder,
15:17:43 mriedem so cinder uses it's default az, which is 'nova'
15:17:51 mriedem because that makes total sense right?!
15:18:05 savvas right lol
15:18:16 mriedem if you want nova to specify the same az for the volume as the vm, you have to set cross_az_attach=False
15:18:19 sdague yay for inherited defaults
15:19:19 mriedem there are apparently at least 3 cinder config options for the default az
15:19:35 mriedem one global one, which if not set defaults to another global one, which defaults to nova, but can be overridden on a per-backend basis
15:19:38 mriedem using yet another option
15:20:17 mriedem savvas: anyway, does that answer your question?
15:20:37 savvas trying it as we speak :)
15:20:52 mriedem maybe we need to update the cross_az_attach config option help docs a bit to make this clear
15:21:09 mriedem i.e. by default nova will not specify an AZ when creating a volume so the volume will be created in the default AZ defined in Cinder
15:21:20 mriedem which may or may not be the same as the AZ that the instance is created in
15:21:57 savvas it still fails on the same error for me though
15:22:27 savvas I set cross_az_attach to False and restarted my scheduler service
15:22:50 savvas but it still tries to create a volume in the nova az, regardless of whether I defined the AZ for the instance to be created in or not
15:23:58 smcginnis savvas: As mriedem explained, the AZ doesn't get passed through to cinder. So that makes sense.
15:24:52 mriedem savvas: that flag is checked in the compute service
15:24:54 savvas smcginnis: yes that's what i noticed as well, but his suggested fix for cross_az_attach would/should help from my understanding
15:24:56 mriedem not scheduler
15:25:05 mriedem nova-compute is what creates the volume
15:25:08 mriedem surprise!
15:25:12 savvas oh I see so it is done on the compute nodes
15:25:28 mriedem yeah - ideally conductor would do this, but...
15:25:43 smcginnis savvas: That will just allow it to be attached across AZs. But there is no way to have it created in the right AZ in the first place unless the right AZ is the default one.
15:25:43 mriedem or even the api so we could fail fast
15:25:46 mriedem i've wanted that for a long time
15:26:11 savvas smcginnis: right that's what i figured, is there any way around that?
15:26:24 savvas mriedem: ye that would make much more sense
15:26:31 mriedem we've also talked about changing the cinder reserve (now attachment_create) api to take an AZ and fail if it doesn't exist, something like that
15:26:37 mriedem so we could fail early in the api
15:26:40 mriedem when reserving the volume
15:27:08 mriedem smcginnis: setting cross_az_attach=False will make nova pass the instance az to cinder when creating the volume
15:27:14 mriedem if that az doesn't exist in cinder, it will fail
15:27:25 mriedem and you'll get an instance in ERROR state and NoValidHost, because we don't do retries during bfv
15:27:29 mriedem bfv == boot from volume
15:27:38 smcginnis mriedem: Ah, missed that part.
15:27:50 mriedem https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L80
15:28:27 mriedem now don't forget that cinder has allow_availability_zone_fallback
15:28:37 mriedem which is yet another terrible wrinkle in this mess, and should probably be deleted
15:28:46 mriedem as it was a hack workaround for some bug in nova, which i think we fixed
15:28:49 smcginnis That was actually the one I was thinking of.
15:29:04 savvas mriedem: your cross_az suggestion does seem to fix it when explicitly selecting a zone to create the instance in
15:29:09 mriedem allow_availability_zone_fallback=True means (i think) if nova says create the instance in this az but that az doesn't exist, don't fail
15:29:16 mriedem savvas: cool
15:29:20 smcginnis That was actually an end user request where they didn't care about nova AZs matching cinder AZs.
15:29:35 mriedem so if you set allow_availability_zone_fallback=True in cinder and cross_az_attach=False in nova, you're just screwing your users
15:29:55 mriedem smcginnis: then they should have set cross_az_attach=False (or left the default)
15:30:03 mriedem sorry, True
15:30:13 mriedem cross_az_attach=True by default, which means we don't care
15:30:34 savvas mriedem: now from a production perspective, it would make sense to have nova define the best pool to provision though. Any suggestions for that? To create an instance with the "any availability zone" selected or via cli left blank?
15:30:38 mriedem i'm fairly certain that when i wrote https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L80 it was to fix a bug which was hacked around by allow_availability_zone_fallback in cinder
15:31:07 mriedem savvas: if you don't need to be in a specific az, then you don't specify one
15:31:23 mriedem smcginnis: we (and by we i mean you or one of your henchmen) should investigate removing allow_availability_zone_fallback
15:31:30 mriedem via deprecation first of course
15:31:33 savvas mriedem: right, but then creation fails as it will try to put the volume in nova again
15:31:45 mriedem savvas: i don't understand that statement
15:31:56 mriedem you mean if you set cross_az_attach=False?
15:31:58 savvas your cross_az_attach fixed that when I define an AZ, it will grab the right volume zone
15:32:09 savvas but if I don't define/select an AZ to create the instance in
15:32:11 savvas it still fails
15:32:13 mriedem savvas: the default instance az in nova matches the default volume az in cinder
15:32:20 mriedem which is 'nova'
15:32:33 smcginnis mriedem: Well, there's the non-Nova case to account for too. So I think it's still needed. You just can't misconfigure your nova and cinder settings like that.
15:32:37 cdent melwitt, edleafe: in case it wasn't clear from my posting, I totally prefer a new porcelain api over transitioning the existing cruft. Was merely trying to pose an alternative/observation/etc. But as sdague eloquently pointed out in #openstack-dev last night, unless we put some muscle behind it, it is for naught. Got spare muscle?
15:32:43 mriedem smcginnis: psh non-nova case
15:33:10 mriedem smcginnis: allow_availability_zone_fallback is also not interoperable
15:33:30 smcginnis mriedem: Not sure I follow what you mean.
15:33:36 savvas mriedem: right, however leaving that blank is what would leave nova to determine the best node to spawn the instance on right? This would be a desirable feature when automating provisioning tasks
15:33:40 mriedem smcginnis: on one cloud i can create a volume with az=foo even though it doesn't exist, on another cloud i can't create a volume in az=foo because it doesn't exist and it's enforced
15:33:43 edleafe cdent: I know that you were just being hypothetical
15:33:57 mriedem smcginnis: it's per-cloud configurable API behavior
15:33:59 edleafe cdent: And no, no spare muscles at the moment :)
15:34:02 mriedem which is by definition not interoperable
15:34:10 cdent not so much hypothetical as omnithetical
15:34:24 sdague cdent: I also think it's far more work than people imagine. Even the code itself is a lot of work, but in lessons learn from nova v3 and the current api-ref effort, the code is only a small amount of the total work
15:34:49 cdent sdague: no dispute
15:35:04 smcginnis mriedem: If that's the concern, we have much bigger fish to fry than AZ enforcement.

Earlier   Later