| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-24 | |||
| 14:10:18 | sdague | larsks: that's at least the place to play around with things | |
| 14:12:59 | mriedem | btw, | |
| 14:13:13 | mriedem | i heard that one of the outcomes of a forum session on apis was that we don't have config drive documented anywhere | |
| 14:13:34 | mriedem | there is this https://docs.openstack.org/user-guide/cli-config-drive.html | |
| 14:13:56 | openstackgerrit | Daniel Pawlik proposed openstack/nova master: Add `hide_hypervisor_id` image property https://review.openstack.org/459753 | |
| 14:13:59 | mriedem | i'm assuming it's more a question of what the contents are | |
| 14:14:09 | mriedem | maybe per version, like what we have in the rest api version history | |
| 14:14:51 | sdague | yep, it's definitely in the state of something that you need to read code to understand | |
| 14:16:33 | dansmith | well, that goes for metadata too | |
| 14:16:34 | mriedem | there was a spec that mgagne recently updated for the network data json blob | |
| 14:16:46 | mriedem | dansmith: right, that's what i was thinking with versions | |
| 14:17:00 | mriedem | e.g. in newton you get a new version with device tags | |
| 14:17:45 | dansmith | yeah | |
| 14:18:38 | mriedem | this one https://review.openstack.org/#/c/337819/ | |
| 14:19:18 | mriedem | let me see if i can find the forum notes and start something in the ML to ask what people are looking for | |
| 14:19:45 | mriedem | i've also been thinking about starting an in-tree doc with a dump of all known flavor extra specs | |
| 14:22:11 | mgagne | I feel like this review is losing momentum, is there a way to highlight it somewhere to get more attention? https://review.openstack.org/#/c/454322/ | |
| 14:22:23 | mgagne | (Add ability to signal and perform online volume size change) | |
| 14:22:55 | mriedem | mgagne: it's on my list | |
| 14:23:04 | mriedem | mgagne: i've been going through some of the random smaller bp changes this week | |
| 14:23:10 | mgagne | thanks =) | |
| 14:41:11 | mriedem | http://lists.openstack.org/pipermail/openstack-dev/2017-May/117401.html | |
| 14:41:17 | mriedem | mgagne: ^ you might have input also | |
| 14:46:26 | mriedem | sdague: from http://lists.openstack.org/pipermail/openstack-dev/2017-May/117374.html - http://enos.irisa.fr/html/seqdiag/v1/trace-boot-and-delete.yaml.html | |
| 14:46:43 | mriedem | i remember someone from ibm research a few years ago wrote some tooling to do tracing like this too, using osprofiler i think | |
| 14:46:53 | mriedem | reminded me of the nova onboarding session | |
| 14:47:48 | sdague | yeh | |
| 14:48:01 | sdague | it would be nice if the output wasn't an eye chart with a soda straw | |
| 14:48:51 | mriedem | yeah it's hard to navigate | |
| 14:49:05 | sdague | and non searchable | |
| 14:49:22 | sdague | the svg thing looks all sexy, until you try to do things with it | |
| 15:09:15 | savvas | Hi guys, I've been hopping from channel to channel trying to isolate the cause of a problem I've been having | |
| 15:09:24 | savvas | when I try to boot an instance as following: nova boot newInstance --flavor testpack --security-groups default --nic net-id=7a2a1e61-1d86-4551-b578-dedf7f8443ea --block-device source=image,id=de61646b-8c48-43ac-a8b1-8f8348757d66,dest=volume,size=10,shutdown=remove,bootindex=0 --availability-zone hv01-zone | |
| 15:10:25 | savvas | nova seems to ignore the availability-zone for cinder, it tries to use the default "nova" zone for creating the volume, instead of the AZ defined in the boot command. Am I missing something in my configuration perhaps that tells nova to look in diff volume AZ? | |
| 15:12:17 | savvas | I should add that if I create a volume manually in the right AZ and then boot an instance from that volume, it works just fine. However, combining those actions into a nova boot / server create command fails | |
| 15:13:38 | mriedem | cfriesen: you might be interested in this thread http://lists.openstack.org/pipermail/openstack-dev/2017-May/116595.html | |
| 15:14:10 | mriedem | savvas: which release? | |
| 15:14:17 | savvas | ocata | |
| 15:14:20 | mriedem | and what's the value of [cinder]cross_az_attach in nvoa.conf? | |
| 15:14:21 | mriedem | *nova.conf | |
| 15:14:31 | mriedem | or is it not set? | |
| 15:14:41 | savvas | that's left to default, I found comments on that but I didn't think it was relevant in my case | |
| 15:14:44 | savvas | not set | |
| 15:15:41 | savvas | I figured based on the AZ I define when creating an instance that it would look for the available AZ as defined by the cinder-volume service | |
| 15:15:59 | mriedem | nova doesn't list AZs from cinder | |
| 15:16:44 | mriedem | have you looked at https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L55 ? | |
| 15:17:12 | mriedem | cross_az_attach defaults to True, so that method is going to return None | |
| 15:17:38 | mriedem | which means when nova creates the volume, it's not going to pass an az to cinder, | |
| 15:17:43 | mriedem | so cinder uses it's default az, which is 'nova' | |
| 15:17:51 | mriedem | because that makes total sense right?! | |
| 15:18:05 | savvas | right lol | |
| 15:18:16 | mriedem | if you want nova to specify the same az for the volume as the vm, you have to set cross_az_attach=False | |
| 15:18:19 | sdague | yay for inherited defaults | |
| 15:19:19 | mriedem | there are apparently at least 3 cinder config options for the default az | |
| 15:19:35 | mriedem | one global one, which if not set defaults to another global one, which defaults to nova, but can be overridden on a per-backend basis | |
| 15:19:38 | mriedem | using yet another option | |
| 15:20:17 | mriedem | savvas: anyway, does that answer your question? | |
| 15:20:37 | savvas | trying it as we speak :) | |
| 15:20:52 | mriedem | maybe we need to update the cross_az_attach config option help docs a bit to make this clear | |
| 15:21:09 | mriedem | i.e. by default nova will not specify an AZ when creating a volume so the volume will be created in the default AZ defined in Cinder | |
| 15:21:20 | mriedem | which may or may not be the same as the AZ that the instance is created in | |
| 15:21:57 | savvas | it still fails on the same error for me though | |
| 15:22:27 | savvas | I set cross_az_attach to False and restarted my scheduler service | |
| 15:22:50 | savvas | but it still tries to create a volume in the nova az, regardless of whether I defined the AZ for the instance to be created in or not | |
| 15:23:58 | smcginnis | savvas: As mriedem explained, the AZ doesn't get passed through to cinder. So that makes sense. | |
| 15:24:52 | mriedem | savvas: that flag is checked in the compute service | |
| 15:24:54 | savvas | smcginnis: yes that's what i noticed as well, but his suggested fix for cross_az_attach would/should help from my understanding | |
| 15:24:56 | mriedem | not scheduler | |
| 15:25:05 | mriedem | nova-compute is what creates the volume | |
| 15:25:08 | mriedem | surprise! | |
| 15:25:12 | savvas | oh I see so it is done on the compute nodes | |
| 15:25:28 | mriedem | yeah - ideally conductor would do this, but... | |
| 15:25:43 | smcginnis | savvas: That will just allow it to be attached across AZs. But there is no way to have it created in the right AZ in the first place unless the right AZ is the default one. | |
| 15:25:43 | mriedem | or even the api so we could fail fast | |
| 15:25:46 | mriedem | i've wanted that for a long time | |
| 15:26:11 | savvas | smcginnis: right that's what i figured, is there any way around that? | |
| 15:26:24 | savvas | mriedem: ye that would make much more sense | |
| 15:26:31 | mriedem | we've also talked about changing the cinder reserve (now attachment_create) api to take an AZ and fail if it doesn't exist, something like that | |
| 15:26:37 | mriedem | so we could fail early in the api | |
| 15:26:40 | mriedem | when reserving the volume | |
| 15:27:08 | mriedem | smcginnis: setting cross_az_attach=False will make nova pass the instance az to cinder when creating the volume | |
| 15:27:14 | mriedem | if that az doesn't exist in cinder, it will fail | |
| 15:27:25 | mriedem | and you'll get an instance in ERROR state and NoValidHost, because we don't do retries during bfv | |
| 15:27:29 | mriedem | bfv == boot from volume | |
| 15:27:38 | smcginnis | mriedem: Ah, missed that part. | |
| 15:27:50 | mriedem | https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L80 | |
| 15:28:27 | mriedem | now don't forget that cinder has allow_availability_zone_fallback | |
| 15:28:37 | mriedem | which is yet another terrible wrinkle in this mess, and should probably be deleted | |
| 15:28:46 | mriedem | as it was a hack workaround for some bug in nova, which i think we fixed | |
| 15:28:49 | smcginnis | That was actually the one I was thinking of. | |
| 15:29:04 | savvas | mriedem: your cross_az suggestion does seem to fix it when explicitly selecting a zone to create the instance in | |
| 15:29:09 | mriedem | allow_availability_zone_fallback=True means (i think) if nova says create the instance in this az but that az doesn't exist, don't fail | |
| 15:29:16 | mriedem | savvas: cool | |
| 15:29:20 | smcginnis | That was actually an end user request where they didn't care about nova AZs matching cinder AZs. | |
| 15:29:35 | mriedem | so if you set allow_availability_zone_fallback=True in cinder and cross_az_attach=False in nova, you're just screwing your users | |
| 15:29:55 | mriedem | smcginnis: then they should have set cross_az_attach=False (or left the default) | |
| 15:30:03 | mriedem | sorry, True | |
| 15:30:13 | mriedem | cross_az_attach=True by default, which means we don't care | |
| 15:30:34 | savvas | mriedem: now from a production perspective, it would make sense to have nova define the best pool to provision though. Any suggestions for that? To create an instance with the "any availability zone" selected or via cli left blank? | |
| 15:30:38 | mriedem | i'm fairly certain that when i wrote https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L80 it was to fix a bug which was hacked around by allow_availability_zone_fallback in cinder | |