| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-24 | |||
| 16:43:29 | mriedem | *Case | |
| 16:44:08 | savvas | mriedem: | |
| 16:44:09 | savvas | 2017-05-24 16:33:26.127 2331 ERROR nova.compute.manager [req-b5388b21-5f75-4946-8ace-56cd91ebdc72 be0e3c8d93f345a6889e76d1507ba130 7509210fbb1a4e30bf62d81aa3b4e369 - - -] [instance: 9484f5a7-3198-47ff-b728-178515a26277] Instance failed block device setup | |
| 16:44:22 | mriedem | savvas: there should be a traceback | |
| 16:44:27 | mriedem | please put that into a paste and link here | |
| 16:44:41 | mriedem | it's using LOG.exception which will traceback | |
| 16:44:53 | savvas | ye grabbed this with cat, looking it up now | |
| 16:45:57 | savvas | http://paste.as47869.net/p/9qbburh7z3w3toi | |
| 16:49:15 | mriedem | VolumeNotCreated: Volume da947c97-66c6-4b7e-9ae6-54eb8128bb75 did not finish being created even after we waited 3 seconds or 2 attempts. And its status is error. | |
| 16:49:23 | mriedem | that is a different issue | |
| 16:49:37 | jaypipes | gosh darn it... | |
| 16:49:52 | mriedem | penick: ^ is why bfv is awful | |
| 16:50:12 | mriedem | savvas: you're failing in here https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/compute/manager.py#L1222 | |
| 16:50:25 | mriedem | it's timing out waiting for the volume that nova created to be available so it can attach it to the instance | |
| 16:50:43 | mriedem | you probably need to setup image cache on the cinder side, and tweak the timeout values on the nova compute side | |
| 16:51:09 | penick | Is its disease of awfulness irreparable? | |
| 16:51:10 | smcginnis | Larger images may take a long time. | |
| 16:52:04 | savvas | mriedem: the volume fails to create though | |
| 16:52:27 | savvas | it fails instantly, as it tries to do so in an AZ which isn't available | |
| 16:52:31 | mriedem | savvas: does the volume go to error state? | |
| 16:52:37 | savvas | yes | |
| 16:52:45 | mriedem | savvas: but i thought nova was passing az=None when creating the volume? | |
| 16:52:51 | savvas | correct | |
| 16:53:12 | savvas | which leads cinder to attempt and create the volume in the default az of "nova" | |
| 16:53:15 | savvas | which fails | |
| 16:53:33 | mriedem | my head is going to explode | |
| 16:53:46 | openstackgerrit | Jay Pipes proposed openstack/nova master: integrate trait cache with TraitList and Trait https://review.openstack.org/462770 | |
| 16:53:54 | mriedem | ok, so is the problem that things get more strict on the cinder side when you actually have AZs in cinder? | |
| 16:54:07 | mriedem | because if you don't have az's in cinder, you can create volumes w/o an az (by default in the nova az) just fine | |
| 16:54:14 | mriedem | but once you add an az, cinder won't allow that anymore? | |
| 16:54:48 | savvas | ye I am sorry if I am making you go in circles here, like I said my lack of exp isn't helping much either I am sure | |
| 16:54:51 | mriedem | by the way, we should totally get that VolumeNotCreated error into https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/compute/manager.py#L1595 which is fed back up and goes into the instance fault | |
| 16:55:07 | mriedem | savvas: don't worry about it, you're doing great - i obviously don't know how anything works here either :) | |
| 16:55:46 | mriedem | what i'd like to do is know a simple recreate so i can write a tempest test for this | |
| 16:56:04 | kashyap | Does anyone know for sure, about whether the double quotes are *mandatory* when you specify a key-value in a Nova config attribute as following: | |
| 16:56:07 | kashyap | [libvirt] | |
| 16:56:10 | kashyap | hw_machine_type="x86_64=pc-i440fx-rhel7.2.0" | |
| 16:56:12 | savvas | I run 3 cinder volume servers, each with their own AZ. The controller has no storage/default AZ defined | |
| 16:56:20 | savvas | mriedem: as in steps to reproduce this error? | |
| 16:57:10 | mriedem | savvas: i think just even knowing if you can create a volume in cinder w/o specifying AZ if there is a non-default AZ defined, just one | |
| 16:57:29 | savvas | no, that doesn't work. At this stage I HAVE to specify an AZ | |
| 16:57:29 | jmlowe | mriedem: yeah, I'm going to trip over this later | |
| 16:58:04 | mriedem | savvas: huh, ok, figured cinder would allow it if az isn't specified | |
| 16:58:12 | mriedem | smcginnis: jgriffith: ^ is that by design? | |
| 16:58:21 | mriedem | hell it might even be how nova does it but i'm not sure | |
| 16:58:39 | smcginnis | mriedem: Didn't follow everything, but are you saying if AZ is None, Cinder fails the call? | |
| 16:59:44 | mriedem | smcginnis: is there is a non-default AZ defined in cinder | |
| 17:00:03 | mriedem | so if i create az=foo in cinder, | |
| 17:00:10 | mriedem | do i have to create all volumes explicitly in that az now? | |
| 17:00:34 | smcginnis | mriedem: I didn't think so. | |
| 17:00:39 | smcginnis | Checking... | |
| 17:01:39 | smcginnis | So if storage_availability_zone is set to something other than the default, you may need to also update default_availability_zone. Still looking though. | |
| 17:02:19 | savvas | smcginnis: I have both values set on each node running cinder-volume service | |
| 17:02:32 | mriedem | dansmith: are we supposed to be in meeting-3 for cells? | |
| 17:02:39 | mriedem | and the interop guys are going long? | |
| 17:02:43 | smcginnis | savvas: So both are set to unique values, but if None is passed in for AZ it fails? | |
| 17:03:40 | savvas | smcginnis: I have both set to the zone available to the node. I run nova compute and cinder volume on the same host for this purpose. So hv01 would have default/storage_availability_zone set to hv01-zone, hv02 to hv02-zone and so on | |
| 17:04:07 | mriedem | i think that's wrong | |
| 17:04:15 | mriedem | i think default_availability_zone is supposed to be global | |
| 17:04:21 | savvas | as these hypervisors are supposed to be dependent of their local storage, they shouldn't be looking to place volumes in any other zones | |
| 17:04:44 | melwitt | mriedem: yeah on my calendar we should have a cells meeting right now | |
| 17:05:24 | mriedem | that's what storage_availability_zone is for, | |
| 17:05:30 | mriedem | and it overrides default_availability_zone | |
| 17:05:34 | mriedem | is my understanding | |
| 17:05:43 | mriedem | 'Default availability zone for new volumes. If not set, ' | |
| 17:05:43 | mriedem | 'the storage_availability_zone option value is used as ' | |
| 17:05:43 | mriedem | 'the default for new volumes.' | |
| 17:06:42 | smcginnis | savvas: So still looking for details, if None is passed in for AZ it fails? | |
| 17:07:02 | savvas | mriedem: ye but let's say I put default_availability_zone to "nova", wouldn't that result in volumes floating across the nodes? I am trying to isolate the volumes to the nodes that serve the instances | |
| 17:07:14 | savvas | smcginnis: sorry, yes if None is passed for AZ, it fails | |
| 17:07:44 | mriedem | savvas: i think the point is you don't set a default_availability_zone value | |
| 17:07:58 | mriedem | setting to 'nova' and leaving unset is probably handled differently in the cinder code | |
| 17:08:24 | smcginnis | mriedem: Oh yeah, that's cinder-wide, not per-backend. | |
| 17:08:41 | mriedem | right, that's what i said above, it's global | |
| 17:08:45 | savvas | mmm | |
| 17:08:50 | mriedem | storage_availability_zone is per node | |
| 17:08:54 | mriedem | default_availability_zone is global | |
| 17:08:55 | savvas | so this value should not be set on the cinder-volumes | |
| 17:09:02 | mriedem | backend_availability_zone is per backed per nod | |
| 17:09:03 | mriedem | *node | |
| 17:09:10 | mriedem | savvas: i don't think so no | |
| 17:09:17 | mriedem | but, i'm not entirely sure, hence roping in smcginnis | |
| 17:09:20 | djohnsto | mriedem: Any thoughts on how to resolve this bug - seems to be disagreement with the fix 193047 | |
| 17:09:47 | mriedem | djohnsto: which bug? | |
| 17:10:02 | mriedem | oh https://review.openstack.org/#/c/193047/ | |
| 17:10:22 | djohnsto | Yep. | |
| 17:10:49 | smcginnis | savvas: I can try to test this out later, but try removing those conflicting default_availability_zone values. | |
| 17:10:58 | mriedem | djohnsto: in a meeting now, | |
| 17:11:04 | mriedem | that probably needs to be in the mailing list | |
| 17:11:08 | mriedem | as it's a change in behavior | |
| 17:11:10 | djohnsto | Ok | |
| 17:14:07 | savvas | smcginnis: I removed it but it didn't change anything in terms of the volume create command failing when AZ is not defined | |
| 17:14:43 | smcginnis | savvas: OK, will have to look in to it then. | |
| 17:17:59 | savvas | smcginnis: appreciate it | |
| 17:19:36 | mriedem | savvas: you restarted the services after changing the config right? | |
| 17:23:58 | savvas | mriedem: yes | |
| 17:53:08 | mriedem | anyone remember when the guy in the nova onboarding session asked how he knows if an instance variable is a dict or an object or something else? | |
| 17:53:20 | mriedem | same question, but block_device_mapping | |
| 17:53:40 | mriedem | is a list, is it a single thing? dict or object? if object, which type of object? | |
| 17:53:42 | mriedem | super fun | |