| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-24 | |||
| 16:23:17 | mriedem | i understand there is a problem, but i'd like to make sure i understand the problem before reviewing a fix, and i'd also like to consider some integration testing for this since it's complicated | |
| 16:27:15 | savvas | mriedem: alright let me see how I can get a more conclusive answer for you | |
| 16:33:25 | mriedem | jmlowe: btw i'm sort of working on backporting that cell0 db name fix to newton https://review.openstack.org/#/c/420461/ but it gets a bit complicated, and at this point i'm not sure if it helps anyone or not, but i'd surely like to help avoid people from falling into this trap in the first place where they have a nova_api_cell0 db in newton and then need to rename that to nova_cell0 in ocata | |
| 16:33:38 | savvas | mriedem: in my cinder scheduler log I caught this : default default] No weighed backend found for volume with properties: None. | |
| 16:33:52 | savvas | after attempting to create an instance without defining the AZ | |
| 16:34:29 | savvas | so nova passed on the None value for the AZ to cinder I guess | |
| 16:34:44 | mriedem | jmlowe: thoughts on that? would that at least be helpful to give people in the operators ML a heads up? | |
| 16:35:20 | mriedem | savvas: right and i'm saying that's no different from the case that cross_az_attach=True, which is why i don't think it would result in the volume create failing | |
| 16:35:28 | mriedem | so i think there is something else happening | |
| 16:36:48 | savvas | mmm, any thoughts as to where I should be looking? I've been tailing the nova and cinder logs to see what happens when I call for a nova boot | |
| 16:38:33 | savvas | mriedem: either way, I'll keep working on it, if I find anything useful I'll be sure to let you know. Thanks for your time! | |
| 16:39:41 | mriedem | savvas: if the instance goes to error state, it's mostly likely something failing in the nova-compute logs | |
| 16:41:27 | savvas | mriedem: ye from those logs I only find the error I initially reported: BuildAbortException: Build of instance 9484f5a7-3198-47ff-b728-178515a26277 aborted: Block Device Mapping is Invalid. | |
| 16:42:11 | mriedem | savvas: ok i think that's resulting from https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L368 | |
| 16:42:53 | mriedem | savvas: don't you see a traceback from here? https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/compute/manager.py#L1593 | |
| 16:43:28 | mriedem | wow i guess we really don't try to help out with the error message in this ase | |
| 16:43:29 | mriedem | *Case | |
| 16:44:08 | savvas | mriedem: | |
| 16:44:09 | savvas | 2017-05-24 16:33:26.127 2331 ERROR nova.compute.manager [req-b5388b21-5f75-4946-8ace-56cd91ebdc72 be0e3c8d93f345a6889e76d1507ba130 7509210fbb1a4e30bf62d81aa3b4e369 - - -] [instance: 9484f5a7-3198-47ff-b728-178515a26277] Instance failed block device setup | |
| 16:44:22 | mriedem | savvas: there should be a traceback | |
| 16:44:27 | mriedem | please put that into a paste and link here | |
| 16:44:41 | mriedem | it's using LOG.exception which will traceback | |
| 16:44:53 | savvas | ye grabbed this with cat, looking it up now | |
| 16:45:57 | savvas | http://paste.as47869.net/p/9qbburh7z3w3toi | |
| 16:49:15 | mriedem | VolumeNotCreated: Volume da947c97-66c6-4b7e-9ae6-54eb8128bb75 did not finish being created even after we waited 3 seconds or 2 attempts. And its status is error. | |
| 16:49:23 | mriedem | that is a different issue | |
| 16:49:37 | jaypipes | gosh darn it... | |
| 16:49:52 | mriedem | penick: ^ is why bfv is awful | |
| 16:50:12 | mriedem | savvas: you're failing in here https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/compute/manager.py#L1222 | |
| 16:50:25 | mriedem | it's timing out waiting for the volume that nova created to be available so it can attach it to the instance | |
| 16:50:43 | mriedem | you probably need to setup image cache on the cinder side, and tweak the timeout values on the nova compute side | |
| 16:51:09 | penick | Is its disease of awfulness irreparable? | |
| 16:51:10 | smcginnis | Larger images may take a long time. | |
| 16:52:04 | savvas | mriedem: the volume fails to create though | |
| 16:52:27 | savvas | it fails instantly, as it tries to do so in an AZ which isn't available | |
| 16:52:31 | mriedem | savvas: does the volume go to error state? | |
| 16:52:37 | savvas | yes | |
| 16:52:45 | mriedem | savvas: but i thought nova was passing az=None when creating the volume? | |
| 16:52:51 | savvas | correct | |
| 16:53:12 | savvas | which leads cinder to attempt and create the volume in the default az of "nova" | |
| 16:53:15 | savvas | which fails | |
| 16:53:33 | mriedem | my head is going to explode | |
| 16:53:46 | openstackgerrit | Jay Pipes proposed openstack/nova master: integrate trait cache with TraitList and Trait https://review.openstack.org/462770 | |
| 16:53:54 | mriedem | ok, so is the problem that things get more strict on the cinder side when you actually have AZs in cinder? | |
| 16:54:07 | mriedem | because if you don't have az's in cinder, you can create volumes w/o an az (by default in the nova az) just fine | |
| 16:54:14 | mriedem | but once you add an az, cinder won't allow that anymore? | |
| 16:54:48 | savvas | ye I am sorry if I am making you go in circles here, like I said my lack of exp isn't helping much either I am sure | |
| 16:54:51 | mriedem | by the way, we should totally get that VolumeNotCreated error into https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/compute/manager.py#L1595 which is fed back up and goes into the instance fault | |
| 16:55:07 | mriedem | savvas: don't worry about it, you're doing great - i obviously don't know how anything works here either :) | |
| 16:55:46 | mriedem | what i'd like to do is know a simple recreate so i can write a tempest test for this | |
| 16:56:04 | kashyap | Does anyone know for sure, about whether the double quotes are *mandatory* when you specify a key-value in a Nova config attribute as following: | |
| 16:56:07 | kashyap | [libvirt] | |
| 16:56:10 | kashyap | hw_machine_type="x86_64=pc-i440fx-rhel7.2.0" | |
| 16:56:12 | savvas | I run 3 cinder volume servers, each with their own AZ. The controller has no storage/default AZ defined | |
| 16:56:20 | savvas | mriedem: as in steps to reproduce this error? | |
| 16:57:10 | mriedem | savvas: i think just even knowing if you can create a volume in cinder w/o specifying AZ if there is a non-default AZ defined, just one | |
| 16:57:29 | savvas | no, that doesn't work. At this stage I HAVE to specify an AZ | |
| 16:57:29 | jmlowe | mriedem: yeah, I'm going to trip over this later | |
| 16:58:04 | mriedem | savvas: huh, ok, figured cinder would allow it if az isn't specified | |
| 16:58:12 | mriedem | smcginnis: jgriffith: ^ is that by design? | |
| 16:58:21 | mriedem | hell it might even be how nova does it but i'm not sure | |
| 16:58:39 | smcginnis | mriedem: Didn't follow everything, but are you saying if AZ is None, Cinder fails the call? | |
| 16:59:44 | mriedem | smcginnis: is there is a non-default AZ defined in cinder | |
| 17:00:03 | mriedem | so if i create az=foo in cinder, | |
| 17:00:10 | mriedem | do i have to create all volumes explicitly in that az now? | |
| 17:00:34 | smcginnis | mriedem: I didn't think so. | |
| 17:00:39 | smcginnis | Checking... | |
| 17:01:39 | smcginnis | So if storage_availability_zone is set to something other than the default, you may need to also update default_availability_zone. Still looking though. | |
| 17:02:19 | savvas | smcginnis: I have both values set on each node running cinder-volume service | |
| 17:02:32 | mriedem | dansmith: are we supposed to be in meeting-3 for cells? | |
| 17:02:39 | mriedem | and the interop guys are going long? | |
| 17:02:43 | smcginnis | savvas: So both are set to unique values, but if None is passed in for AZ it fails? | |
| 17:03:40 | savvas | smcginnis: I have both set to the zone available to the node. I run nova compute and cinder volume on the same host for this purpose. So hv01 would have default/storage_availability_zone set to hv01-zone, hv02 to hv02-zone and so on | |
| 17:04:07 | mriedem | i think that's wrong | |
| 17:04:15 | mriedem | i think default_availability_zone is supposed to be global | |
| 17:04:21 | savvas | as these hypervisors are supposed to be dependent of their local storage, they shouldn't be looking to place volumes in any other zones | |
| 17:04:44 | melwitt | mriedem: yeah on my calendar we should have a cells meeting right now | |
| 17:05:24 | mriedem | that's what storage_availability_zone is for, | |
| 17:05:30 | mriedem | and it overrides default_availability_zone | |
| 17:05:34 | mriedem | is my understanding | |
| 17:05:43 | mriedem | 'Default availability zone for new volumes. If not set, ' | |
| 17:05:43 | mriedem | 'the storage_availability_zone option value is used as ' | |
| 17:05:43 | mriedem | 'the default for new volumes.' | |
| 17:06:42 | smcginnis | savvas: So still looking for details, if None is passed in for AZ it fails? | |
| 17:07:02 | savvas | mriedem: ye but let's say I put default_availability_zone to "nova", wouldn't that result in volumes floating across the nodes? I am trying to isolate the volumes to the nodes that serve the instances | |
| 17:07:14 | savvas | smcginnis: sorry, yes if None is passed for AZ, it fails | |
| 17:07:44 | mriedem | savvas: i think the point is you don't set a default_availability_zone value | |
| 17:07:58 | mriedem | setting to 'nova' and leaving unset is probably handled differently in the cinder code | |
| 17:08:24 | smcginnis | mriedem: Oh yeah, that's cinder-wide, not per-backend. | |
| 17:08:41 | mriedem | right, that's what i said above, it's global | |
| 17:08:45 | savvas | mmm | |
| 17:08:50 | mriedem | storage_availability_zone is per node | |
| 17:08:54 | mriedem | default_availability_zone is global | |
| 17:08:55 | savvas | so this value should not be set on the cinder-volumes | |
| 17:09:02 | mriedem | backend_availability_zone is per backed per nod | |
| 17:09:03 | mriedem | *node | |
| 17:09:10 | mriedem | savvas: i don't think so no | |
| 17:09:17 | mriedem | but, i'm not entirely sure, hence roping in smcginnis | |
| 17:09:20 | djohnsto | mriedem: Any thoughts on how to resolve this bug - seems to be disagreement with the fix 193047 | |
| 17:09:47 | mriedem | djohnsto: which bug? | |