Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-24
16:12:35 mriedem is it during attach?
16:13:04 mriedem savvas: the volume gets created successfully though right? but then something else fails?
16:15:36 savvas mriedem: when the AZ upon instance creation is defined, yes. When the AZ upon instance creation is not defined, then no. When I create a volume manually via the CLI in the nova zone (so create a volume without defining AZ), it creates just fine
16:15:37 mriedem i think i see it
16:15:49 jmlowe I blame my complete lack of reading comprehension on my new monitor where the text is all very small and cannot be increased lest it injure my pride
16:16:28 savvas mriedem: let me correct that, the creation fails, sorry it accepts the command but it fails to create
16:16:28 mriedem jmlowe: so now that you've gone through this, it would be interesting to know which docs you read before you started upgrading
16:16:43 mriedem savvas: i think it fails here https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L368
16:16:48 savvas which in my opinion is logical behavior since there are no cinder-volume services running in the nova zone
16:16:59 jmlowe I half read this https://docs.openstack.org/releasenotes/nova/newton.html
16:17:23 mriedem jmlowe: did you see any of https://docs.openstack.org/developer/nova/cells.html#upgrade-minimal ?
16:17:32 jmlowe wasn't really thinking about what I was doing, was irritated by the cells0 db missing warning from nova-manage db sync
16:18:03 mriedem ^ is really the page with the basic instructions on upgrading to cells v2
16:18:20 mriedem so it'd be good info to know if people aren't aware of it, or it doesn't make sense in some parts, or there are things missing
16:18:25 jmlowe no, and that warning about the db name would definitely save me some headaches
16:18:42 mriedem which warning?
16:18:46 mriedem oh i see
16:18:47 mriedem in the doc
16:19:12 mriedem yeah, so simple_cell_setup defaulting the cell0 db name to nova_api_cell0 is a bug
16:19:13 savvas mriedem: I don't think so but i could be mistaken. To me it looks like a bug in determining where to deploy. If you explicitly define a zone, it will grab the right storage az. If you don't define a zone, it will grab an available HV AZ but it won't grab its corresponding available storage az
16:19:43 jmlowe so I'm still a little concerned that my transport_url for cell0 is none:///
16:19:59 jmlowe is that really not a problem?
16:20:00 dansmith jmlowe: yeah, that's wrong
16:20:11 dansmith jmlowe: you'll fail when some instance doesn't schedule if tat
16:20:25 dansmith jmlowe: but you don't need to do anything other than fix that record at this point.. no need to remap or anything
16:20:34 dansmith oh
16:20:35 dansmith sorry
16:20:36 dansmith transport_url
16:20:40 mriedem that's normal
16:20:44 dansmith yeah, there's no transport for cell0
16:20:48 dansmith yeah sorry, was thinking database
16:21:00 mriedem cell0 is a db which is a graveyard for instances that failed to schedule
16:21:19 jmlowe ah, ok that makes more sense
16:21:33 mriedem maybe, "Why is the transport_url for cell0 none:/// ?" should be in the FAQs
16:22:18 mriedem savvas: i've asked in https://review.openstack.org/#/c/366724/ which is the current proposed fix for the bug,
16:22:35 mriedem at this point i really need a clear idea of what the actual failure is, either in reply to that question in the patch or in the bug report
16:22:41 mriedem but we're kind of going in circle
16:22:44 mriedem *circles
16:23:17 mriedem i understand there is a problem, but i'd like to make sure i understand the problem before reviewing a fix, and i'd also like to consider some integration testing for this since it's complicated
16:27:15 savvas mriedem: alright let me see how I can get a more conclusive answer for you
16:33:25 mriedem jmlowe: btw i'm sort of working on backporting that cell0 db name fix to newton https://review.openstack.org/#/c/420461/ but it gets a bit complicated, and at this point i'm not sure if it helps anyone or not, but i'd surely like to help avoid people from falling into this trap in the first place where they have a nova_api_cell0 db in newton and then need to rename that to nova_cell0 in ocata
16:33:38 savvas mriedem: in my cinder scheduler log I caught this : default default] No weighed backend found for volume with properties: None.
16:33:52 savvas after attempting to create an instance without defining the AZ
16:34:29 savvas so nova passed on the None value for the AZ to cinder I guess
16:34:44 mriedem jmlowe: thoughts on that? would that at least be helpful to give people in the operators ML a heads up?
16:35:20 mriedem savvas: right and i'm saying that's no different from the case that cross_az_attach=True, which is why i don't think it would result in the volume create failing
16:35:28 mriedem so i think there is something else happening
16:36:48 savvas mmm, any thoughts as to where I should be looking? I've been tailing the nova and cinder logs to see what happens when I call for a nova boot
16:38:33 savvas mriedem: either way, I'll keep working on it, if I find anything useful I'll be sure to let you know. Thanks for your time!
16:39:41 mriedem savvas: if the instance goes to error state, it's mostly likely something failing in the nova-compute logs
16:41:27 savvas mriedem: ye from those logs I only find the error I initially reported: BuildAbortException: Build of instance 9484f5a7-3198-47ff-b728-178515a26277 aborted: Block Device Mapping is Invalid.
16:42:11 mriedem savvas: ok i think that's resulting from https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/virt/block_device.py#L368
16:42:53 mriedem savvas: don't you see a traceback from here? https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/compute/manager.py#L1593
16:43:28 mriedem wow i guess we really don't try to help out with the error message in this ase
16:43:29 mriedem *Case
16:44:08 savvas mriedem:
16:44:09 savvas 2017-05-24 16:33:26.127 2331 ERROR nova.compute.manager [req-b5388b21-5f75-4946-8ace-56cd91ebdc72 be0e3c8d93f345a6889e76d1507ba130 7509210fbb1a4e30bf62d81aa3b4e369 - - -] [instance: 9484f5a7-3198-47ff-b728-178515a26277] Instance failed block device setup
16:44:22 mriedem savvas: there should be a traceback
16:44:27 mriedem please put that into a paste and link here
16:44:41 mriedem it's using LOG.exception which will traceback
16:44:53 savvas ye grabbed this with cat, looking it up now
16:45:57 savvas http://paste.as47869.net/p/9qbburh7z3w3toi
16:49:15 mriedem VolumeNotCreated: Volume da947c97-66c6-4b7e-9ae6-54eb8128bb75 did not finish being created even after we waited 3 seconds or 2 attempts. And its status is error.
16:49:23 mriedem that is a different issue
16:49:37 jaypipes gosh darn it...
16:49:52 mriedem penick: ^ is why bfv is awful
16:50:12 mriedem savvas: you're failing in here https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/compute/manager.py#L1222
16:50:25 mriedem it's timing out waiting for the volume that nova created to be available so it can attach it to the instance
16:50:43 mriedem you probably need to setup image cache on the cinder side, and tweak the timeout values on the nova compute side
16:51:09 penick Is its disease of awfulness irreparable?
16:51:10 smcginnis Larger images may take a long time.
16:52:04 savvas mriedem: the volume fails to create though
16:52:27 savvas it fails instantly, as it tries to do so in an AZ which isn't available
16:52:31 mriedem savvas: does the volume go to error state?
16:52:37 savvas yes
16:52:45 mriedem savvas: but i thought nova was passing az=None when creating the volume?
16:52:51 savvas correct
16:53:12 savvas which leads cinder to attempt and create the volume in the default az of "nova"
16:53:15 savvas which fails
16:53:33 mriedem my head is going to explode
16:53:46 openstackgerrit Jay Pipes proposed openstack/nova master: integrate trait cache with TraitList and Trait https://review.openstack.org/462770
16:53:54 mriedem ok, so is the problem that things get more strict on the cinder side when you actually have AZs in cinder?
16:54:07 mriedem because if you don't have az's in cinder, you can create volumes w/o an az (by default in the nova az) just fine
16:54:14 mriedem but once you add an az, cinder won't allow that anymore?
16:54:48 savvas ye I am sorry if I am making you go in circles here, like I said my lack of exp isn't helping much either I am sure
16:54:51 mriedem by the way, we should totally get that VolumeNotCreated error into https://github.com/openstack/nova/blob/81bdbd0b50aeac9a677a0cef9001081008a2c407/nova/compute/manager.py#L1595 which is fed back up and goes into the instance fault
16:55:07 mriedem savvas: don't worry about it, you're doing great - i obviously don't know how anything works here either :)
16:55:46 mriedem what i'd like to do is know a simple recreate so i can write a tempest test for this
16:56:04 kashyap Does anyone know for sure, about whether the double quotes are *mandatory* when you specify a key-value in a Nova config attribute as following:
16:56:07 kashyap [libvirt]
16:56:10 kashyap hw_machine_type="x86_64=pc-i440fx-rhel7.2.0"
16:56:12 savvas I run 3 cinder volume servers, each with their own AZ. The controller has no storage/default AZ defined
16:56:20 savvas mriedem: as in steps to reproduce this error?
16:57:10 mriedem savvas: i think just even knowing if you can create a volume in cinder w/o specifying AZ if there is a non-default AZ defined, just one
16:57:29 savvas no, that doesn't work. At this stage I HAVE to specify an AZ
16:57:29 jmlowe mriedem: yeah, I'm going to trip over this later
16:58:04 mriedem savvas: huh, ok, figured cinder would allow it if az isn't specified
16:58:12 mriedem smcginnis: jgriffith: ^ is that by design?
16:58:21 mriedem hell it might even be how nova does it but i'm not sure
16:58:39 smcginnis mriedem: Didn't follow everything, but are you saying if AZ is None, Cinder fails the call?

Earlier   Later