| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-24 | |||
| 16:00:17 | mriedem | jmlowe: they should be separate databases yes | |
| 16:00:27 | mriedem | you will have 3 separate databases: nova_api, nova_cell0 and nova | |
| 16:00:36 | jmlowe | ok, just checking | |
| 16:00:53 | mriedem | so i think you need to purge the instance_mappings table, and then re-map with the proper cell uuid 0bbfb73d-f761-4c43-ad72-ab47a559615e | |
| 16:01:05 | jmlowe | no harm in ' delete from instance_mappings;' ? | |
| 16:01:07 | mriedem | purging that table shouldn't be a problem since it's all wrong anyway | |
| 16:01:19 | mriedem | dansmith: right? ^ | |
| 16:01:30 | dansmith | yeah | |
| 16:01:50 | jmlowe | rerunning mapping | |
| 16:01:54 | mriedem | savvas: ok can you reply to the question i put in the commit messagE? | |
| 16:02:32 | mriedem | savvas: i.e. the volume gets created successfully but in the default 'nova' AZ, | |
| 16:02:44 | mriedem | which is different from the explicit non-default AZ that the instance is in, | |
| 16:02:47 | mriedem | and that's the bug, right? | |
| 16:02:51 | mriedem | but the boot request doesn't actually fail? | |
| 16:03:31 | savvas | no the boot request is accepted, it then tries to create a volume in AZ nova (default), when it fails it removes the attempted volume creation and leaves the new instance in error state | |
| 16:04:05 | mriedem | savvas: so is nova actually passing availability_zone='nova' to cinder when creating the volume? | |
| 16:04:08 | mriedem | or None? | |
| 16:04:33 | savvas | I believe it selects nova for it not being defined, but I can't tell for sure | |
| 16:04:37 | mriedem | edleafe: in the absence of bauzas, can a compute host live in more than one AZ? | |
| 16:04:47 | savvas | I've been using openstack for a week so I am still finding my way ;) | |
| 16:04:53 | mriedem | savvas: that's kind of important, because if nova is passing az=None then it's a different issue | |
| 16:04:58 | mriedem | savvas: you're a pro now | |
| 16:05:00 | mriedem | congratulations | |
| 16:05:01 | edleafe | mriedem: nope | |
| 16:05:25 | mriedem | ok, but a compute host can be in more than one host aggregate | |
| 16:05:29 | mriedem | that's where i get confused | |
| 16:05:30 | edleafe | sure | |
| 16:05:40 | edleafe | yes, it's very terrible | |
| 16:05:55 | mriedem | https://docs.openstack.org/developer/nova/aggregates.html | |
| 16:05:59 | mriedem | "one host can be in multiple aggregates, but it can only be in one availability zone" | |
| 16:06:03 | mriedem | i guess i should have read the manual | |
| 16:06:28 | savvas | mriedem: | OS-EXT-AZ:availability_zone | | |
| 16:06:33 | savvas | it leaves the zone empty when not defined | |
| 16:06:40 | savvas | so it doesn't explicitly pass on nova | |
| 16:07:19 | mriedem | savvas: ok so i don't see why the volume creation fails then | |
| 16:07:30 | mriedem | if nova isn't passing an AZ, cinder will create the volume in it's default AZ, which is 'nova' | |
| 16:08:01 | mriedem | i can see why / how the instance and volume can get into different AZs here, which is a bug | |
| 16:08:05 | savvas | mriedem: exactly, I don't have any cinder-volumes available in nova though | |
| 16:08:36 | savvas | right now I have 3 hypervisors setup in my test environment, all 3 in separate aggregates with separate storage zones | |
| 16:08:45 | savvas | in my case it looks like this: | |
| 16:08:55 | savvas | host-aggregate hv01-zone, has 1 member, hv01 | |
| 16:09:05 | savvas | cinder on hv01 has storage/default zone set to hv01-zone | |
| 16:09:16 | savvas | and I follow the same structure for hv02 and hv03 | |
| 16:09:32 | savvas | but the default nova group has no usable cinder volumes available in my setup | |
| 16:09:45 | savvas | so when the AZ is not defined, it fails | |
| 16:10:12 | mriedem | are you setting storage_availability_zone in cinder.conf on your 3 storage nodes? | |
| 16:10:54 | mriedem | i don't really know how cinder handles the AZs on their side | |
| 16:11:07 | mriedem | so i don't know what the actual error looks like when creating the volume on the cinder side, | |
| 16:11:13 | mriedem | if you have those details you could put them in that bug | |
| 16:12:00 | savvas | mriedem: yes, they are all on all 3 nodes | |
| 16:12:01 | jmlowe | ok, looking much better | |
| 16:12:18 | mriedem | jmlowe: sweet | |
| 16:12:32 | mriedem | rpodolyaka: can you explain where we fail with https://review.openstack.org/#/c/366724/ during boot? | |
| 16:12:35 | mriedem | is it during attach? | |
| 16:13:04 | mriedem | savvas: the volume gets created successfully though right? but then something else fails? | |
| 16:15:36 | savvas | mriedem: when the AZ upon instance creation is defined, yes. When the AZ upon instance creation is not defined, then no. When I create a volume manually via the CLI in the nova zone (so create a volume without defining AZ), it creates just fine | |
| 16:15:37 | mriedem | i think i see it | |
| 16:15:49 | jmlowe | I blame my complete lack of reading comprehension on my new monitor where the text is all very small and cannot be increased lest it injure my pride | |
| 16:16:28 | savvas | mriedem: let me correct that, the creation fails, sorry it accepts the command but it fails to create | |
| 16:16:28 | mriedem | jmlowe: so now that you've gone through this, it would be interesting to know which docs you read before you started upgrading | |
| 16:16:43 | mriedem | savvas: i think it fails here https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L368 | |
| 16:16:48 | savvas | which in my opinion is logical behavior since there are no cinder-volume services running in the nova zone | |
| 16:16:59 | jmlowe | I half read this https://docs.openstack.org/releasenotes/nova/newton.html | |
| 16:17:23 | mriedem | jmlowe: did you see any of https://docs.openstack.org/developer/nova/cells.html#upgrade-minimal ? | |
| 16:17:32 | jmlowe | wasn't really thinking about what I was doing, was irritated by the cells0 db missing warning from nova-manage db sync | |
| 16:18:03 | mriedem | ^ is really the page with the basic instructions on upgrading to cells v2 | |
| 16:18:20 | mriedem | so it'd be good info to know if people aren't aware of it, or it doesn't make sense in some parts, or there are things missing | |
| 16:18:25 | jmlowe | no, and that warning about the db name would definitely save me some headaches | |
| 16:18:42 | mriedem | which warning? | |
| 16:18:46 | mriedem | oh i see | |
| 16:18:47 | mriedem | in the doc | |
| 16:19:12 | mriedem | yeah, so simple_cell_setup defaulting the cell0 db name to nova_api_cell0 is a bug | |
| 16:19:13 | savvas | mriedem: I don't think so but i could be mistaken. To me it looks like a bug in determining where to deploy. If you explicitly define a zone, it will grab the right storage az. If you don't define a zone, it will grab an available HV AZ but it won't grab its corresponding available storage az | |
| 16:19:43 | jmlowe | so I'm still a little concerned that my transport_url for cell0 is none:/// | |
| 16:19:59 | jmlowe | is that really not a problem? | |
| 16:20:00 | dansmith | jmlowe: yeah, that's wrong | |
| 16:20:11 | dansmith | jmlowe: you'll fail when some instance doesn't schedule if tat | |
| 16:20:25 | dansmith | jmlowe: but you don't need to do anything other than fix that record at this point.. no need to remap or anything | |
| 16:20:34 | dansmith | oh | |
| 16:20:35 | dansmith | sorry | |
| 16:20:36 | dansmith | transport_url | |
| 16:20:40 | mriedem | that's normal | |
| 16:20:44 | dansmith | yeah, there's no transport for cell0 | |
| 16:20:48 | dansmith | yeah sorry, was thinking database | |
| 16:21:00 | mriedem | cell0 is a db which is a graveyard for instances that failed to schedule | |
| 16:21:19 | jmlowe | ah, ok that makes more sense | |
| 16:21:33 | mriedem | maybe, "Why is the transport_url for cell0 none:/// ?" should be in the FAQs | |
| 16:22:18 | mriedem | savvas: i've asked in https://review.openstack.org/#/c/366724/ which is the current proposed fix for the bug, | |
| 16:22:35 | mriedem | at this point i really need a clear idea of what the actual failure is, either in reply to that question in the patch or in the bug report | |
| 16:22:41 | mriedem | but we're kind of going in circle | |
| 16:22:44 | mriedem | *circles | |
| 16:23:17 | mriedem | i understand there is a problem, but i'd like to make sure i understand the problem before reviewing a fix, and i'd also like to consider some integration testing for this since it's complicated | |
| 16:27:15 | savvas | mriedem: alright let me see how I can get a more conclusive answer for you | |
| 16:33:25 | mriedem | jmlowe: btw i'm sort of working on backporting that cell0 db name fix to newton https://review.openstack.org/#/c/420461/ but it gets a bit complicated, and at this point i'm not sure if it helps anyone or not, but i'd surely like to help avoid people from falling into this trap in the first place where they have a nova_api_cell0 db in newton and then need to rename that to nova_cell0 in ocata | |
| 16:33:38 | savvas | mriedem: in my cinder scheduler log I caught this : default default] No weighed backend found for volume with properties: None. | |
| 16:33:52 | savvas | after attempting to create an instance without defining the AZ | |
| 16:34:29 | savvas | so nova passed on the None value for the AZ to cinder I guess | |
| 16:34:44 | mriedem | jmlowe: thoughts on that? would that at least be helpful to give people in the operators ML a heads up? | |
| 16:35:20 | mriedem | savvas: right and i'm saying that's no different from the case that cross_az_attach=True, which is why i don't think it would result in the volume create failing | |
| 16:35:28 | mriedem | so i think there is something else happening | |