| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-24 | |||
| 15:58:38 | mriedem | jmlowe: the problem is the instance is mapped to cell0 | |
| 15:58:49 | jmlowe | yes | |
| 15:59:02 | openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for resource classes https://review.openstack.org/467596 | |
| 15:59:26 | savvas | mriedem: https://review.openstack.org/#/c/225119/ this right here what you just sent me is exactly what I am seeing | |
| 15:59:55 | mriedem | jmlowe: so i think what probably happened was, the instances got mapped to cell0 by mistake, somehow, i'm not sure how - but now when you run map_instances with the correct cell 0bbfb73d-f761-4c43-ad72-ab47a559615e it does nothing b/c they are already mapped, | |
| 16:00:02 | jmlowe | should the database connection for cell0 and the legacy nova be different? | |
| 16:00:05 | mriedem | jmlowe: so i think you basically need to purge your instance_mappings table | |
| 16:00:17 | mriedem | jmlowe: they should be separate databases yes | |
| 16:00:27 | mriedem | you will have 3 separate databases: nova_api, nova_cell0 and nova | |
| 16:00:36 | jmlowe | ok, just checking | |
| 16:00:53 | mriedem | so i think you need to purge the instance_mappings table, and then re-map with the proper cell uuid 0bbfb73d-f761-4c43-ad72-ab47a559615e | |
| 16:01:05 | jmlowe | no harm in ' delete from instance_mappings;' ? | |
| 16:01:07 | mriedem | purging that table shouldn't be a problem since it's all wrong anyway | |
| 16:01:19 | mriedem | dansmith: right? ^ | |
| 16:01:30 | dansmith | yeah | |
| 16:01:50 | jmlowe | rerunning mapping | |
| 16:01:54 | mriedem | savvas: ok can you reply to the question i put in the commit messagE? | |
| 16:02:32 | mriedem | savvas: i.e. the volume gets created successfully but in the default 'nova' AZ, | |
| 16:02:44 | mriedem | which is different from the explicit non-default AZ that the instance is in, | |
| 16:02:47 | mriedem | and that's the bug, right? | |
| 16:02:51 | mriedem | but the boot request doesn't actually fail? | |
| 16:03:31 | savvas | no the boot request is accepted, it then tries to create a volume in AZ nova (default), when it fails it removes the attempted volume creation and leaves the new instance in error state | |
| 16:04:05 | mriedem | savvas: so is nova actually passing availability_zone='nova' to cinder when creating the volume? | |
| 16:04:08 | mriedem | or None? | |
| 16:04:33 | savvas | I believe it selects nova for it not being defined, but I can't tell for sure | |
| 16:04:37 | mriedem | edleafe: in the absence of bauzas, can a compute host live in more than one AZ? | |
| 16:04:47 | savvas | I've been using openstack for a week so I am still finding my way ;) | |
| 16:04:53 | mriedem | savvas: that's kind of important, because if nova is passing az=None then it's a different issue | |
| 16:04:58 | mriedem | savvas: you're a pro now | |
| 16:05:00 | mriedem | congratulations | |
| 16:05:01 | edleafe | mriedem: nope | |
| 16:05:25 | mriedem | ok, but a compute host can be in more than one host aggregate | |
| 16:05:29 | mriedem | that's where i get confused | |
| 16:05:30 | edleafe | sure | |
| 16:05:40 | edleafe | yes, it's very terrible | |
| 16:05:55 | mriedem | https://docs.openstack.org/developer/nova/aggregates.html | |
| 16:05:59 | mriedem | "one host can be in multiple aggregates, but it can only be in one availability zone" | |
| 16:06:03 | mriedem | i guess i should have read the manual | |
| 16:06:28 | savvas | mriedem: | OS-EXT-AZ:availability_zone | | |
| 16:06:33 | savvas | it leaves the zone empty when not defined | |
| 16:06:40 | savvas | so it doesn't explicitly pass on nova | |
| 16:07:19 | mriedem | savvas: ok so i don't see why the volume creation fails then | |
| 16:07:30 | mriedem | if nova isn't passing an AZ, cinder will create the volume in it's default AZ, which is 'nova' | |
| 16:08:01 | mriedem | i can see why / how the instance and volume can get into different AZs here, which is a bug | |
| 16:08:05 | savvas | mriedem: exactly, I don't have any cinder-volumes available in nova though | |
| 16:08:36 | savvas | right now I have 3 hypervisors setup in my test environment, all 3 in separate aggregates with separate storage zones | |
| 16:08:45 | savvas | in my case it looks like this: | |
| 16:08:55 | savvas | host-aggregate hv01-zone, has 1 member, hv01 | |
| 16:09:05 | savvas | cinder on hv01 has storage/default zone set to hv01-zone | |
| 16:09:16 | savvas | and I follow the same structure for hv02 and hv03 | |
| 16:09:32 | savvas | but the default nova group has no usable cinder volumes available in my setup | |
| 16:09:45 | savvas | so when the AZ is not defined, it fails | |
| 16:10:12 | mriedem | are you setting storage_availability_zone in cinder.conf on your 3 storage nodes? | |
| 16:10:54 | mriedem | i don't really know how cinder handles the AZs on their side | |
| 16:11:07 | mriedem | so i don't know what the actual error looks like when creating the volume on the cinder side, | |
| 16:11:13 | mriedem | if you have those details you could put them in that bug | |
| 16:12:00 | savvas | mriedem: yes, they are all on all 3 nodes | |
| 16:12:01 | jmlowe | ok, looking much better | |
| 16:12:18 | mriedem | jmlowe: sweet | |
| 16:12:32 | mriedem | rpodolyaka: can you explain where we fail with https://review.openstack.org/#/c/366724/ during boot? | |
| 16:12:35 | mriedem | is it during attach? | |
| 16:13:04 | mriedem | savvas: the volume gets created successfully though right? but then something else fails? | |
| 16:15:36 | savvas | mriedem: when the AZ upon instance creation is defined, yes. When the AZ upon instance creation is not defined, then no. When I create a volume manually via the CLI in the nova zone (so create a volume without defining AZ), it creates just fine | |
| 16:15:37 | mriedem | i think i see it | |
| 16:15:49 | jmlowe | I blame my complete lack of reading comprehension on my new monitor where the text is all very small and cannot be increased lest it injure my pride | |
| 16:16:28 | mriedem | jmlowe: so now that you've gone through this, it would be interesting to know which docs you read before you started upgrading | |
| 16:16:28 | savvas | mriedem: let me correct that, the creation fails, sorry it accepts the command but it fails to create | |
| 16:16:43 | mriedem | savvas: i think it fails here https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L368 | |
| 16:16:48 | savvas | which in my opinion is logical behavior since there are no cinder-volume services running in the nova zone | |
| 16:16:59 | jmlowe | I half read this https://docs.openstack.org/releasenotes/nova/newton.html | |
| 16:17:23 | mriedem | jmlowe: did you see any of https://docs.openstack.org/developer/nova/cells.html#upgrade-minimal ? | |
| 16:17:32 | jmlowe | wasn't really thinking about what I was doing, was irritated by the cells0 db missing warning from nova-manage db sync | |
| 16:18:03 | mriedem | ^ is really the page with the basic instructions on upgrading to cells v2 | |
| 16:18:20 | mriedem | so it'd be good info to know if people aren't aware of it, or it doesn't make sense in some parts, or there are things missing | |
| 16:18:25 | jmlowe | no, and that warning about the db name would definitely save me some headaches | |
| 16:18:42 | mriedem | which warning? | |
| 16:18:46 | mriedem | oh i see | |
| 16:18:47 | mriedem | in the doc | |
| 16:19:12 | mriedem | yeah, so simple_cell_setup defaulting the cell0 db name to nova_api_cell0 is a bug | |
| 16:19:13 | savvas | mriedem: I don't think so but i could be mistaken. To me it looks like a bug in determining where to deploy. If you explicitly define a zone, it will grab the right storage az. If you don't define a zone, it will grab an available HV AZ but it won't grab its corresponding available storage az | |
| 16:19:43 | jmlowe | so I'm still a little concerned that my transport_url for cell0 is none:/// | |
| 16:19:59 | jmlowe | is that really not a problem? | |
| 16:20:00 | dansmith | jmlowe: yeah, that's wrong | |
| 16:20:11 | dansmith | jmlowe: you'll fail when some instance doesn't schedule if tat | |
| 16:20:25 | dansmith | jmlowe: but you don't need to do anything other than fix that record at this point.. no need to remap or anything | |
| 16:20:34 | dansmith | oh | |
| 16:20:35 | dansmith | sorry | |
| 16:20:36 | dansmith | transport_url | |
| 16:20:40 | mriedem | that's normal | |
| 16:20:44 | dansmith | yeah, there's no transport for cell0 | |
| 16:20:48 | dansmith | yeah sorry, was thinking database | |
| 16:21:00 | mriedem | cell0 is a db which is a graveyard for instances that failed to schedule | |
| 16:21:19 | jmlowe | ah, ok that makes more sense | |
| 16:21:33 | mriedem | maybe, "Why is the transport_url for cell0 none:/// ?" should be in the FAQs | |
| 16:22:18 | mriedem | savvas: i've asked in https://review.openstack.org/#/c/366724/ which is the current proposed fix for the bug, | |
| 16:22:35 | mriedem | at this point i really need a clear idea of what the actual failure is, either in reply to that question in the patch or in the bug report | |
| 16:22:41 | mriedem | but we're kind of going in circle | |
| 16:22:44 | mriedem | *circles | |
| 16:23:17 | mriedem | i understand there is a problem, but i'd like to make sure i understand the problem before reviewing a fix, and i'd also like to consider some integration testing for this since it's complicated | |
| 16:27:15 | savvas | mriedem: alright let me see how I can get a more conclusive answer for you | |