| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-24 | |||
| 15:41:23 | jmlowe | ran the simple setup, figured out I forgot to create a new db for cell0, created db, did map_cell0/map_cell_and_hosts/map_instances | |
| 15:41:24 | mriedem | will tell you what you have setup, i'm guessing there are 2 entries | |
| 15:41:47 | mriedem | ok, when you run map_instances, it should be pointing at the non-cell0 cell | |
| 15:41:53 | mriedem | which is your existing main 'nova' database | |
| 15:41:55 | mriedem | with the instances | |
| 15:42:55 | jmlowe | yep, ok, that was not what I was doing | |
| 15:45:28 | jmlowe | so that I can stop panicking, is there a way to go back to no cells in newton? | |
| 15:47:09 | mriedem | i don't think nova-manage cell_v2 delete_cell existed in newton, | |
| 15:47:14 | mriedem | i mean you could hack the db, | |
| 15:47:26 | mriedem | but honestly i can't really advise how to rollback | |
| 15:47:48 | mriedem | jmlowe: i think the reason you're getting the 404 on instance delete is because the instance isn't mapped to the cell probably | |
| 15:48:12 | mriedem | we might not have had the list_cells command either - you'd have to see what's in the nova_api db cell_mappings table | |
| 15:48:15 | mriedem | are there 2 entries in there? | |
| 15:48:25 | jmlowe | there are two in there | |
| 15:48:36 | mriedem | ok, one is pointing at your nova_cell0 db? | |
| 15:48:39 | mriedem | and the other is pointing at your nova db? | |
| 15:48:53 | jmlowe | ok, did map_instances, this time with the correct uuid, no luck with nova show | |
| 15:49:19 | mriedem | do you see the uuid for the instance in the nova_api.instance_mappings table? | |
| 15:50:01 | mriedem | savvas: i'm not sure how that is going to affect this - yes nova will pick a host which may be in an AZ, but i didn't think we set instance.availability_zone in that case if the user didn't explicitly request an az | |
| 15:50:36 | jmlowe | http://paste.openstack.org/show/610573/ | |
| 15:50:55 | mriedem | savvas: is this at all related to what you're seeing? https://bugs.launchpad.net/nova/+bug/1497253 | |
| 15:50:55 | openstack | Launchpad bug 1497253 in OpenStack Compute (nova) "different availability zone for nova and cinder when AZ is not explicitly given" [Low,In progress] - Assigned to Roman Podoliaka (rpodolyaka) | |
| 15:52:05 | mriedem | jmlowe: right so 0bbfb73d-f761-4c43-ad72-ab47a559615e is your main cell | |
| 15:52:06 | mriedem | with the nova db | |
| 15:52:30 | jmlowe | correct, I did 'nova-manage cell_v2 map_instances --cell_uuid 0bbfb73d-f761-4c43-ad72-ab47a559615e' | |
| 15:52:43 | mriedem | jmlowe: is there an entry in the nova_api.instance_mappings table with the instance uuid you're looking for and cell 0bbfb73d-f761-4c43-ad72-ab47a559615e ? | |
| 15:53:36 | savvas | mriedem: yes, let me explain why I would like to have nova determine the best node. Basically these would be nodes with local ssd storage, not clustered. So I want instances to have volumes on their own hypervisors but all hypervisors serve the same purpose. Therefore when an instance is to be created, all AZs would qualify and based on available resources nova should determine which AZ to pick. Now I figured if you don't | |
| 15:53:36 | savvas | explicitly define the AZ, it will trigger nova to determine the best candidate, however that action fails as it will try to create the volume in the default "nova" AZ again | |
| 15:55:02 | mriedem | savvas: does this patch fix your issue? https://review.openstack.org/#/c/225119/ | |
| 15:55:02 | savvas | so your cross_az_attach fix works like a charm when explicitly defining an AZ, but not when trying to let nova determine the best AZ/node | |
| 15:55:27 | jmlowe | http://paste.openstack.org/show/610574/ | |
| 15:55:59 | savvas | mriedem: YES! That looks exactly like it | |
| 15:56:33 | savvas | mriedem: thanks for the help! | |
| 15:56:38 | mriedem | savvas: so where does the failure actually happen? on the cinder side? | |
| 15:57:10 | jmlowe | yes, the particular instance I'm after is present in the instance mappings mapped to cell0 | |
| 15:57:34 | jmlowe | is it a problem that there is no transport url for cell0 in cell_mappings? | |
| 15:57:58 | savvas | mriedem: well cinder works just fine, it does what it is beign told. Nova's just not telling cinder which AZ to use if you don't explicitly define the AZ for instance creating | |
| 15:58:38 | mriedem | jmlowe: the problem is the instance is mapped to cell0 | |
| 15:58:49 | jmlowe | yes | |
| 15:59:02 | openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for resource classes https://review.openstack.org/467596 | |
| 15:59:26 | savvas | mriedem: https://review.openstack.org/#/c/225119/ this right here what you just sent me is exactly what I am seeing | |
| 15:59:55 | mriedem | jmlowe: so i think what probably happened was, the instances got mapped to cell0 by mistake, somehow, i'm not sure how - but now when you run map_instances with the correct cell 0bbfb73d-f761-4c43-ad72-ab47a559615e it does nothing b/c they are already mapped, | |
| 16:00:02 | jmlowe | should the database connection for cell0 and the legacy nova be different? | |
| 16:00:05 | mriedem | jmlowe: so i think you basically need to purge your instance_mappings table | |
| 16:00:17 | mriedem | jmlowe: they should be separate databases yes | |
| 16:00:27 | mriedem | you will have 3 separate databases: nova_api, nova_cell0 and nova | |
| 16:00:36 | jmlowe | ok, just checking | |
| 16:00:53 | mriedem | so i think you need to purge the instance_mappings table, and then re-map with the proper cell uuid 0bbfb73d-f761-4c43-ad72-ab47a559615e | |
| 16:01:05 | jmlowe | no harm in ' delete from instance_mappings;' ? | |
| 16:01:07 | mriedem | purging that table shouldn't be a problem since it's all wrong anyway | |
| 16:01:19 | mriedem | dansmith: right? ^ | |
| 16:01:30 | dansmith | yeah | |
| 16:01:50 | jmlowe | rerunning mapping | |
| 16:01:54 | mriedem | savvas: ok can you reply to the question i put in the commit messagE? | |
| 16:02:32 | mriedem | savvas: i.e. the volume gets created successfully but in the default 'nova' AZ, | |
| 16:02:44 | mriedem | which is different from the explicit non-default AZ that the instance is in, | |
| 16:02:47 | mriedem | and that's the bug, right? | |
| 16:02:51 | mriedem | but the boot request doesn't actually fail? | |
| 16:03:31 | savvas | no the boot request is accepted, it then tries to create a volume in AZ nova (default), when it fails it removes the attempted volume creation and leaves the new instance in error state | |
| 16:04:05 | mriedem | savvas: so is nova actually passing availability_zone='nova' to cinder when creating the volume? | |
| 16:04:08 | mriedem | or None? | |
| 16:04:33 | savvas | I believe it selects nova for it not being defined, but I can't tell for sure | |
| 16:04:37 | mriedem | edleafe: in the absence of bauzas, can a compute host live in more than one AZ? | |
| 16:04:47 | savvas | I've been using openstack for a week so I am still finding my way ;) | |
| 16:04:53 | mriedem | savvas: that's kind of important, because if nova is passing az=None then it's a different issue | |
| 16:04:58 | mriedem | savvas: you're a pro now | |
| 16:05:00 | mriedem | congratulations | |
| 16:05:01 | edleafe | mriedem: nope | |
| 16:05:25 | mriedem | ok, but a compute host can be in more than one host aggregate | |
| 16:05:29 | mriedem | that's where i get confused | |
| 16:05:30 | edleafe | sure | |
| 16:05:40 | edleafe | yes, it's very terrible | |
| 16:05:55 | mriedem | https://docs.openstack.org/developer/nova/aggregates.html | |
| 16:05:59 | mriedem | "one host can be in multiple aggregates, but it can only be in one availability zone" | |
| 16:06:03 | mriedem | i guess i should have read the manual | |
| 16:06:28 | savvas | mriedem: | OS-EXT-AZ:availability_zone | | |
| 16:06:33 | savvas | it leaves the zone empty when not defined | |
| 16:06:40 | savvas | so it doesn't explicitly pass on nova | |
| 16:07:19 | mriedem | savvas: ok so i don't see why the volume creation fails then | |
| 16:07:30 | mriedem | if nova isn't passing an AZ, cinder will create the volume in it's default AZ, which is 'nova' | |
| 16:08:01 | mriedem | i can see why / how the instance and volume can get into different AZs here, which is a bug | |
| 16:08:05 | savvas | mriedem: exactly, I don't have any cinder-volumes available in nova though | |
| 16:08:36 | savvas | right now I have 3 hypervisors setup in my test environment, all 3 in separate aggregates with separate storage zones | |
| 16:08:45 | savvas | in my case it looks like this: | |
| 16:08:55 | savvas | host-aggregate hv01-zone, has 1 member, hv01 | |
| 16:09:05 | savvas | cinder on hv01 has storage/default zone set to hv01-zone | |
| 16:09:16 | savvas | and I follow the same structure for hv02 and hv03 | |
| 16:09:32 | savvas | but the default nova group has no usable cinder volumes available in my setup | |
| 16:09:45 | savvas | so when the AZ is not defined, it fails | |
| 16:10:12 | mriedem | are you setting storage_availability_zone in cinder.conf on your 3 storage nodes? | |
| 16:10:54 | mriedem | i don't really know how cinder handles the AZs on their side | |
| 16:11:07 | mriedem | so i don't know what the actual error looks like when creating the volume on the cinder side, | |
| 16:11:13 | mriedem | if you have those details you could put them in that bug | |
| 16:12:00 | savvas | mriedem: yes, they are all on all 3 nodes | |
| 16:12:01 | jmlowe | ok, looking much better | |
| 16:12:18 | mriedem | jmlowe: sweet | |
| 16:12:32 | mriedem | rpodolyaka: can you explain where we fail with https://review.openstack.org/#/c/366724/ during boot? | |
| 16:12:35 | mriedem | is it during attach? | |
| 16:13:04 | mriedem | savvas: the volume gets created successfully though right? but then something else fails? | |
| 16:15:36 | savvas | mriedem: when the AZ upon instance creation is defined, yes. When the AZ upon instance creation is not defined, then no. When I create a volume manually via the CLI in the nova zone (so create a volume without defining AZ), it creates just fine | |