| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-08 | |||
| 18:25:36 | mriedem | it's in the instance_extras table | |
| 18:25:42 | dansmith | mgagne: the instance has the flavor stashed with it in the cell db now | |
| 18:25:43 | dansmith | yeah that | |
| 18:25:48 | mriedem | the instance.flavor is stored separately keyed per the the instance | |
| 18:25:50 | mgagne | oh, you get a copy of the flavor right | |
| 18:25:51 | mriedem | in case the flavor is deleted | |
| 18:25:51 | dansmith | so you only need it in the api db for build and list | |
| 18:27:08 | mgagne | so this would mean I could delete content of instance_types in compute cell db just like the original migration tried to do. | |
| 18:27:32 | mriedem | yeah, the api isn't going to read flavors out of the cell dbs | |
| 18:27:43 | mgagne | awesome news then | |
| 18:27:45 | dansmith | probably should to make sure nothing falls back to that when it shouldn't, since it'll go away | |
| 18:28:00 | mgagne | yes, that's my main concern | |
| 18:28:38 | mgagne | so if I skip flavor migration in cell, does it mean entries are now orphan and can be deleted (and not migrated). I guess that's my question. | |
| 18:28:49 | mgagne | I suppose I can test a bit. | |
| 18:28:54 | mriedem | i believe so | |
| 18:30:02 | mriedem | when migrating flavors from child cells to the api db, it handles duplicates (per the flavorid) so should be fine https://github.com/openstack/nova/blob/stable/ocata/nova/objects/flavor.py#L755 | |
| 18:30:15 | dansmith | mgagne: if they're all the same then yeah | |
| 18:30:23 | dansmith | mgagne: which I think they have to be right? | |
| 18:30:23 | mgagne | yes, they are the same. | |
| 18:30:25 | mriedem | as in you could have m1.tiny in 4 cells but with different primary key ids, but the online data migration wouldn't carry over the primary key id, just create a new flavor in the api db | |
| 18:30:49 | mriedem | right, for each flavor in cell db, create in api db, ignore duplicates | |
| 18:30:50 | mgagne | mriedem: afaik, you can't have different primary key id =) | |
| 18:31:02 | dansmith | yeah, | |
| 18:31:07 | mriedem | well, i mean you could have m1.tiny in different cells with the same id=1 | |
| 18:31:11 | dansmith | because it's referenced directly by the instance by id | |
| 18:31:14 | mriedem | but you can only have one m1.tiny in the api db | |
| 18:31:14 | dansmith | mriedem: I don't hink so | |
| 18:31:20 | mgagne | lot of gotcha in cellsv1 that assumes primary key is the same everywhere | |
| 18:31:27 | mriedem | oh, well that i didn't know | |
| 18:31:34 | dansmith | mriedem: in old school nova, instance.instance_type_id refers to flavor.id | |
| 18:31:35 | mriedem | all the more reason to get off cells v1 :) | |
| 18:31:39 | dansmith | so they have to be identical AFAIK | |
| 18:31:40 | mgagne | I got bitten in the past... | |
| 18:31:59 | dansmith | are we comparing scars now? | |
| 18:32:03 | mgagne | hehehe | |
| 18:32:08 | dansmith | :) | |
| 18:32:11 | mriedem | so i guess the 354 data migration blocker isn't so bad for you | |
| 18:32:20 | mriedem | you just needed to run the online data migratoin on each cell db | |
| 18:32:22 | mgagne | I'm sure yours are more numerous than mines | |
| 18:33:00 | mgagne | alright, will go back to my ffu dungeon then :D | |
| 18:34:27 | mriedem | say hi to lyaaaaaaaarwood for me | |
| 18:35:40 | mgagne | is it a friend or some beast I should find and fight in the dungeon? :D | |
| 18:36:18 | mgagne | oh, a ffu expert :D | |
| 18:36:47 | openstackgerrit | Chris Dent proposed openstack/nova stable/queens: Ensure resource class cache when listing usages https://review.openstack.org/573811 | |
| 18:38:18 | mriedem | lyaaaaaaaarwood is a level 12 FFU paladin with a ring of +3 "soft speaking" | |
| 18:46:17 | mriedem | cfriesen: https://review.openstack.org/573813 | |
| 18:49:18 | mriedem | cfriesen: openstack server migration list / abort etc is probably good yeah | |
| 18:49:21 | mriedem | since we have openstack server migrate | |
| 18:49:29 | mriedem | and don't want to get confused with openstack volume migrate | |
| 18:49:37 | mriedem | i leave stuff like that up to the osc ux wizards | |
| 20:12:19 | jroll | is it expected that we should be able to run 'show' or 'delete' via openstackclient on instances in cell0? | |
| 20:12:22 | jroll | using ocata | |
| 20:12:50 | jroll | even list bombs out | |
| 20:13:19 | jroll | http://paste.openstack.org/show/6YrSmjMSo0lIxyFjbPIz/ | |
| 20:13:35 | jroll | looks like it's hitting the wrong database when trying to refresh the instance in _load_flavor() | |
| 20:14:21 | jroll | looks like this bug which expired: https://bugs.launchpad.net/nova/+bug/1749167 | |
| 20:14:23 | openstack | Launchpad bug 1749167 in OpenStack Compute (nova) "nova show can not get an instance information, and this instance can be queried from nova list." [Undecided,Expired] | |
| 20:14:42 | jroll | by the way bowser was trying to ask about the actual scheduling problem, sounds like maybe this is expected? | |
| 20:17:21 | melwitt | jroll: I don't think it's expected | |
| 20:18:22 | melwitt | we had bugs around instance list/show back then, which we fixed. I'm looking through bugs to see if any were this | |
| 20:18:27 | temka | The fact that it's hitting it in flavor... | |
| 20:18:41 | jroll | I didn't immediately see it searching everything for "cell0" | |
| 20:18:43 | melwitt | you have the latest ocata or an earlier one? | |
| 20:18:52 | jroll | should be latest, lemme verify | |
| 20:18:54 | melwitt | yeah, I'm not immediately finding anything either | |
| 20:19:03 | melwitt | it's hard to find these things | |
| 20:19:59 | jroll | well wtf, our latest upstream commit is 125dd1f30fdaf50182256c56808a5199856383c7 | |
| 20:20:04 | jroll | which was february | |
| 20:20:22 | melwitt | I'm not sure if it matters, just wanted to make sure I understand which code you have | |
| 20:20:49 | jroll | it matters that I have a broken assumption :) | |
| 20:22:24 | jroll | similar: https://github.com/openstack/nova/commit/e0c1d461af0701adb94e6974f363e12395ed0162 | |
| 20:22:42 | jroll | but I don't think it's the same | |
| 20:23:11 | temka | jroll, 2e68b2298e94a15d1282c0fb46804b9efa6c8b3a ? | |
| 20:23:30 | temka | Seems old tho | |
| 20:23:40 | jroll | yeah, we would have that | |
| 20:24:19 | melwitt | it's trying to lazy-load the flavor from the database, first thing it does while doing that is the lookup the instance, and that is not found because it's looking in the wrong database | |
| 20:24:29 | jroll | exactly | |
| 20:24:32 | melwitt | this sounds so familiar, just not finding a bug that matches yet | |
| 20:24:41 | temka | melwitt, yep, same here | |
| 20:24:46 | temka | I swear I've seen this | |
| 20:25:40 | jroll | melwitt: in the meantime, we can manually delete it from cell0.instances and nova_api.instance_mappings, trigger a quota reset... will all the other tables get cleaned up on delete? | |
| 20:28:10 | melwitt | I will say that ocata is particularly fraught with problems because the way we were doing cell targeting back then had some fundamental problems which we changed in pike, but was a huge change that backporting would involve backporting a ton of disjointed things in pike, so we abandoned it | |
| 20:28:57 | jroll | mmm | |
| 20:28:59 | melwitt | (assuming that things are okay-enough in ocata. if they're not, then we have to find another way to solve it) | |
| 20:29:53 | melwitt | jroll: yeah, I think that would do it as a manual cleanup. are you seeing this on/during an upgrade or on already-upgraded-and-been-running clusters? | |
| 20:30:24 | jroll | melwitt: already upgraded | |
| 20:30:39 | jroll | we shut down the API for this upgrade | |
| 20:30:42 | melwitt | jroll: okay, so any new instance that fails to schedule gets in this un-listable state | |
| 20:31:02 | jroll | trying to verify that now | |
| 20:31:10 | jroll | this instance is still in BUILD, rather than ERROR | |
| 20:31:19 | jroll | and is the only thing in instance_mappings with cell_id=NULL | |
| 20:31:44 | melwitt | oh, that's interesting. but yet it's in cell0, in BUILD state but not ERROR | |
| 20:32:17 | jroll | yep | |
| 20:32:22 | jroll | and built > 16 hours ago | |
| 20:32:28 | melwitt | cell_id=NULL is a problem. that should be cell0 if it's in cell0 | |
| 20:32:32 | melwitt | hmm | |
| 20:32:40 | jroll | task_state=scheduling | |
| 20:32:51 | jroll | so it's like it got dropped by rabbit or something? idk | |
| 20:33:53 | melwitt | yeah, I dunno. we have also heard from CERN folks of some instance_mappings ending up with cell_id=NULL but I've not understood the conditions under which it happened | |
| 20:35:21 | melwitt | while instances are in the middle of scheduling, they are supposed to be listed/shown using their build_requests in the nova_api database, IIRC | |