| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-27 | |||
| 13:09:43 | jaypipes | jinx | |
| 13:10:41 | mriedem | there could be a slide of all the things jay wants to do but i won't let happen because i'm a bad guy, however, | |
| 13:10:50 | mriedem | something like that is bound to be taken seroiusly and blow up in our faces | |
| 13:10:55 | mriedem | *seriously | |
| 13:11:09 | mriedem | or offend someone that actually wants said things | |
| 13:11:18 | mdbooth | dansmith: You up and about, yet? I'm currently working on https://bugs.launchpad.net/nova/+bug/1686703 and I keep bouncing off dead ends like a Roomba. | |
| 13:11:18 | openstack | Launchpad bug 1686703 in OpenStack Compute (nova) "Error in finish_migration results in image deletion on source with no copy" [Undecided,New] | |
| 13:11:18 | jaypipes | RiteshPaiboina: hi! | |
| 13:11:32 | jaypipes | RiteshPaiboina: I'm actually in EDT timezone, not PDT :) | |
| 13:11:42 | mdbooth | dansmith: Looking for any relevant insight into migration records you may have. | |
| 13:15:53 | jaypipes | RiteshPaiboina: basically, you will want to add a unit test to here: https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/test_imagebackend.py#L1257 that ensures that on calling the RbD imagebackend's clone() method, that the code ends up calling the Rbd image driver's flatten() method. | |
| 13:31:58 | dansmith | mdbooth: yeah, I dunno if it's reliable enough to block cache manager expiration, but I don't know what else to do | |
| 13:33:00 | mdbooth | dansmith: At least I feel better knowing it's not just me :) | |
| 13:34:01 | zioproto | I have a user that has a instance booting from volume. And there are two cinder volumes attached. He opened a ticket that he cannot start this vm anymore. I confirm he cant.. this is my nova-compute log file. WTF ! http://paste.openstack.org/show/608202/ | |
| 13:34:12 | zioproto | What is this Duplicate ID 'drive-ide0-0-0' for drive ?? | |
| 13:38:22 | mriedem | zioproto: might be helpful to get the block_device_mappings table rows for that instance | |
| 13:41:07 | mriedem | zioproto: does the domain xml get logged during _create_domain? | |
| 13:42:50 | mriedem | _get_guest_xml should log it actually | |
| 13:44:12 | mriedem | so i'd check the dumped guest xml against the bdms in the db, specifically the connection_info values in the bdms entries | |
| 13:44:18 | mriedem | see if there is some duplicate id in there | |
| 13:48:37 | edleafe | Any cores want to take a look at https://review.openstack.org/#/c/453862/ and push it through? Already has a +2 from jaypipes | |
| 13:51:18 | dansmith | mriedem: so some ironic people just brought up an issue with me around host mappings and aggregates | |
| 13:51:55 | dansmith | mriedem: if you recall, as of like newton I think, the ironic driver uses this hash ring to choose which ironic nodes get owned by each compute service | |
| 13:52:35 | dansmith | mriedem: if you have more compute services than nodes in ironic (like during setup), then you have some compute services that don't have any ComputeNode records | |
| 13:52:37 | mriedem | i recall that but never was involved in it | |
| 13:53:19 | dansmith | so the problem is, if you're bootstrapping an environment, you can't add those nova-compute services to an aggregate because they don't have ComputeNode records and thus don't have HostMappings | |
| 13:53:32 | dansmith | you can't add the ones that don't "own" any ironic nodes yet I mean | |
| 13:54:09 | dansmith | since you could before we cell-ified the aggregate api, they see it as a regression, which I kindof understand, | |
| 13:54:13 | mriedem | i guess i don't know about the ironic setup issue where we have more compute services than nodes, unless i'm missing something obvious | |
| 13:54:46 | mriedem | or is this, | |
| 13:54:49 | dansmith | mriedem: during bootstrapping.. like you're setting up nova with ironic, but you haven't enrolled any nodes | |
| 13:54:50 | mriedem | you start nova-compute, | |
| 13:55:06 | mriedem | that asks ironic for available resources (nodes) which don't exist yet in ironic, | |
| 13:55:12 | mriedem | so that compute service record is mapped to 0 nodes? | |
| 13:55:17 | mriedem | ok | |
| 13:55:26 | mriedem | unlike libvirt where we have 1x1 | |
| 13:55:26 | dansmith | yep | |
| 13:55:28 | dansmith | right | |
| 13:56:05 | dansmith | so before we cellified that api, we just looked up the service in the one database, and if it was there, then we add it to the aggregate and move on | |
| 13:56:18 | dansmith | now, we also verify that there is a hostmapping, | |
| 13:56:33 | mriedem | i'll have to look up the aggregates api code | |
| 13:56:34 | dansmith | but we use that hostmapping to know which cell db to look for the service | |
| 13:56:47 | dansmith | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4734-L4760 | |
| 13:56:57 | dansmith | damn, well, a few lines off | |
| 13:57:10 | dansmith | add_host_to_aggregate() is what you want in there | |
| 13:57:29 | dansmith | anyway, there are a couple of things we can do: | |
| 13:57:43 | dansmith | 1. Just say you can't do this, mock ironic for being weird, and move on | |
| 13:57:52 | mriedem | so this is https://developer.openstack.org/api-ref/compute/#add-host | |
| 13:58:12 | dansmith | 2. If we don't find a hostmapping, we scan each cell for a matching service as a fallback, and if we find it, proceed | |
| 13:58:33 | dansmith | it would be a single get-by-hostname per cell, so not *too* bad | |
| 13:58:35 | dansmith | yeah that | |
| 13:59:32 | mriedem | ok i see this guy https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4748 | |
| 13:59:32 | mriedem | yeah | |
| 13:59:39 | mriedem | ugh | |
| 13:59:55 | mriedem | so this is kind of like the same issue we have that prompted the need for using uuids in the services and hypervisors APIs | |
| 14:00:07 | dansmith | it's fairly dumb because this only affects ironic and only in that kindof silly case, but.. | |
| 14:00:12 | mriedem | and you had to workaround in the HostAPI code for now | |
| 14:00:23 | mriedem | so i think we do the same thing you already did in the HostAPI to find a service/host | |
| 14:00:33 | dansmith | why is it the same? | |
| 14:00:40 | mriedem | iterate the cells until we find a match, if we don't find one, 404, if we find multiple, 409 | |
| 14:00:43 | dansmith | just that we don't know the cell by the uuid you mean I guess | |
| 14:01:05 | dansmith | well, I don't know that we need to handle the multiple case necessarily | |
| 14:01:30 | dansmith | if you have multiple compute services with the same hostname you have already messed up your mappings anyway | |
| 14:01:40 | dansmith | we did the 409 because we were deleting something, | |
| 14:01:46 | mriedem | i'm talking about this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4413 | |
| 14:01:48 | dansmith | but I think in this case it's okay to take the first and stop looping | |
| 14:01:58 | dansmith | yeah I know | |
| 14:02:07 | zioproto | mriedem: I am looking at the database, select connection_info from block_device_mapping where instance_uuid= ... | |
| 14:02:16 | zioproto | mriedem: what exactly should I check ? | |
| 14:02:41 | zioproto | everything looks exactly the same | |
| 14:02:44 | zioproto | except the volume uuid | |
| 14:05:03 | mriedem | dansmith: not sure what you mean by "but I think in this case it's okay to take the first and stop looping" | |
| 14:05:26 | mriedem | your #2 is start looping the cells looking for the service by hostname right? | |
| 14:05:33 | mriedem | if we find a match we use that cell context | |
| 14:05:39 | dansmith | mriedem: you *will* have duplicate service ids by no fault of your own because it's autoincrement.. you only have duplicate service hostnames if you're doing it wrong | |
| 14:05:50 | dansmith | mriedem: so I think it's okay to not do the full scan dupe check if you're looking up by hostname | |
| 14:06:05 | mriedem | this came up in the spec and i think johnthetubaguy said duplicate hostnames is a thing | |
| 14:06:09 | zioproto | mriedem: in the xml file both disks have this same line | |
| 14:06:12 | zioproto | <address type='drive' controller='0' bus='0' target='0' unit='0'/> | |
| 14:06:27 | mriedem | zioproto: that seems like the cause then | |
| 14:06:34 | dansmith | mriedem: can't be, otherwise our HostMapping doesn't work :) | |
| 14:06:39 | zioproto | but this xml file is generated by nova, correct ? | |
| 14:06:58 | mriedem | zioproto: yes, would have to lookup the code | |
| 14:07:04 | mriedem | dansmith: let me check the spec | |
| 14:07:10 | mriedem | because one of these APIs takes hostname and i remember talking about htis | |
| 14:07:11 | mriedem | *this | |
| 14:08:07 | dansmith | mriedem: we do all our rpc routing by hostname, have a unique constraint there, etc | |
| 14:08:34 | dansmith | mriedem: if we had dupes, we'd be sending rpc messages to the wrong hosts all over the place | |
| 14:08:43 | mriedem | dansmith: but in a cells v1 world your unique constraint is only scoped to the cell | |
| 14:08:46 | mriedem | right? | |
| 14:08:58 | zioproto | mriedem: most likely this machine booted in Liberty and not in Mitaka it cannot boot anymore :( | |
| 14:09:03 | dansmith | mriedem: it's all mirrored up to the top level db, where there is a constraint | |
| 14:09:04 | zioproto | mriedem: should I open a bug | |
| 14:09:06 | zioproto | ? | |
| 14:09:22 | mriedem | zioproto: i'm not sure what liberty vs mitaka has to do with this | |
| 14:09:23 | dansmith | mriedem: i.e. all the services are in the cell dbs and in the main unified one at the top | |
| 14:09:37 | mriedem | dansmith: ok, well, i'm just telling you what i was told | |
| 14:09:47 | mriedem | you'll have to get the details from johnthetubaguy's horses mouth | |
| 14:09:49 | dansmith | mriedem: anyway, finish your convo with zioproto, we can argue about this detail in the review | |
| 14:12:04 | mriedem | dansmith: ok i think i remember now | |