| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-15 | |||
| 19:14:35 | cfriesen | mriedem: back to the nova-compute service delete issue, when deleting a service via the API, nova.db.sqlalchemy.api.service_destroy() will soft-delete both the service and the service and the compute_node entry, but placement is still around. Then we create the compute node again and get new service and compute_node entries with a different uuid but the same hostname. | |
| 19:14:36 | sean-k-mooney | cpu pinning does not make sense in a phyical server context | |
| 19:14:39 | dansmith | it wouldn't matter these days anyway, ironic nodes report a CUSTOM_IRONIC_FOO resource and no cpu/mem | |
| 19:15:36 | mgoddard_ | sean-k-mooney: ok, you're right about pinning. hyperthreading could (but doesn't) work though | |
| 19:15:53 | dansmith | cfriesen: mriedem because we store service_id in the compute node, so we won't find the existing one when we re-create the service | |
| 19:16:15 | dansmith | cfriesen: mriedem I bet that api was never updated when we added the node concept.. it probably needs to delete the node(s) as well when it does that | |
| 19:16:19 | sean-k-mooney | mgoddard_: hyperthreading is a bios config option and should work but nova does not allow you to enable hyperthreading as a flavor extra spec | |
| 19:16:21 | dansmith | (and thus placement( | |
| 19:16:49 | sean-k-mooney | mgoddard_: the closest you have to that is setting the tread count in the cpu topology extra specs | |
| 19:17:24 | mriedem | i didn't realize that service_destroy also deleted the related compute node record | |
| 19:17:40 | cfriesen | me either...had to go read the code. | |
| 19:18:55 | dansmith | cfriesen: I thought you said it didn't? | |
| 19:18:56 | mriedem | ok so if service delete also deletes the compute node record, then yeah we should also remove the compute node RP in placement | |
| 19:19:09 | mriedem | i said deleting the service didn't delete the compute node record | |
| 19:19:19 | dansmith | oh okay | |
| 19:19:31 | mriedem | there was some bug about a year ago we were both talking about this | |
| 19:19:41 | mriedem | but https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L404 | |
| 19:21:31 | cfriesen | I should be able to open a bug and post a WIP fix later today | |
| 19:21:31 | mriedem | https://github.com/openstack/nova/commit/f0d44c5b09f3f3c84038d40b621bb629a1f8110e#diff-3104166b3e802b86db6c5fa92ad08f43 | |
| 19:21:36 | mriedem | ^ is why i thought this | |
| 19:23:28 | mriedem | see the exchange between myself and alex | |
| 19:24:23 | mriedem | so in this case, they deleted the service (and compute node record) but didn't stop the nova-compute service, | |
| 19:24:32 | mriedem | so it re-created the compute node record | |
| 19:25:27 | mriedem | so in that bug, when they listed compute nodes, the api tries to find the related service which was deleted and the api blows up | |
| 19:25:34 | mriedem | because you don't recreate the service until you restart the service | |
| 19:26:16 | melwitt | gibi: I'm working on the nova/neutron session summary, would be helpful if you could fill in any gaps for the bandwidth-based scheduling agreements/decisions when you get a chance https://etherpad.openstack.org/p/nova-ptg-rocky-neutron-summary | |
| 19:26:25 | mriedem | melwitt: he's on vacation | |
| 19:27:01 | melwitt | urgh right | |
| 19:27:34 | melwitt | got focused only on the april vacation dates | |
| 19:27:37 | mriedem | cfriesen: yeah so to summarize that, deleting the service deletes the compute node record, and the compute node record shouldn't show back up unless the service is running | |
| 19:28:04 | dansmith | ...and we should delete the RP in placement | |
| 19:28:09 | mriedem | what about the host mapping? | |
| 19:28:20 | mriedem | that's also orphaned right? | |
| 19:28:20 | dansmith | we probably should delete it in placement before we delete in our db even | |
| 19:28:26 | dansmith | yeah | |
| 19:28:59 | mriedem | so, | |
| 19:29:13 | mriedem | 1. delete RP in placement (because if we fail to delete the compute node or the service, it will get re-created), | |
| 19:29:16 | mriedem | 2. delete the service/node, | |
| 19:29:20 | mriedem | 3. delete the host mapping | |
| 19:29:21 | mriedem | ? | |
| 19:29:39 | dansmith | well, 1. also because if we fail, they can't do it again if we delete the local entry | |
| 19:30:00 | dansmith | technically, we can find the service record again if we delete the host mapping, | |
| 19:30:13 | dansmith | so probably good to delete that first, then node, then service last of all | |
| 19:30:20 | cfriesen | what's the host mapping used for? | |
| 19:30:21 | dansmith | so placement, mapping, node, service | |
| 19:30:26 | dansmith | cfriesen: finding it by name in a cell | |
| 19:30:27 | mriedem | cfriesen: to map the host | |
| 19:30:29 | cfriesen | ah, right | |
| 19:30:49 | mriedem | node and service are in the same db api call, | |
| 19:30:57 | mriedem | unless we delete the compute node record(s?) directly in the api | |
| 19:31:15 | mriedem | which would be my preference b/c the auto/implicit delete in the db api is kind of gross | |
| 19:32:13 | dansmith | same db call, but node could go first | |
| 19:32:22 | dansmith | but yeah, don't so much care about those two, | |
| 19:32:51 | dansmith | as I'm not sure there are any FKs that would prevent us from deleting one and not the other | |
| 19:33:46 | mriedem | compute_nodes.service_id doesn't have a FK on it | |
| 19:33:55 | dansmith | yeah | |
| 19:34:08 | cfriesen | dansmith: what are the implications of a stale host mapping? | |
| 19:34:18 | dansmith | cfriesen: cruft | |
| 19:34:45 | dansmith | with no easy way to find and clean it up later | |
| 19:35:00 | cfriesen | got it, thanks | |
| 19:35:04 | mriedem | also, | |
| 19:35:12 | mriedem | i think you could re-discover the compute host in another cell, | |
| 19:35:18 | dansmith | true | |
| 19:35:19 | mriedem | and then the old host mapping would be pointed at the wrong cell | |
| 19:35:21 | dansmith | yeah | |
| 19:35:26 | mriedem | i feel like cern mentioned that | |
| 19:36:25 | claudiub|2 | did they express a con...cern? | |
| 19:36:31 | cfriesen | ew | |
| 19:36:31 | mriedem | cfriesen: i think this block is where you add the new deletes https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/services.py#L213 | |
| 19:36:36 | dansmith | claudiub|2: nice :) | |
| 19:36:39 | mriedem | claudiub|2: excellent | |
| 19:36:41 | claudiub|2 | :D | |
| 19:48:15 | cfriesen | mriedem: you think that's better than https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4820 ? | |
| 19:49:23 | mriedem | cfriesen: could happen in compute api code too, doesn't matter a ton to me | |
| 19:49:57 | mriedem | cfriesen: note that this will also need an upgrade impact release note because the api will now require placement to be configured to do this | |
| 19:50:50 | mriedem | i think we should backport this though...so not sure how sticky that gets | |
| 19:50:55 | cfriesen | mriedem: if placement isn't configured, wouldn't it be okay to just continue on since there wouldn't be any records? | |
| 19:51:18 | mriedem | i mean [placement] configured specifically in the nova.conf you're running against nova-api, | |
| 19:51:24 | mriedem | which hasn't technically required placement before this | |
| 19:51:26 | mriedem | as far as i know | |
| 19:51:35 | mriedem | conductor, compute and scheduler all require placement already, but api doesnt | |
| 19:51:40 | cfriesen | ah, got it. I guess that'd be one advantage of doing it in nova-compute | |
| 19:51:56 | mriedem | there is no rpc cast from api to compute to delete itself | |
| 19:52:30 | cfriesen | yeah, got confused by the nova/compute/api.py path for a sec....too many things going on today | |
| 19:59:00 | cdent | jaypipes, efried, edleafe: on consumer generations. does that need an owner? is anything depending on it? I figure I can probably do it if it isn't needed asap (got forbidden first) | |
| 19:59:41 | efried | cdent: It does need an owner afaik. And it is not immediately urgent afaik. Thank you for volunteering. | |
| 19:59:52 | efried | mriedem: ^ | |
| 20:00:08 | jaypipes | cdent: needs an owner, yes. | |
| 20:01:04 | edleafe | I can pair with someone on that | |
| 20:01:36 | cfriesen | mriedem: on the mixed baremetal/hypervisor scheduler thing, I suspect the NUMATopologyFilter is actually okay as long as we don't have spec_obj.numa_topology | |
| 20:03:03 | mriedem | cfriesen: in that case the filter should just filter out an ironic host/node | |
| 20:03:08 | mriedem | but that's what i was trying to figure out | |
| 20:04:07 | mriedem | i.e. if NUMATopologyFilter just doesn't work with an all ironic deployment, then don't enable it, but if you're using a single scheduler for mixed VM/BM, that filter would also kick out requests for numa for ironic hosts in the deployment, | |
| 20:04:14 | mriedem | but i think that should be ok? | |
| 20:04:47 | cfriesen | mriedem: as long as requests for ironic instances don't specify a desired numa topology then I think it should be good | |
| 20:04:54 | mriedem | if we have a libvirt and ironic host as candidates, the ironic would is filtered out and the libvirt one should be ok | |
| 20:04:54 | cdent | edleafe: have you got something queued up after member_of? | |
| 20:05:19 | mriedem | cfriesen: well yeah, baremetal flavors shouldn't have numa things in them | |
| 20:05:35 | edleafe | in OpenStack, just the api-sig stuff: http and version discovery | |
| 20:06:48 | cdent | do you want something immediately queued up? If so you can be primary on consumer generations and I can be your buddy. Otherwise I can start it up in a few weeks | |