| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-20 | |||
| 13:22:32 | mnaser | FK might not do anything anyways, because we just change deleted field | |
| 13:22:42 | mnaser | so for all that sql cares, FK is still satisified | |
| 13:23:06 | bauzas | well, actually that's a good point | |
| 13:23:10 | bauzas | we shadow the services table | |
| 13:23:22 | bauzas | so a CASCADE DELETE isn't like super important | |
| 13:24:31 | bauzas | BBIAB | |
| 13:26:38 | mnaser | filed a bug: https://bugs.launchpad.net/nova/+bug/1793533 -- i can take care of pushing the code to fix it if i can get some guidance on the best approach | |
| 13:26:38 | openstack | Launchpad bug 1793533 in OpenStack Compute (nova) "Deleting a service with nova-compute binary doesn't remove compute node" [Undecided,New] | |
| 13:31:00 | mriedem | mnaser: deleting the compute service should also delete the compute node record in the db | |
| 13:31:08 | mnaser | o | |
| 13:31:10 | mriedem | mnaser: the problem is you need to also have nova-compute for that host stopped | |
| 13:31:16 | mriedem | otherwise the RT will re-create the compute node record | |
| 13:31:33 | mriedem | https://developer.openstack.org/api-ref/compute/#delete-compute-service | |
| 13:32:20 | mnaser | i'm pretty sure it was stopped when i did this (actually, the hypervisor was rebooted with a new hostname so i just deleted the old one) | |
| 13:32:48 | mnaser | let me see the created_at time for the compute node record if there is one | |
| 13:34:14 | mriedem | https://github.com/openstack/nova/blob/d87852ae6a1987b6faa3cb5851f9758b47ef4636/nova/db/sqlalchemy/api.py#L404 | |
| 13:34:32 | mriedem | ^ is the DB API deleting the service and compute node record when you delete the service via DELETE /os-services/{service_id} | |
| 13:34:38 | mnaser | mriedem: created_at for compute node record is '2018-04-13 00:21:53', created_at for service (that was deleted) is '2018-04-13 00:21:53' | |
| 13:35:51 | mnaser | oo | |
| 13:35:58 | mriedem | do the host values align? | |
| 13:35:58 | mnaser | deleted=16 on compute node | |
| 13:36:04 | mnaser | so it was soft deleted | |
| 13:36:50 | mnaser | yes, they do, it was actually soft deleted | |
| 13:37:00 | mnaser | could it be that loop doesn't grab non-soft deleted compute nodes? | |
| 13:37:18 | mnaser | that would be the only logical thing happening | |
| 13:37:47 | mnaser | OR | |
| 13:37:51 | mriedem | by default that query is only going to look up non-soft-deleted cmpute node records that match the query | |
| 13:37:51 | mnaser | it looks like it uses _get_computes_for_cells ? | |
| 13:38:00 | mnaser | so maybe it is looking in nova_api compute node lists? | |
| 13:38:06 | mriedem | which loop are you talking about? | |
| 13:38:27 | mnaser | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L713 | |
| 13:38:32 | mnaser | whatever comes in here is including a deleted item | |
| 13:38:58 | mnaser | i just confirmed it by adding LOG.debug(computes) | |
| 13:39:03 | mnaser | and i see the deleted one that it complains about | |
| 13:39:28 | mriedem | is there a resource provider in placement for the deleted compute node still? | |
| 13:39:37 | mriedem | openstack resource provider show <compute node uuid> | |
| 13:40:18 | mnaser | yes | |
| 13:40:35 | mriedem | hmm, you're on rocky right? is nova-api configured to talk to placement? | |
| 13:40:42 | mnaser | this is a queens deployment | |
| 13:40:46 | mnaser | nova-api talking to placement | |
| 13:40:52 | mriedem | ok, well we backported a fix for that to queens also, | |
| 13:40:55 | mnaser | and also, shouldn't we need to delete it from nova_api mappings? maybe thats where its getting it from? | |
| 13:41:02 | mriedem | but only if nova-api is configured to talk to placement, | |
| 13:41:12 | mriedem | then the api will also delete the RP record in placement when the compute node is deleted | |
| 13:41:20 | mriedem | host mappings? | |
| 13:41:24 | mriedem | that's also part of that fix | |
| 13:41:25 | tssurya | HMs are also deleted | |
| 13:41:28 | mriedem | lemme dig it up | |
| 13:41:52 | mnaser | nova_api host_mappings still contains that compute node | |
| 13:42:09 | mnaser | now this is early queens so that could very well be the cause, be good to see if i have that fix or not | |
| 13:42:14 | tssurya | https://review.openstack.org/#/c/563698/ | |
| 13:42:22 | mnaser | aaaa | |
| 13:42:25 | mriedem | yup that's the one | |
| 13:42:37 | mnaser | let me double check that doesnt exist | |
| 13:43:43 | mnaser | nope | |
| 13:43:46 | mnaser | that's probably it | |
| 13:43:58 | mnaser | ill mark my bug as duplicate of that one | |
| 13:44:04 | mriedem | yeah so the RP is still in placement, and it gave it back to the scheduler, and the scheduler somehow looked it up | |
| 13:44:23 | mriedem | i'm curious as to why the scheduler is looking up deleted compute nodes though | |
| 13:44:40 | mnaser | yeah, that doesn't seem right i guess | |
| 13:44:56 | dansmith | um, guys, obvi | |
| 13:45:00 | dansmith | it's scheduling a deleted instances | |
| 13:45:05 | mnaser | objects.ComputeNodeList.get_all(cctxt) | |
| 13:45:44 | mriedem | i see the bug | |
| 13:45:51 | mriedem | 1756179 | |
| 13:45:53 | mriedem | oops | |
| 13:45:59 | mriedem | def _db_compute_node_get_all_by_uuids(context, compute_uuids): | |
| 13:45:59 | mriedem | db_computes = context.session.query(models.ComputeNode).filter( | |
| 13:45:59 | mriedem | models.ComputeNode.uuid.in_(compute_uuids)).all() | |
| 13:45:59 | mriedem | return db_computes | |
| 13:46:08 | mriedem | isn't using model_query so it's not filtering out soft deleted records | |
| 13:46:14 | tssurya | yikes | |
| 13:47:02 | mnaser | do i want to mark my bug as duplicate and create another one for that ^ or use the same one to cover that | |
| 13:47:32 | dansmith | hmm, I wonder why we did that | |
| 13:47:39 | dansmith | instead of writing it in sqla/api.py | |
| 13:47:57 | mriedem | because the new jazz is to write db stuff in the objects | |
| 13:48:08 | mriedem | mnaser: we'll use yours for the query thing | |
| 13:48:12 | mriedem | i'll un-invalidate it | |
| 13:48:31 | dansmith | mriedem: only for api objects | |
| 13:48:54 | mriedem | oh, then idk | |
| 13:48:55 | dansmith | or at least, I've been still putting main stuff in there | |
| 13:49:19 | dansmith | IIRC we agreed to not put api stuff int here, but keep doing it for the main stuff, but | |
| 13:49:48 | mriedem | you'd have to ask papa bear https://review.openstack.org/#/c/418134/ | |
| 13:50:00 | mriedem | dansmith: https://bugs.launchpad.net/nova/+bug/1793533 | |
| 13:50:00 | openstack | Launchpad bug 1793533 in OpenStack Compute (nova) "Scheduler doesn't filter out deleted compute node records based on placement RP UUIDs" [Medium,Triaged] | |
| 13:50:05 | bauzas | I'm back | |
| 13:50:12 | bauzas | PapaOurs is here | |
| 13:50:32 | mriedem | dansmith: so you got this one? | |
| 13:50:33 | bauzas | any regression I made, mmm ? | |
| 13:50:55 | tssurya | Q: placement should not even return deleted RPs right ? the list of compute_uuids coming from placement should be legit ? or its okay because the post-scheduler compute filter is supposed to check for these conditions like it doesn for enabled/disabled ? | |
| 13:51:04 | dansmith | mriedem: but the real problem is not in placement but that we keep finding the compute node in our search yeah? | |
| 13:51:11 | bauzas | tssurya: we have a filter for this | |
| 13:51:13 | mriedem | tssurya: mnaser didn't have https://review.openstack.org/#/c/563698/ | |
| 13:51:20 | mriedem | tssurya: so yes this is a side effect that shouldn't otherwise happen | |
| 13:51:33 | bauzas | we have the ComputeFilter | |
| 13:51:44 | tssurya | (I understand that beore this fix, the RPs were not deleted so it they are returned, but in current situation it should be okay right ?) | |
| 13:51:49 | dansmith | mriedem: I got this one what? you want me to fix it up? | |
| 13:51:49 | tssurya | bauzas: yea that filter | |
| 13:52:02 | mriedem | dansmith: i assumed you were asking for the bug in order to push a change | |
| 13:52:21 | mnaser | if we get a fix it'd be nice to get a dot release for nova, last queens release was end of may | |