| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-20 | |||
| 12:43:10 | tssurya | but now that you mention "up_calls" we have the same problem for nova-manage db archive_deleted_rows | |
| 12:43:34 | Kevin_Zheng | hmm | |
| 12:52:36 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Only update cells' transport_url and database_connection when needed. https://review.openstack.org/603998 | |
| 12:52:54 | openstackgerrit | huanhongda proposed openstack/nova master: Fix instance delete stuck in deleting task_state https://review.openstack.org/598084 | |
| 13:04:16 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/pike: libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/604039 | |
| 13:16:42 | mnaser | there isn't a way to delete a compute node from the db, is there? | |
| 13:16:53 | bauzas | mnaser: by SQL goos ? | |
| 13:16:57 | mnaser | well, without sql | |
| 13:17:07 | bauzas | ah | |
| 13:17:10 | mnaser | nova service-delete deletes the service but the compute node record is still there | |
| 13:17:23 | mnaser | and scheduler complains constantly about 'No compute service record found for host <foo>' | |
| 13:17:36 | bauzas | that's a good point | |
| 13:17:49 | bauzas | compute_node records are automatically populated by the resource tracker | |
| 13:17:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Run ServerMovingTests with nested resources https://review.openstack.org/604084 | |
| 13:17:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use provider tree in virt FakeDriver https://review.openstack.org/604083 | |
| 13:18:10 | gibi | bauzas, tetsuro ^^ | |
| 13:18:13 | mnaser | maybe that error should check if there is an existing service, or something | |
| 13:18:21 | bauzas | mnaser: I thought we were checking this in the RT code | |
| 13:18:41 | mnaser | but this is on the nova-scheduler code, i think RT code sits in nova-compute ? | |
| 13:18:49 | bauzas | gibi: all cool, been frankly diverted by working on the reshape patch | |
| 13:18:51 | bauzas | (and by meetings) | |
| 13:19:06 | bauzas | mnaser: yup, run by nova-compute service | |
| 13:19:22 | bauzas | I'm just saying we populate the entries there | |
| 13:19:41 | mnaser | yeah but in this case it's taking a hypervisor out of service forever | |
| 13:19:51 | mnaser | so turn it off, nova service-delete <foo> and that error doesnt go ahway | |
| 13:19:54 | mnaser | s/ahway/away/ | |
| 13:20:12 | lbragstad | johnthetubaguy gmann nice work on https://review.openstack.org/#/c/547850/ | |
| 13:20:13 | mnaser | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L716-L720 | |
| 13:20:13 | bauzas | yup, I see the problem | |
| 13:20:24 | mnaser | maybe we should clean up there (or delete compute record when a nova-compute binary is deleted) | |
| 13:21:05 | bauzas | mnaser: here is where we create the entry in the DB https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L587 | |
| 13:21:22 | bauzas | mnaser: we had a FK before | |
| 13:21:31 | bauzas | between service and CN | |
| 13:21:41 | mnaser | but i guess with the os-services api addition and making it managable by api | |
| 13:21:43 | mnaser | that might have disappeared | |
| 13:21:53 | mnaser | or some other historic reason you might know more than me :) | |
| 13:22:18 | mnaser | oh you know what | |
| 13:22:32 | mnaser | FK might not do anything anyways, because we just change deleted field | |
| 13:22:42 | mnaser | so for all that sql cares, FK is still satisified | |
| 13:23:06 | bauzas | well, actually that's a good point | |
| 13:23:10 | bauzas | we shadow the services table | |
| 13:23:22 | bauzas | so a CASCADE DELETE isn't like super important | |
| 13:24:31 | bauzas | BBIAB | |
| 13:26:38 | openstack | Launchpad bug 1793533 in OpenStack Compute (nova) "Deleting a service with nova-compute binary doesn't remove compute node" [Undecided,New] | |
| 13:26:38 | mnaser | filed a bug: https://bugs.launchpad.net/nova/+bug/1793533 -- i can take care of pushing the code to fix it if i can get some guidance on the best approach | |
| 13:31:00 | mriedem | mnaser: deleting the compute service should also delete the compute node record in the db | |
| 13:31:08 | mnaser | o | |
| 13:31:10 | mriedem | mnaser: the problem is you need to also have nova-compute for that host stopped | |
| 13:31:16 | mriedem | otherwise the RT will re-create the compute node record | |
| 13:31:33 | mriedem | https://developer.openstack.org/api-ref/compute/#delete-compute-service | |
| 13:32:20 | mnaser | i'm pretty sure it was stopped when i did this (actually, the hypervisor was rebooted with a new hostname so i just deleted the old one) | |
| 13:32:48 | mnaser | let me see the created_at time for the compute node record if there is one | |
| 13:34:14 | mriedem | https://github.com/openstack/nova/blob/d87852ae6a1987b6faa3cb5851f9758b47ef4636/nova/db/sqlalchemy/api.py#L404 | |
| 13:34:32 | mriedem | ^ is the DB API deleting the service and compute node record when you delete the service via DELETE /os-services/{service_id} | |
| 13:34:38 | mnaser | mriedem: created_at for compute node record is '2018-04-13 00:21:53', created_at for service (that was deleted) is '2018-04-13 00:21:53' | |
| 13:35:51 | mnaser | oo | |
| 13:35:58 | mnaser | deleted=16 on compute node | |
| 13:35:58 | mriedem | do the host values align? | |
| 13:36:04 | mnaser | so it was soft deleted | |
| 13:36:50 | mnaser | yes, they do, it was actually soft deleted | |
| 13:37:00 | mnaser | could it be that loop doesn't grab non-soft deleted compute nodes? | |
| 13:37:18 | mnaser | that would be the only logical thing happening | |
| 13:37:47 | mnaser | OR | |
| 13:37:51 | mnaser | it looks like it uses _get_computes_for_cells ? | |
| 13:37:51 | mriedem | by default that query is only going to look up non-soft-deleted cmpute node records that match the query | |
| 13:38:00 | mnaser | so maybe it is looking in nova_api compute node lists? | |
| 13:38:06 | mriedem | which loop are you talking about? | |
| 13:38:27 | mnaser | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L713 | |
| 13:38:32 | mnaser | whatever comes in here is including a deleted item | |
| 13:38:58 | mnaser | i just confirmed it by adding LOG.debug(computes) | |
| 13:39:03 | mnaser | and i see the deleted one that it complains about | |
| 13:39:28 | mriedem | is there a resource provider in placement for the deleted compute node still? | |
| 13:39:37 | mriedem | openstack resource provider show <compute node uuid> | |
| 13:40:18 | mnaser | yes | |
| 13:40:35 | mriedem | hmm, you're on rocky right? is nova-api configured to talk to placement? | |
| 13:40:42 | mnaser | this is a queens deployment | |
| 13:40:46 | mnaser | nova-api talking to placement | |
| 13:40:52 | mriedem | ok, well we backported a fix for that to queens also, | |
| 13:40:55 | mnaser | and also, shouldn't we need to delete it from nova_api mappings? maybe thats where its getting it from? | |
| 13:41:02 | mriedem | but only if nova-api is configured to talk to placement, | |
| 13:41:12 | mriedem | then the api will also delete the RP record in placement when the compute node is deleted | |
| 13:41:20 | mriedem | host mappings? | |
| 13:41:24 | mriedem | that's also part of that fix | |
| 13:41:25 | tssurya | HMs are also deleted | |
| 13:41:28 | mriedem | lemme dig it up | |
| 13:41:52 | mnaser | nova_api host_mappings still contains that compute node | |
| 13:42:09 | mnaser | now this is early queens so that could very well be the cause, be good to see if i have that fix or not | |
| 13:42:14 | tssurya | https://review.openstack.org/#/c/563698/ | |
| 13:42:22 | mnaser | aaaa | |
| 13:42:25 | mriedem | yup that's the one | |
| 13:42:37 | mnaser | let me double check that doesnt exist | |
| 13:43:43 | mnaser | nope | |
| 13:43:46 | mnaser | that's probably it | |
| 13:43:58 | mnaser | ill mark my bug as duplicate of that one | |
| 13:44:04 | mriedem | yeah so the RP is still in placement, and it gave it back to the scheduler, and the scheduler somehow looked it up | |
| 13:44:23 | mriedem | i'm curious as to why the scheduler is looking up deleted compute nodes though | |
| 13:44:40 | mnaser | yeah, that doesn't seem right i guess | |
| 13:44:56 | dansmith | um, guys, obvi | |
| 13:45:00 | dansmith | it's scheduling a deleted instances | |
| 13:45:05 | mnaser | objects.ComputeNodeList.get_all(cctxt) | |