| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-20 | |||
| 14:57:24 | belmoreira | Hi, I need some help... | |
| 14:57:41 | belmoreira | nova-compute (dedicated to ironic) market all ironic nodes (compute_nodes table) and as consequence removed all the resource_providers/inventories/allocations for those nodes | |
| 14:59:09 | belmoreira | restarting nova-compute and removing the soft delete of the compute_nodes, the resource provider/inventory is created again | |
| 14:59:31 | belmoreira | but not the allocations... I remember that this was removed from nova some time ago | |
| 14:59:54 | belmoreira | any suggestion how I can recreate thousands of allocations? | |
| 15:00:35 | dansmith | belmoreira: I assume heal instance allocations, just a sec | |
| 15:00:51 | belmoreira | I remember that mnaser and bauzas did some kind of tool | |
| 15:01:10 | belmoreira | dansmith I using stein | |
| 15:01:15 | dansmith | in nova-manage, I think there's an instance heal | |
| 15:01:18 | dansmith | oh | |
| 15:02:17 | bauzas | placement-audit you mean ? | |
| 15:02:50 | belmoreira | placement-audit doesn't recreate allocations if they are missing? | |
| 15:03:05 | dansmith | heal_allocations is the subcmd | |
| 15:03:25 | dansmith | belmoreira: that was in stein | |
| 15:03:34 | mnaser | yeah heal allocations is the thing you need to add missing entries, placement audit is what you need to remove extra entries | |
| 15:04:54 | belmoreira | thanks all, let me have a look | |
| 15:28:23 | mriedem | i think https://review.opendev.org/#/c/678100/ is still relevant to ^ for the compute side delete of allocations there | |
| 15:28:26 | mriedem | but it needs love | |
| 15:33:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/pike: Avoid circular reference during serialization https://review.opendev.org/714148 | |
| 15:39:19 | belmoreira | thanks dansmith, mnaser, bauzas. 4305 allocations recreated! In Stein it goes through all deployment. Needed to patch it to do it per cell (this will be a great addition to master). | |
| 15:39:28 | belmoreira | For some reason nova deleted all the ironic resource providers/inventories/allocations. | |
| 15:39:44 | belmoreira | I'm investigating what trigger this... no idea for now... I will let you know when I have more info | |
| 15:40:30 | dansmith | I thought it does do it per cell in master, but I'd have to go back and look | |
| 15:42:56 | belmoreira | it iterates per cell but is not possible to select the cell | |
| 15:43:44 | dansmith | ah okay | |
| 15:51:38 | mnaser | belmoreira: changed hostname or compute node by any chance ? | |
| 15:55:32 | belmoreira | mnaser maybe... I'm configuring "conductor_groups" for ironic. To avoid this precise issue I configured only one conductor group to keep the same nova-compute... and move from there. Still looking... for now I don't understand what went wrong | |
| 15:55:58 | mnaser | belmoreira: maybe check the compute nodes table | |
| 15:56:42 | belmoreira | mnaser the compute_nodes table had all the resources deleted | |
| 15:59:12 | belmoreira | mnaser to recover I made the entries available again (deleted=0, deleted_at=NULL) | |
| 15:59:36 | belmoreira | new entries are not created because the uuid conflict | |
| 15:59:49 | mnaser | belmoreira: pretty sure you might look at the compute logs and see that it was deleting them | |
| 16:00:07 | mnaser | also I think I ran into that and mriedem fixed it at the time | |
| 16:01:14 | belmoreira | yeah, I need to go deep into the logs... for now I'm still adding the nodes into the right placement_aggregate for them to be available | |
| 16:36:19 | mriedem | see that link above - if you're doing rebalancing of nodes when the old compute sees the nodes are no longer reported there it deletes them along with the providers and their allocations | |
| 16:36:51 | belmoreira | thanks mriedem | |
| 16:36:59 | mriedem | https://review.opendev.org/#/c/678100/3/nova/compute/manager.py@9208 | |
| 16:37:09 | mriedem | that cascading delete thing was added back when the computes still self-healed the allocations in the RT | |
| 16:37:15 | mriedem | but that self-heal was removed a few releases ago | |
| 16:37:21 | mriedem | so now we just delete your stuff on re-balance | |
| 16:37:26 | mriedem | you're welcome | |
| 16:38:54 | mriedem | belmoreira: i think this is the todo you're looking for for heal_allocations yes? https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L2119 | |
| 16:39:36 | mriedem | that's a very easy change https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L2173 | |
| 16:41:33 | belmoreira | mriedem yes, it's easy to do... I did't under stress and just ran it :) | |
| 16:42:05 | belmoreira | we have a new member in the team, this is a good low hanging fruit for him | |
| 17:16:01 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-flavor-access https://review.opendev.org/713559 | |
| 18:11:03 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 18:11:05 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 18:32:17 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 18:32:18 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 18:34:56 | openstackgerrit | melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116 | |
| 18:34:57 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 18:49:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: func tests: move _run_periodics() into own mixin https://review.opendev.org/705545 | |
| 18:49:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: tests: work around malformed serial XML https://review.opendev.org/705546 | |
| 18:49:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 18:49:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146 https://review.opendev.org/687404 | |
| 18:49:27 | openstack | bug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms) | |
| 19:03:40 | artom | dansmith, ^^ if you have the time and energy this Friday afternoon. Next week is fine too :) I'm going to go interact with my spawns for a bit | |
| 19:30:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: func tests: move _run_periodics() into own mixin https://review.opendev.org/705545 | |
| 19:30:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: tests: work around malformed serial XML https://review.opendev.org/705546 | |
| 19:30:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 19:30:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146 https://review.opendev.org/687404 | |
| 19:30:28 | openstack | bug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms) | |
| 19:50:03 | artom | dansmith, so, you'd rather the class inheriting from the Mixin sets self.computes? | |
| 19:50:34 | artom | My thinking was, the mixin is a batteries-included thing to help you run_periodics() | |
| 19:50:56 | dansmith | artom: I'd rather not have the mixin at all, but yes, I'd rather the classes set them and let that blow up if they don't instead effectively delegating setUp() to a later time, and in a mixin | |
| 19:51:01 | dansmith | I understand | |
| 19:52:17 | artom | dansmith, you mean move it directly under _IntegratedTestBase? Not everything can inherit from that... | |
| 19:52:41 | artom | Can it? | |
| 19:53:43 | dansmith | I dunno, you tell me.. I didn't go look at the whole inheritance tree. | |
| 19:54:33 | artom | Yeah, looking now | |
| 19:54:36 | dansmith | the massive mess of where and how all those tests get their functions make it super confusion to figure out where stuff is defined, which is why I don't like this | |
| 19:55:10 | dansmith | the number of places we have duplicated helper functions for checking server state, waiting for server state to change, server state to become a thing, etc is just maddening | |
| 19:57:19 | artom | dansmith, I know - stephenfin's done some work to improve that | |
| 19:57:54 | artom | So, I checked and the two regression tests that I changed would be a a PITA to make inherit from the _IntegratedTestBase | |
| 19:58:02 | artom | They have their own lengthy setup | |
| 19:59:16 | artom | The mixin is probably cleaner, that at point. But OK, let's make the child setup self.computes then | |
| 19:59:20 | artom | *at that point | |
| 19:59:42 | dansmith | start_service is a method on the *base* TestCase right? | |
| 19:59:48 | dansmith | so why not just put all of it there? | |
| 19:59:51 | artom | The very base, yeah | |
| 19:59:59 | dansmith | if that's where they meet in the inheritance tree, that's ...where they meet | |
| 20:00:30 | artom | I suppose | |
| 20:00:47 | artom | Isn't it weird to have func test-specific stuff in the base class? | |
| 20:01:03 | dansmith | like what's already there you mean? | |
| 20:01:13 | artom | Heh, yeah :/ | |
| 20:01:33 | dansmith | you could also put another layer in between the two.. _IntegratedTestBaseNoSeriouslyThisIsTheFunctionalBaseForReals | |
| 20:01:41 | artom | lol | |
| 21:02:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: tests: work around malformed serial XML https://review.opendev.org/705546 | |
| 21:02:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 21:02:32 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146 https://review.opendev.org/687404 | |
| 21:02:34 | openstack | bug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms) | |
| 21:02:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: func tests: move _run_periodics() into base class https://review.opendev.org/714228 | |
| 22:09:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 22:09:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146 https://review.opendev.org/687404 | |
| 22:09:09 | openstack | bug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms) | |
| #openstack-nova - 2020-03-21 | |||
| 04:58:28 | openstackgerrit | Merged openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470 | |
| 06:08:12 | openstackgerrit | Brin Zhang proposed openstack/nova master: Make serialize_args handle exception messages safely https://review.opendev.org/712697 | |
| 06:08:35 | openstackgerrit | Brin Zhang proposed openstack/nova master: Store instance action event exc_val fault details https://review.opendev.org/694428 | |
| 06:09:10 | openstackgerrit | Brin Zhang proposed openstack/nova master: Expose instance action event details out of the API https://review.opendev.org/694430 | |