Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-11
21:10:27 mriedem if these are volume-backed servers
21:10:37 eandersson oh so nvm :D
21:12:59 mriedem ceryx: in a nutshell, when you start a cold or live migration, the allocations held by the instance (instances.uuid) on the source node provider are moved to a migration record consumer (consumer_id=migrations.uuid), and the dest node provider allocations are held by the instance during scheduling.
21:13:11 mriedem https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html
21:13:45 mriedem when the cold migration / resize is confirmed the allocations against the source node held by the migration record should be deleted
21:14:04 mriedem and you should just be left with the instance consuming allocations from the target provider
21:14:42 mriedem so in this paste is fec5409b-010e-4316-845c-ef68440d3593 an instance uuid or migration uuid?
21:15:10 mriedem looks like resource_class_id=0 is for VCPU and resource_class_id=1 is for MEMORY_MB
21:15:17 ceryx In this case the computes that we see unexpected allocations on were target hypervisors of an errored migration. So it looks like heal_allocations might be creating allocations for migrations in error state?
21:15:21 mriedem and this is a volume-backed server because there is no DISK_GB allocation
21:15:33 ceryx And yes - this has a cinder backed root disk
21:16:32 mriedem heal_allocations shouldn't even be looking at migrations, just instances, but it's been awhile since i dug into this
21:17:01 openstackgerrit Merged openstack/nova stable/stein: Add functional recreate test for bug 1819963 https://review.openstack.org/648401
21:17:02 openstack bug 1819963 in OpenStack Compute (nova) stein "Reverting a resize does not update the instance.availability_zone value to the source az" [Medium,In progress] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem)
21:21:17 mriedem so heal_allocations should skip the instance if it doesn't have a node set, but if the migration failed the node shouldn't keep changing
21:21:50 mriedem and it also shouldn't PUT new allocations if the instance already has allocations in placement, which it should have if you tried migrating it since you upgraded to rocky
21:21:55 eandersson We don't use azs btw
21:22:10 eandersson (or we only have one rather :p)
21:22:17 mriedem note sure why azs would have anything to do with this
21:23:34 eandersson ah I was looking at the bug above :p
21:23:42 eandersson Didn't realize that it was unrelated
21:24:38 mriedem so for all 6 compute node resource providers in that paste, are there actually 6 matching compute_nodes in the cell db with the same uuid as the resource provider?
21:24:53 mriedem i wonder if you maybe have duplicate compute nodes?
21:24:57 mriedem but with different uuids
21:25:49 mriedem although unless the hostname changed i'm not sure how that could happen since there is a unique constraint on that table for host/hypervisor_hostname
21:26:27 mriedem melwitt: ^ this kind of problem is why switching counting quota usage from placement by default worries me
21:26:31 ceryx Yeah, all migration attempts were post-rocky upgrade. Each one of the resource provides does match a different compute_node that are all still online, they were just past targets for the failed migration.
21:27:10 mriedem ceryx: hmm, so maybe the scheduler created the allocations for that instance and each of those providers, but then the migration failed and we failed to cleanup the allocations created by the scheduler
21:27:13 ceryx In the allocations DB they were all created when we ran heal_allocations though according to the created_at date, so these aren't old allocations that were failed
21:27:48 ceryx All the allocations for this consumer_id across all 6 resource providers were created at 2019-04-11 20:25:57
21:27:57 mriedem like i said, heal_allocations should only create allocations if the instance doesn't have any and even then should only be against the same node for the instance.host/node values
21:28:07 mriedem do you have the output of the command?
21:28:34 melwitt mriedem: ack. the more examples I see, the more I lean toward making it opt-in by default for now. until we have allocations issues more sorted out
21:29:00 dansmith also why I want to make the image filter (and other prefilters) opt-in
21:29:06 dansmith until we're sure they're 100% right for everyone
21:29:44 mriedem ceryx: this is the method that does the actual work per instance https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1811
21:30:01 mriedem so the instance has to be on a node here https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1838
21:30:13 mriedem then we check to see if it already has allocations https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1848
21:30:52 mriedem if it doesn't, we get the compute node uuid which should match the provider https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1883
21:31:18 mriedem so i'm not sure why that would create allocations for the same instance against 6 different providers
21:31:48 mriedem what would be more likely to me is what i said before - the scheduler created the allocations during the migration, it failed, and then we didn't cleanup somewher
21:32:07 ceryx mriedem: I don't have the full output, was not expecting this many allocations to get created and it ate up all my scrollback.
21:32:47 ceryx What would the process be for cleaning up allocations that exist but shouldn't? Would it be relatively safe to delete the allocations for this one consumer_id, then rerun heal_allocations and confirm what was added back?
21:32:57 mriedem sec
21:33:06 mriedem mnaser has a script i think
21:33:30 mriedem https://bugs.launchpad.net/nova/+bug/1793569
21:33:30 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
21:35:00 mriedem ceryx: so that bug has a link to a script from mnaser which it looks like just dumps commands to run,
21:35:15 mriedem and links to another tool from larsks
21:36:27 mriedem ceryx: "Would it be relatively safe to delete the allocations for this one consumer_id, then rerun heal_allocations and confirm what was added back?" - i think so, but i'd also like it better if you had a --dry-run option when doing that with heal_allocations as well,
21:36:31 mriedem which i could probably wip up real quick
21:38:18 ceryx That would be awesome :D
21:38:48 mriedem ok will crank something out here
21:40:24 imacdonn_ so I seem to have a problem with Stein .. haven't fully diagnosed yet, but if I run online_data_migrations a second time, fill_virtual_interface_list fails with:
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 4050, in _security_group_ensure_default
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage default_group = _security_group_get_by_names(context, ['default'])[0]
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage TypeError: 'NoneType' object has no attribute '__getitem__'
21:40:54 imacdonn_ ring any bells ?
21:42:23 imacdonn_ http://paste.openstack.org/show/73aAO3bB23d62wBjt5nL/
21:42:31 mriedem imacdonn_: not for me
21:43:20 imacdonn_ k. I'll try to dig into it a bit. Tnx.
21:46:21 efried imacdonn_: Looking at that method, that should be impossible
21:47:21 efried oh
21:47:30 efried imacdonn_: Do you have more than one security group named 'default'?
21:47:51 imacdonn_ well, each project has one.....
21:48:55 efried It oughtta be filtering by project ID
21:49:01 imacdonn_ I do see two rows in the security_groups table with name="default" but project_id NULL .. not sure if that should be
21:49:01 efried but that's the only way that method can return None
21:49:10 efried mhm, that'd do it.
21:49:25 efried if you called with project_id NULL somehow
21:49:27 efried in your context
21:49:46 efried imacdonn_: Repeatable?
21:49:58 mriedem ah,
21:50:05 mriedem the online_data_migratoin is using an admin context,
21:50:08 mriedem which doesn't have a project_id
21:50:42 imacdonn_ I'm seeing it in two different installations - one was a fresh install, the other upgraded from Rocky
21:51:26 efried mriedem: but that works fine unless there's multiple default security groups with project_id NULL, yah?
21:51:54 efried imacdonn_: So I can WIP a patch that ought to make the problem go away; or you can manually delete the extra row from your security groups table.
21:52:03 mriedem efried: he said he's running it twice
21:52:05 efried course it'd be nice to know how it got there
21:52:05 melwitt I really hope this isn't related to the user_id instance mapping migration somehow
21:52:06 mriedem and blows up the 2nd time right?
21:52:15 mriedem melwitt: he said it was the vifs one
21:52:22 mriedem "if I run online_data_migrations a second time, fill_virtual_interface_list fails with:"
21:52:32 melwitt yeah, but I added user_id to that, which shouldn't hurt
21:53:19 imacdonn_ yeah, it seemed to work OK the first time, but blows on subsequent attempts ... not sure where these NULL security groups are coming from
21:54:11 imacdonn_ in the frest install case, the projects probably didn't exist when migrations were run the first time
21:55:29 melwitt ok, I added the user_id for the fill virtual interface list instance mapping marker record. so shouldn't be related but just wanted to mention there was a change in stein there
22:07:33 mriedem the vifs migration was also new in stein
22:09:45 melwitt oh. nevermind me
22:10:20 mriedem imacdonn_: w/o looking at the code i think the migration is creating a marker instance record
22:10:29 mriedem which is why it's using an empty admin context with no project,
22:10:34 mriedem it should be using a sentinel for the project_id probably
22:10:59 melwitt it uses a sentinel of all zeros uuid
22:10:59 mriedem should be fairly easy to reproduce that by just modifying an existing test to run the command twice
22:11:45 melwitt https://github.com/openstack/nova/blob/master/nova/objects/virtual_interface.py#L303
22:11:50 mriedem melwitt: but the context doesn't have that
22:11:54 mriedem and thta's what the db api is looking for i think
22:12:33 melwitt oh, other direction
22:12:38 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/db/sqlalchemy/api.py#n4037

Earlier   Later