Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-11
20:51:29 dansmith yes, let's deploy and depend on etcd instead of people using ansible handlers! :P
20:52:17 melwitt ok, I'll reply with guidance on what to do. thanks y'all
20:52:19 dansmith mriedem: make sure the spec says that etcd is only running when nova-manage is used, and stopped at runtime
20:53:17 eandersson Do you need to run nova-manage placement heal_allocations when upgrading from non-cell to Cell V2?
20:53:27 mriedem eandersson: no
20:53:32 mriedem doesn't have anything to do with cells
20:53:44 eandersson Let me re-phrase, when upgrading from Mitaka to Rocky
20:53:56 mriedem it was built for people migrating from caching scheduler - which didn't use placement and thus didn't create allocations - to filter scheduler
20:54:10 mriedem eandersson: in that case...maybe
20:54:12 eandersson because placement is empty for us
20:54:15 mriedem heh
20:54:17 eandersson and it was never in any of the upgrade steps
20:54:31 mriedem because "upgrading from 1983 to now" isn't a supported upgrade
20:54:45 eandersson well we didn't upgrade from mitaka to rocky
20:54:51 eandersson we upgraded from one version to another
20:55:03 eandersson also not a helpful comment
20:55:08 mriedem eandersson: sorry,
20:55:15 mriedem you've been at rocky for a couple of weeks now haven't you/
20:55:16 mriedem ?
20:55:26 eandersson Yes - and things have been working great
20:55:31 eandersson but we started seeing some odd errors
20:55:38 dansmith eandersson: don't mind mriedem he's just cranky today.. it's about nap time
20:55:42 openstackgerrit Merged openstack/nova stable/stein: Add retry_on_deadlock to migration_update DB API https://review.openstack.org/648428
20:55:51 mriedem so you're at rocky but still not seeing anything in placement? like resource providers? or just allocations?
20:56:09 mriedem i'm not sure how you could create a server...
20:56:19 eandersson We see new servers in placement
20:56:26 mriedem but not the old ones
20:56:39 mriedem *you don't see allocations for old servers in placement
20:56:40 eandersson The problem is that now we try to run the above command and it fails
20:57:03 mriedem eandersson: paste me the error
20:57:11 mriedem i should have put a dry run on that command
20:57:18 mriedem it was in the todo list
20:57:20 melwitt nova meeting in a few min
20:57:35 eandersson The first error we saw was this
20:57:36 eandersson > Instance xxx has allocations against this compute host but is not found in the database.
20:58:26 eandersson Next we ran the heal command, but for some reason it thinks the overprovisoned host is too full and fails to complete
20:58:53 mriedem the heal command is trying to PUT /allocations against a given resource provider (compute node) for a given instance,
20:59:07 mriedem and i've you've migrated or created servers on that compute node, placement is going to say "you're already at capacity"
20:59:20 mriedem which is why that PUT is probably failing even though there is already a server on it consuming resources
21:00:00 mriedem eandersson: so in that case, it'd be helpful to look at the inventory reported by placement for that compute node
21:00:16 efried nova meeting now #openstack-meeting
21:00:22 mriedem eandersson: which you can get with this https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-show
21:00:29 mriedem uuid is the uuid of the compute node
21:00:41 mriedem sorry https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list is probably more helpful
21:02:23 ceryx After running heal_allocations we have the same consumer_id (which I believe is the compute instance ID) have allocations created across 6 separate resource providers, each being a different compute node
21:02:48 mriedem ceryx: are you and eandersson in kahoots or just coincidental?
21:02:56 eandersson kahoots :D
21:03:00 mriedem ok
21:03:01 ceryx For RAM, 5 of those allocations are for 32GB but one is for 64GB which seems also strange (and yes sorry, was about to clarify, working with eandersson)
21:03:01 mriedem good
21:04:19 mriedem the consumer id is the instance id yes
21:04:32 mriedem has the instance been migrating around?
21:05:27 efried that 64... if we're leaking allocations and we move an instance away and back...?
21:07:21 ceryx http://paste.openstack.org/show/749212/ this is what we're seeing for one consumer ID. I checked nova.migrations and don't see this instance mentioned.
21:07:30 openstackgerrit Merged openstack/nova stable/stein: Use Selection object to fill request group mapping https://review.openstack.org/647713
21:07:42 mriedem it could be a migration record
21:07:48 mriedem check your migrations table in the cell db for that id
21:08:31 mriedem do you see errors in the logs when you're doing a migration?
21:09:24 mriedem my guess is you've done cold migrations / resize and the source node allocations, which are held by the migration record consumer, are not getting cleaned up for some reason
21:10:22 mriedem the multiple allocations for the same consumer and provider are because of different resource classes, mostly like VCPU and MEMORY_MB
21:10:26 ceryx Ah yep, I was checking uuid and not instance_uuid from migrations. This had previously (about a month ago) had 6 failed migrations, then one confirmed migration.
21:10:26 eandersson We have had migrations disabled for a very long time
21:10:27 mriedem if these are volume-backed servers
21:10:37 eandersson oh so nvm :D
21:12:59 mriedem ceryx: in a nutshell, when you start a cold or live migration, the allocations held by the instance (instances.uuid) on the source node provider are moved to a migration record consumer (consumer_id=migrations.uuid), and the dest node provider allocations are held by the instance during scheduling.
21:13:11 mriedem https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html
21:13:45 mriedem when the cold migration / resize is confirmed the allocations against the source node held by the migration record should be deleted
21:14:04 mriedem and you should just be left with the instance consuming allocations from the target provider
21:14:42 mriedem so in this paste is fec5409b-010e-4316-845c-ef68440d3593 an instance uuid or migration uuid?
21:15:10 mriedem looks like resource_class_id=0 is for VCPU and resource_class_id=1 is for MEMORY_MB
21:15:17 ceryx In this case the computes that we see unexpected allocations on were target hypervisors of an errored migration. So it looks like heal_allocations might be creating allocations for migrations in error state?
21:15:21 mriedem and this is a volume-backed server because there is no DISK_GB allocation
21:15:33 ceryx And yes - this has a cinder backed root disk
21:16:32 mriedem heal_allocations shouldn't even be looking at migrations, just instances, but it's been awhile since i dug into this
21:17:01 openstackgerrit Merged openstack/nova stable/stein: Add functional recreate test for bug 1819963 https://review.openstack.org/648401
21:17:02 openstack bug 1819963 in OpenStack Compute (nova) stein "Reverting a resize does not update the instance.availability_zone value to the source az" [Medium,In progress] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem)
21:21:17 mriedem so heal_allocations should skip the instance if it doesn't have a node set, but if the migration failed the node shouldn't keep changing
21:21:50 mriedem and it also shouldn't PUT new allocations if the instance already has allocations in placement, which it should have if you tried migrating it since you upgraded to rocky
21:21:55 eandersson We don't use azs btw
21:22:10 eandersson (or we only have one rather :p)
21:22:17 mriedem note sure why azs would have anything to do with this
21:23:34 eandersson ah I was looking at the bug above :p
21:23:42 eandersson Didn't realize that it was unrelated
21:24:38 mriedem so for all 6 compute node resource providers in that paste, are there actually 6 matching compute_nodes in the cell db with the same uuid as the resource provider?
21:24:53 mriedem i wonder if you maybe have duplicate compute nodes?
21:24:57 mriedem but with different uuids
21:25:49 mriedem although unless the hostname changed i'm not sure how that could happen since there is a unique constraint on that table for host/hypervisor_hostname
21:26:27 mriedem melwitt: ^ this kind of problem is why switching counting quota usage from placement by default worries me
21:26:31 ceryx Yeah, all migration attempts were post-rocky upgrade. Each one of the resource provides does match a different compute_node that are all still online, they were just past targets for the failed migration.
21:27:10 mriedem ceryx: hmm, so maybe the scheduler created the allocations for that instance and each of those providers, but then the migration failed and we failed to cleanup the allocations created by the scheduler
21:27:13 ceryx In the allocations DB they were all created when we ran heal_allocations though according to the created_at date, so these aren't old allocations that were failed
21:27:48 ceryx All the allocations for this consumer_id across all 6 resource providers were created at 2019-04-11 20:25:57
21:27:57 mriedem like i said, heal_allocations should only create allocations if the instance doesn't have any and even then should only be against the same node for the instance.host/node values
21:28:07 mriedem do you have the output of the command?
21:28:34 melwitt mriedem: ack. the more examples I see, the more I lean toward making it opt-in by default for now. until we have allocations issues more sorted out
21:29:00 dansmith also why I want to make the image filter (and other prefilters) opt-in
21:29:06 dansmith until we're sure they're 100% right for everyone
21:29:44 mriedem ceryx: this is the method that does the actual work per instance https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1811
21:30:01 mriedem so the instance has to be on a node here https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1838
21:30:13 mriedem then we check to see if it already has allocations https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1848

Earlier   Later