Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-11
20:46:38 melwitt dansmith: yeah... I was thinking maybe they're doing that in a central place that is deploying compute nodes in parallel but not necessarily running nova-manage _on_ the compute nodes?
20:47:26 dansmith melwitt: who are we talking about? I thought you said "per compute" so I thought you meant on a compute, but maybe you just mean spawning a bunch of nova-manage commands on one node one for each new host?
20:47:32 melwitt I'll reply on the bug with guidance to run discover_hosts once after deploying compute hosts. and then I think I'll have a docs patch in my future
20:47:33 dansmith that's even more.. crazy
20:47:49 dansmith should be using anisble triggers for that, which AFAIK, would mean a thing runs one
20:47:52 dansmith *once
20:48:08 dansmith like, you don't restart apache for every module you enable, you enable a bunch of modules and run restart at the end :)
20:48:25 melwitt dansmith: I don't actually know the details of how the deployment works, was just thinking it seems unlikely they're putting api db creds on compute hosts and running nova-manage on them
20:48:26 dansmith s/triggers/handlers/
20:48:45 dansmith melwitt: ack, I just thought that was what you originally asserted
20:49:14 openstackgerrit Matt Riedemann proposed openstack/python-novaclient stable/stein: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651925
20:49:15 openstack bug 1746534 in python-novaclient "encoding error when doing console-log" [High,Fix released] https://launchpad.net/bugs/1746534 - Assigned to Thomas Goirand (thomas-goirand)
20:49:19 melwitt no, I didn't mean to make it sound like that
20:49:28 openstackgerrit Matt Riedemann proposed openstack/python-novaclient stable/stein: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651925
20:50:14 dansmith melwitt: so, we could do an external lock on nova-manage which would prevent the specific case of running multiple copies on a single host,
20:50:27 dansmith but it might also be confusing because it won't fix the case where it's multiple on different hosts
20:50:52 dansmith we could do some janky locking in the database to try to catch it, but I'd rather just say "do not do this lest ye burn in nova purgatory"
20:50:55 dansmith or, you know, something
20:51:10 mriedem i smell etcd
20:51:11 mriedem and tooz
20:51:20 melwitt lol.. yes
20:51:23 mriedem that's like 2 etcd references for nova in the last few weeks
20:51:29 dansmith yes, let's deploy and depend on etcd instead of people using ansible handlers! :P
20:52:17 melwitt ok, I'll reply with guidance on what to do. thanks y'all
20:52:19 dansmith mriedem: make sure the spec says that etcd is only running when nova-manage is used, and stopped at runtime
20:53:17 eandersson Do you need to run nova-manage placement heal_allocations when upgrading from non-cell to Cell V2?
20:53:27 mriedem eandersson: no
20:53:32 mriedem doesn't have anything to do with cells
20:53:44 eandersson Let me re-phrase, when upgrading from Mitaka to Rocky
20:53:56 mriedem it was built for people migrating from caching scheduler - which didn't use placement and thus didn't create allocations - to filter scheduler
20:54:10 mriedem eandersson: in that case...maybe
20:54:12 eandersson because placement is empty for us
20:54:15 mriedem heh
20:54:17 eandersson and it was never in any of the upgrade steps
20:54:31 mriedem because "upgrading from 1983 to now" isn't a supported upgrade
20:54:45 eandersson well we didn't upgrade from mitaka to rocky
20:54:51 eandersson we upgraded from one version to another
20:55:03 eandersson also not a helpful comment
20:55:08 mriedem eandersson: sorry,
20:55:15 mriedem you've been at rocky for a couple of weeks now haven't you/
20:55:16 mriedem ?
20:55:26 eandersson Yes - and things have been working great
20:55:31 eandersson but we started seeing some odd errors
20:55:38 dansmith eandersson: don't mind mriedem he's just cranky today.. it's about nap time
20:55:42 openstackgerrit Merged openstack/nova stable/stein: Add retry_on_deadlock to migration_update DB API https://review.openstack.org/648428
20:55:51 mriedem so you're at rocky but still not seeing anything in placement? like resource providers? or just allocations?
20:56:09 mriedem i'm not sure how you could create a server...
20:56:19 eandersson We see new servers in placement
20:56:26 mriedem but not the old ones
20:56:39 mriedem *you don't see allocations for old servers in placement
20:56:40 eandersson The problem is that now we try to run the above command and it fails
20:57:03 mriedem eandersson: paste me the error
20:57:11 mriedem i should have put a dry run on that command
20:57:18 mriedem it was in the todo list
20:57:20 melwitt nova meeting in a few min
20:57:35 eandersson The first error we saw was this
20:57:36 eandersson > Instance xxx has allocations against this compute host but is not found in the database.
20:58:26 eandersson Next we ran the heal command, but for some reason it thinks the overprovisoned host is too full and fails to complete
20:58:53 mriedem the heal command is trying to PUT /allocations against a given resource provider (compute node) for a given instance,
20:59:07 mriedem and i've you've migrated or created servers on that compute node, placement is going to say "you're already at capacity"
20:59:20 mriedem which is why that PUT is probably failing even though there is already a server on it consuming resources
21:00:00 mriedem eandersson: so in that case, it'd be helpful to look at the inventory reported by placement for that compute node
21:00:16 efried nova meeting now #openstack-meeting
21:00:22 mriedem eandersson: which you can get with this https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-show
21:00:29 mriedem uuid is the uuid of the compute node
21:00:41 mriedem sorry https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list is probably more helpful
21:02:23 ceryx After running heal_allocations we have the same consumer_id (which I believe is the compute instance ID) have allocations created across 6 separate resource providers, each being a different compute node
21:02:48 mriedem ceryx: are you and eandersson in kahoots or just coincidental?
21:02:56 eandersson kahoots :D
21:03:00 mriedem ok
21:03:01 mriedem good
21:03:01 ceryx For RAM, 5 of those allocations are for 32GB but one is for 64GB which seems also strange (and yes sorry, was about to clarify, working with eandersson)
21:04:19 mriedem the consumer id is the instance id yes
21:04:32 mriedem has the instance been migrating around?
21:05:27 efried that 64... if we're leaking allocations and we move an instance away and back...?
21:07:21 ceryx http://paste.openstack.org/show/749212/ this is what we're seeing for one consumer ID. I checked nova.migrations and don't see this instance mentioned.
21:07:30 openstackgerrit Merged openstack/nova stable/stein: Use Selection object to fill request group mapping https://review.openstack.org/647713
21:07:42 mriedem it could be a migration record
21:07:48 mriedem check your migrations table in the cell db for that id
21:08:31 mriedem do you see errors in the logs when you're doing a migration?
21:09:24 mriedem my guess is you've done cold migrations / resize and the source node allocations, which are held by the migration record consumer, are not getting cleaned up for some reason
21:10:22 mriedem the multiple allocations for the same consumer and provider are because of different resource classes, mostly like VCPU and MEMORY_MB
21:10:26 eandersson We have had migrations disabled for a very long time
21:10:26 ceryx Ah yep, I was checking uuid and not instance_uuid from migrations. This had previously (about a month ago) had 6 failed migrations, then one confirmed migration.
21:10:27 mriedem if these are volume-backed servers
21:10:37 eandersson oh so nvm :D
21:12:59 mriedem ceryx: in a nutshell, when you start a cold or live migration, the allocations held by the instance (instances.uuid) on the source node provider are moved to a migration record consumer (consumer_id=migrations.uuid), and the dest node provider allocations are held by the instance during scheduling.
21:13:11 mriedem https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html
21:13:45 mriedem when the cold migration / resize is confirmed the allocations against the source node held by the migration record should be deleted
21:14:04 mriedem and you should just be left with the instance consuming allocations from the target provider
21:14:42 mriedem so in this paste is fec5409b-010e-4316-845c-ef68440d3593 an instance uuid or migration uuid?
21:15:10 mriedem looks like resource_class_id=0 is for VCPU and resource_class_id=1 is for MEMORY_MB
21:15:17 ceryx In this case the computes that we see unexpected allocations on were target hypervisors of an errored migration. So it looks like heal_allocations might be creating allocations for migrations in error state?
21:15:21 mriedem and this is a volume-backed server because there is no DISK_GB allocation
21:15:33 ceryx And yes - this has a cinder backed root disk
21:16:32 mriedem heal_allocations shouldn't even be looking at migrations, just instances, but it's been awhile since i dug into this
21:17:01 openstackgerrit Merged openstack/nova stable/stein: Add functional recreate test for bug 1819963 https://review.openstack.org/648401
21:17:02 openstack bug 1819963 in OpenStack Compute (nova) stein "Reverting a resize does not update the instance.availability_zone value to the source az" [Medium,In progress] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem)
21:21:17 mriedem so heal_allocations should skip the instance if it doesn't have a node set, but if the migration failed the node shouldn't keep changing
21:21:50 mriedem and it also shouldn't PUT new allocations if the instance already has allocations in placement, which it should have if you tried migrating it since you upgraded to rocky

Earlier   Later