Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-11
20:40:33 melwitt this came up in downstream bug triage, cells v2 discover_hosts can fail with DBDuplicateEntry if run in parallel per compute host deployed
20:41:01 dansmith melwitt: I'm sure
20:41:10 melwitt found related issues from OSA and chef: https://bugs.launchpad.net/openstack-ansible/+bug/1752540 https://github.com/bloomberg/chef-bcpc/issues/1378
20:41:11 dansmith why would someone run it on each compute node
20:41:11 openstack Launchpad bug 1752540 in openstack-ansible "os_nova cell_v2 discover failure" [Undecided,In progress] - Assigned to git-harry (git-harry)
20:42:06 dansmith ah the OSA one seems to be that they're running it per conductor
20:42:18 melwitt maybe lack of clear documentation but probably just thinking deploy compute host => discover host and made it part of that task piece
20:42:18 dansmith which is also not really what should be happening, but less bad than compute I guess
20:42:45 dansmith except you have to give the compute node db credentials it doesn't otherwise need in order to do that :)
20:42:55 dansmith but yeah, misunderstanding of what should be done, I'm sure
20:43:42 mriedem i want to say mdbooth_ opened some bug like this
20:43:47 mriedem concurrency something or other
20:44:07 mriedem but it was like db sync concurrently i think
20:44:08 dansmith it's not really a concurrency thing in the way he normally looks for such issues,
20:44:28 dansmith yeah, that also is one of those "don't do that" things, IMHO
20:44:33 melwitt hm, I wasn't thinking that they give db credentials to compute node, just running discover_host per, centrally? I dunno, nevermind
20:44:45 dansmith unless we want to do our own locking in the DB to prevent it, which is kinda silly
20:44:58 mriedem it was this https://bugs.launchpad.net/nova/+bug/1804652
20:44:59 openstack Launchpad bug 1804652 in OpenStack Compute (nova) "nova.db.sqlalchemy.migration.db_version is racy" [Low,In progress] - Assigned to Matthew Booth (mbooth-9)
20:45:13 dansmith melwitt: you have to have api db credentials to run discover hosts, which are normally not on the compute node (or shouldn't be)
20:45:51 dansmith mriedem: yeah, would love to WONTFIX that
20:46:05 mriedem you could -1 the change
20:46:07 mriedem start a war
20:46:22 mriedem but this is now the month of positivity and motivation
20:46:38 melwitt dansmith: yeah... I was thinking maybe they're doing that in a central place that is deploying compute nodes in parallel but not necessarily running nova-manage _on_ the compute nodes?
20:47:26 dansmith melwitt: who are we talking about? I thought you said "per compute" so I thought you meant on a compute, but maybe you just mean spawning a bunch of nova-manage commands on one node one for each new host?
20:47:32 melwitt I'll reply on the bug with guidance to run discover_hosts once after deploying compute hosts. and then I think I'll have a docs patch in my future
20:47:33 dansmith that's even more.. crazy
20:47:49 dansmith should be using anisble triggers for that, which AFAIK, would mean a thing runs one
20:47:52 dansmith *once
20:48:08 dansmith like, you don't restart apache for every module you enable, you enable a bunch of modules and run restart at the end :)
20:48:25 melwitt dansmith: I don't actually know the details of how the deployment works, was just thinking it seems unlikely they're putting api db creds on compute hosts and running nova-manage on them
20:48:26 dansmith s/triggers/handlers/
20:48:45 dansmith melwitt: ack, I just thought that was what you originally asserted
20:49:14 openstackgerrit Matt Riedemann proposed openstack/python-novaclient stable/stein: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651925
20:49:15 openstack bug 1746534 in python-novaclient "encoding error when doing console-log" [High,Fix released] https://launchpad.net/bugs/1746534 - Assigned to Thomas Goirand (thomas-goirand)
20:49:19 melwitt no, I didn't mean to make it sound like that
20:49:28 openstackgerrit Matt Riedemann proposed openstack/python-novaclient stable/stein: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651925
20:50:14 dansmith melwitt: so, we could do an external lock on nova-manage which would prevent the specific case of running multiple copies on a single host,
20:50:27 dansmith but it might also be confusing because it won't fix the case where it's multiple on different hosts
20:50:52 dansmith we could do some janky locking in the database to try to catch it, but I'd rather just say "do not do this lest ye burn in nova purgatory"
20:50:55 dansmith or, you know, something
20:51:10 mriedem i smell etcd
20:51:11 mriedem and tooz
20:51:20 melwitt lol.. yes
20:51:23 mriedem that's like 2 etcd references for nova in the last few weeks
20:51:29 dansmith yes, let's deploy and depend on etcd instead of people using ansible handlers! :P
20:52:17 melwitt ok, I'll reply with guidance on what to do. thanks y'all
20:52:19 dansmith mriedem: make sure the spec says that etcd is only running when nova-manage is used, and stopped at runtime
20:53:17 eandersson Do you need to run nova-manage placement heal_allocations when upgrading from non-cell to Cell V2?
20:53:27 mriedem eandersson: no
20:53:32 mriedem doesn't have anything to do with cells
20:53:44 eandersson Let me re-phrase, when upgrading from Mitaka to Rocky
20:53:56 mriedem it was built for people migrating from caching scheduler - which didn't use placement and thus didn't create allocations - to filter scheduler
20:54:10 mriedem eandersson: in that case...maybe
20:54:12 eandersson because placement is empty for us
20:54:15 mriedem heh
20:54:17 eandersson and it was never in any of the upgrade steps
20:54:31 mriedem because "upgrading from 1983 to now" isn't a supported upgrade
20:54:45 eandersson well we didn't upgrade from mitaka to rocky
20:54:51 eandersson we upgraded from one version to another
20:55:03 eandersson also not a helpful comment
20:55:08 mriedem eandersson: sorry,
20:55:15 mriedem you've been at rocky for a couple of weeks now haven't you/
20:55:16 mriedem ?
20:55:26 eandersson Yes - and things have been working great
20:55:31 eandersson but we started seeing some odd errors
20:55:38 dansmith eandersson: don't mind mriedem he's just cranky today.. it's about nap time
20:55:42 openstackgerrit Merged openstack/nova stable/stein: Add retry_on_deadlock to migration_update DB API https://review.openstack.org/648428
20:55:51 mriedem so you're at rocky but still not seeing anything in placement? like resource providers? or just allocations?
20:56:09 mriedem i'm not sure how you could create a server...
20:56:19 eandersson We see new servers in placement
20:56:26 mriedem but not the old ones
20:56:39 mriedem *you don't see allocations for old servers in placement
20:56:40 eandersson The problem is that now we try to run the above command and it fails
20:57:03 mriedem eandersson: paste me the error
20:57:11 mriedem i should have put a dry run on that command
20:57:18 mriedem it was in the todo list
20:57:20 melwitt nova meeting in a few min
20:57:35 eandersson The first error we saw was this
20:57:36 eandersson > Instance xxx has allocations against this compute host but is not found in the database.
20:58:26 eandersson Next we ran the heal command, but for some reason it thinks the overprovisoned host is too full and fails to complete
20:58:53 mriedem the heal command is trying to PUT /allocations against a given resource provider (compute node) for a given instance,
20:59:07 mriedem and i've you've migrated or created servers on that compute node, placement is going to say "you're already at capacity"
20:59:20 mriedem which is why that PUT is probably failing even though there is already a server on it consuming resources
21:00:00 mriedem eandersson: so in that case, it'd be helpful to look at the inventory reported by placement for that compute node
21:00:16 efried nova meeting now #openstack-meeting
21:00:22 mriedem eandersson: which you can get with this https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-show
21:00:29 mriedem uuid is the uuid of the compute node
21:00:41 mriedem sorry https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list is probably more helpful
21:02:23 ceryx After running heal_allocations we have the same consumer_id (which I believe is the compute instance ID) have allocations created across 6 separate resource providers, each being a different compute node
21:02:48 mriedem ceryx: are you and eandersson in kahoots or just coincidental?
21:02:56 eandersson kahoots :D
21:03:00 mriedem ok
21:03:01 ceryx For RAM, 5 of those allocations are for 32GB but one is for 64GB which seems also strange (and yes sorry, was about to clarify, working with eandersson)
21:03:01 mriedem good
21:04:19 mriedem the consumer id is the instance id yes
21:04:32 mriedem has the instance been migrating around?
21:05:27 efried that 64... if we're leaking allocations and we move an instance away and back...?
21:07:21 ceryx http://paste.openstack.org/show/749212/ this is what we're seeing for one consumer ID. I checked nova.migrations and don't see this instance mentioned.

Earlier   Later