| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-11 | |||
| 19:22:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | |
| 19:22:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650 | |
| 19:22:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | |
| 19:22:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | |
| 19:22:35 | efried | sean-k-mooney: you still awake? | |
| 19:36:40 | melwitt | efried: fyi, patch to add a few post-release items to the ptl guide https://review.openstack.org/651009 | |
| 19:40:55 | efried | melwitt: ack, on the radar, thank you. | |
| 19:41:52 | efried | oh, that was easy. +2 | |
| 20:00:09 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [DNM: extra logs] Revert resize: wait for external events in compute manager https://review.openstack.org/644881 | |
| 20:08:10 | mriedem | heh i didn't realize there was a ptl guide | |
| 20:14:28 | openstackgerrit | Merged openstack/python-novaclient master: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651827 | |
| 20:14:29 | openstack | bug 1746534 in python-novaclient "encoding error when doing console-log" [High,Fix released] https://launchpad.net/bugs/1746534 - Assigned to Thomas Goirand (thomas-goirand) | |
| 20:22:15 | melwitt | mriedem: I added it recently. was just a copy-paste of a google doc I had made to help me as ptl | |
| 20:40:33 | melwitt | this came up in downstream bug triage, cells v2 discover_hosts can fail with DBDuplicateEntry if run in parallel per compute host deployed | |
| 20:41:01 | dansmith | melwitt: I'm sure | |
| 20:41:10 | melwitt | found related issues from OSA and chef: https://bugs.launchpad.net/openstack-ansible/+bug/1752540 https://github.com/bloomberg/chef-bcpc/issues/1378 | |
| 20:41:11 | openstack | Launchpad bug 1752540 in openstack-ansible "os_nova cell_v2 discover failure" [Undecided,In progress] - Assigned to git-harry (git-harry) | |
| 20:41:11 | dansmith | why would someone run it on each compute node | |
| 20:42:06 | dansmith | ah the OSA one seems to be that they're running it per conductor | |
| 20:42:18 | dansmith | which is also not really what should be happening, but less bad than compute I guess | |
| 20:42:18 | melwitt | maybe lack of clear documentation but probably just thinking deploy compute host => discover host and made it part of that task piece | |
| 20:42:45 | dansmith | except you have to give the compute node db credentials it doesn't otherwise need in order to do that :) | |
| 20:42:55 | dansmith | but yeah, misunderstanding of what should be done, I'm sure | |
| 20:43:42 | mriedem | i want to say mdbooth_ opened some bug like this | |
| 20:43:47 | mriedem | concurrency something or other | |
| 20:44:07 | mriedem | but it was like db sync concurrently i think | |
| 20:44:08 | dansmith | it's not really a concurrency thing in the way he normally looks for such issues, | |
| 20:44:28 | dansmith | yeah, that also is one of those "don't do that" things, IMHO | |
| 20:44:33 | melwitt | hm, I wasn't thinking that they give db credentials to compute node, just running discover_host per, centrally? I dunno, nevermind | |
| 20:44:45 | dansmith | unless we want to do our own locking in the DB to prevent it, which is kinda silly | |
| 20:44:58 | mriedem | it was this https://bugs.launchpad.net/nova/+bug/1804652 | |
| 20:44:59 | openstack | Launchpad bug 1804652 in OpenStack Compute (nova) "nova.db.sqlalchemy.migration.db_version is racy" [Low,In progress] - Assigned to Matthew Booth (mbooth-9) | |
| 20:45:13 | dansmith | melwitt: you have to have api db credentials to run discover hosts, which are normally not on the compute node (or shouldn't be) | |
| 20:45:51 | dansmith | mriedem: yeah, would love to WONTFIX that | |
| 20:46:05 | mriedem | you could -1 the change | |
| 20:46:07 | mriedem | start a war | |
| 20:46:22 | mriedem | but this is now the month of positivity and motivation | |
| 20:46:38 | melwitt | dansmith: yeah... I was thinking maybe they're doing that in a central place that is deploying compute nodes in parallel but not necessarily running nova-manage _on_ the compute nodes? | |
| 20:47:26 | dansmith | melwitt: who are we talking about? I thought you said "per compute" so I thought you meant on a compute, but maybe you just mean spawning a bunch of nova-manage commands on one node one for each new host? | |
| 20:47:32 | melwitt | I'll reply on the bug with guidance to run discover_hosts once after deploying compute hosts. and then I think I'll have a docs patch in my future | |
| 20:47:33 | dansmith | that's even more.. crazy | |
| 20:47:49 | dansmith | should be using anisble triggers for that, which AFAIK, would mean a thing runs one | |
| 20:47:52 | dansmith | *once | |
| 20:48:08 | dansmith | like, you don't restart apache for every module you enable, you enable a bunch of modules and run restart at the end :) | |
| 20:48:25 | melwitt | dansmith: I don't actually know the details of how the deployment works, was just thinking it seems unlikely they're putting api db creds on compute hosts and running nova-manage on them | |
| 20:48:26 | dansmith | s/triggers/handlers/ | |
| 20:48:45 | dansmith | melwitt: ack, I just thought that was what you originally asserted | |
| 20:49:14 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient stable/stein: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651925 | |
| 20:49:15 | openstack | bug 1746534 in python-novaclient "encoding error when doing console-log" [High,Fix released] https://launchpad.net/bugs/1746534 - Assigned to Thomas Goirand (thomas-goirand) | |
| 20:49:19 | melwitt | no, I didn't mean to make it sound like that | |
| 20:49:28 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient stable/stein: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651925 | |
| 20:50:14 | dansmith | melwitt: so, we could do an external lock on nova-manage which would prevent the specific case of running multiple copies on a single host, | |
| 20:50:27 | dansmith | but it might also be confusing because it won't fix the case where it's multiple on different hosts | |
| 20:50:52 | dansmith | we could do some janky locking in the database to try to catch it, but I'd rather just say "do not do this lest ye burn in nova purgatory" | |
| 20:50:55 | dansmith | or, you know, something | |
| 20:51:10 | mriedem | i smell etcd | |
| 20:51:11 | mriedem | and tooz | |
| 20:51:20 | melwitt | lol.. yes | |
| 20:51:23 | mriedem | that's like 2 etcd references for nova in the last few weeks | |
| 20:51:29 | dansmith | yes, let's deploy and depend on etcd instead of people using ansible handlers! :P | |
| 20:52:17 | melwitt | ok, I'll reply with guidance on what to do. thanks y'all | |
| 20:52:19 | dansmith | mriedem: make sure the spec says that etcd is only running when nova-manage is used, and stopped at runtime | |
| 20:53:17 | eandersson | Do you need to run nova-manage placement heal_allocations when upgrading from non-cell to Cell V2? | |
| 20:53:27 | mriedem | eandersson: no | |
| 20:53:32 | mriedem | doesn't have anything to do with cells | |
| 20:53:44 | eandersson | Let me re-phrase, when upgrading from Mitaka to Rocky | |
| 20:53:56 | mriedem | it was built for people migrating from caching scheduler - which didn't use placement and thus didn't create allocations - to filter scheduler | |
| 20:54:10 | mriedem | eandersson: in that case...maybe | |
| 20:54:12 | eandersson | because placement is empty for us | |
| 20:54:15 | mriedem | heh | |
| 20:54:17 | eandersson | and it was never in any of the upgrade steps | |
| 20:54:31 | mriedem | because "upgrading from 1983 to now" isn't a supported upgrade | |
| 20:54:45 | eandersson | well we didn't upgrade from mitaka to rocky | |
| 20:54:51 | eandersson | we upgraded from one version to another | |
| 20:55:03 | eandersson | also not a helpful comment | |
| 20:55:08 | mriedem | eandersson: sorry, | |
| 20:55:15 | mriedem | you've been at rocky for a couple of weeks now haven't you/ | |
| 20:55:16 | mriedem | ? | |
| 20:55:26 | eandersson | Yes - and things have been working great | |
| 20:55:31 | eandersson | but we started seeing some odd errors | |
| 20:55:38 | dansmith | eandersson: don't mind mriedem he's just cranky today.. it's about nap time | |
| 20:55:42 | openstackgerrit | Merged openstack/nova stable/stein: Add retry_on_deadlock to migration_update DB API https://review.openstack.org/648428 | |
| 20:55:51 | mriedem | so you're at rocky but still not seeing anything in placement? like resource providers? or just allocations? | |
| 20:56:09 | mriedem | i'm not sure how you could create a server... | |
| 20:56:19 | eandersson | We see new servers in placement | |
| 20:56:26 | mriedem | but not the old ones | |
| 20:56:39 | mriedem | *you don't see allocations for old servers in placement | |
| 20:56:40 | eandersson | The problem is that now we try to run the above command and it fails | |
| 20:57:03 | mriedem | eandersson: paste me the error | |
| 20:57:11 | mriedem | i should have put a dry run on that command | |
| 20:57:18 | mriedem | it was in the todo list | |
| 20:57:20 | melwitt | nova meeting in a few min | |
| 20:57:35 | eandersson | The first error we saw was this | |
| 20:57:36 | eandersson | > Instance xxx has allocations against this compute host but is not found in the database. | |
| 20:58:26 | eandersson | Next we ran the heal command, but for some reason it thinks the overprovisoned host is too full and fails to complete | |
| 20:58:53 | mriedem | the heal command is trying to PUT /allocations against a given resource provider (compute node) for a given instance, | |
| 20:59:07 | mriedem | and i've you've migrated or created servers on that compute node, placement is going to say "you're already at capacity" | |
| 20:59:20 | mriedem | which is why that PUT is probably failing even though there is already a server on it consuming resources | |
| 21:00:00 | mriedem | eandersson: so in that case, it'd be helpful to look at the inventory reported by placement for that compute node | |
| 21:00:16 | efried | nova meeting now #openstack-meeting | |