| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-12 | |||
| 15:23:18 | bbobrov | oh nice | |
| 15:23:19 | dansmith | tables? | |
| 15:23:25 | bbobrov | so we backport the fixes to train | |
| 15:23:37 | dansmith | never? | |
| 15:23:44 | mriedem | that's what i thought | |
| 15:23:47 | stephenfin | bbobrov: You mean you (SUSE) have done it already | |
| 15:23:55 | stephenfin | dansmith: You sit on a throne of LIES | |
| 15:25:01 | stephenfin | Search for 'assertTableNotExists' | |
| 15:25:10 | mriedem | i guess there are some old migrations that dropped really old tables | |
| 15:25:10 | stephenfin | in 'nova/tests/unit/db/test_migrations.py' | |
| 15:25:19 | mriedem | 232, 233 | |
| 15:25:25 | dansmith | mriedem: what were the tables? | |
| 15:25:30 | stephenfin | 346 | |
| 15:25:40 | stephenfin | https://review.opendev.org/#/c/698705/1/nova/tests/unit/db/test_migrations.py@234 | |
| 15:25:44 | mriedem | dump_*, compute_node_stats, instance_group_metadata | |
| 15:26:14 | mriedem | iscsi_targets, volumes | |
| 15:26:30 | mriedem | 292 was the most recent table drop migration | |
| 15:26:31 | dansmith | oh, gosh of course, from glance and cinder splits | |
| 15:26:49 | mriedem | https://review.opendev.org/#/c/166214/ being the most recent | |
| 15:26:50 | dansmith | stephenfin: 346 was a column | |
| 15:27:25 | mriedem | liberty, 2015 | |
| 15:27:28 | stephenfin | ah, that explains why it told me I wasn't allowed drop a column when I was dropping a table. Same heuristic, I guess | |
| 15:27:30 | mriedem | the wild west of yore | |
| 15:28:02 | stephenfin | I assume there's a reason not to drop tables? | |
| 15:28:13 | dansmith | mriedem: liberty was the 292? | |
| 15:28:18 | mriedem | dansmith: yeah | |
| 15:28:26 | dansmith | mriedem: if so, those tables must've been unused for longer than that | |
| 15:28:32 | stephenfin | The big scary comments say don't do it but not why | |
| 15:28:41 | mriedem | dansmith: likely since grizzly/folsom after the cinder split | |
| 15:29:05 | dansmith | stephenfin: for the same reasons as not dropping columns, but if the code is gone it's different | |
| 15:29:25 | dansmith | stephenfin: so the only thing that worries me is doing the models and the migrations at the same time | |
| 15:30:08 | dansmith | i.e. I worry about applying the migrations from U to T at runtime (which we should be able to do) and then introspection being mad | |
| 15:30:16 | mriedem | https://review.opendev.org/#/c/698705/ (snapshots table) should have likely gone with the 292 migration since it's related to volume snapshots | |
| 15:30:45 | dansmith | stephenfin: so if you can do the migrations first and the model drops after and not have anything complain, that'd be a good test | |
| 15:31:08 | stephenfin | I'll give that a lash | |
| 15:31:14 | stephenfin | Need some TLC anyway, clearly | |
| 15:31:17 | stephenfin | *needs | |
| 15:31:19 | mriedem | stephenfin: in case you're taking notes, quota tables are in the api db too https://review.opendev.org/#/c/698706/1 | |
| 15:31:24 | dansmith | stephenfin: also, I know you're just going for stats, but there's no reason to add a bunch of new migrations each to drop a thing | |
| 15:31:59 | dansmith | stephenfin: so, it'd be better if you put the drops all in one patch ahead of these, and then you can remove the models character by character in the following patches if you want | |
| 15:32:25 | dansmith | each migration slows down the process of building the schema on a new machine | |
| 15:32:47 | stephenfin | Kewl. I was trying to separate them logically | |
| 15:32:54 | stephenfin | didn't think about perf | |
| 15:33:18 | stephenfin | do we need to still care about placement-related tables? | |
| 15:33:42 | stephenfin | i.e. will the separate placement service ever use nova's DB? | |
| 15:33:53 | dansmith | well, I was going to say no, | |
| 15:34:08 | dansmith | but... people could easily have just pointed placement at nova and kept going, until some point | |
| 15:34:16 | dansmith | has placement had a db migration since they left? | |
| 15:34:16 | mriedem | the cutover was stein, you'd use the nova api db during the upgrade, but yeah..that | |
| 15:34:23 | mriedem | i think so | |
| 15:34:36 | dansmith | either way, probably worth a healthy warning | |
| 15:35:06 | mriedem | placement added a couple of blocker migrations | |
| 15:35:07 | dansmith | and tbh, if we're being very responsible, we might want a nova-manage delete-old-placement-stuff, which empties the tables and then drop the tables in the next release, contingent on them being empty | |
| 15:35:31 | stephenfin | your call | |
| 15:36:02 | TheJulia | melwitt: to your cpu arch question, I suspect you can rip out shipping the cpu arch from nova to ironic. Most of the code seems to rely upon stated properties of the machine's cpuarch and not the instance_info's cpu arch which makes sense from a deployment standpoint. | |
| 15:36:34 | TheJulia | melwitt: worth trying and making sure ironic doesn't explode with a test change? | |
| 15:36:45 | mriedem | dansmith: and what happens if someone doesn't run delete-old-placement-stuff? does the drop migration in V become a blocker migration until delete-old-placement-stuff is run? | |
| 15:37:09 | dansmith | mriedem: yeah, the drop just fails because we check for emptiness and raise | |
| 15:37:25 | mriedem | and is delete-old-placement-stuff checking for anything or just deleting records? | |
| 15:37:47 | dansmith | mriedem: no, just deleting.. figure if you're running a command like that you're opting in | |
| 15:38:50 | mriedem | so it's purely a signal to say yes i've migrated the placement tables from nova_api to placement back in stein (or whenever i upgraded) and i'm good to go, | |
| 15:38:54 | mriedem | and if not, i can't get to V | |
| 15:40:26 | dansmith | right | |
| 15:40:38 | dansmith | we could just nuke and drop and say "well, you should have known" | |
| 15:40:55 | mriedem | and in V+3 releases someone will be like, "wtf bro!" | |
| 15:40:57 | dansmith | but given the original prescription was to use this db as the way to migrate, it'd be super easy for people to have not moved | |
| 15:41:27 | mriedem | though thankfully you can rebuild the basic placement db stuff pretty easily as long as you're not using qos ports of cyborg thingies | |
| 15:41:38 | mriedem | *or cyborg thingies | |
| 15:41:56 | mriedem | ...or blazar... | |
| 15:42:15 | mriedem | or whatever VIO hacks into it for out of tree unicorns | |
| 15:42:18 | mriedem | it's all very stable | |
| 15:43:57 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Revert "docs: Highlight the current broken state of SEV" (partially) https://review.opendev.org/698746 | |
| 15:49:47 | mriedem | TheJulia: while you're about, i've marked this as incomplete but does it make any sense to you? https://bugs.launchpad.net/nova/+bug/1853389 | |
| 15:49:47 | openstack | Launchpad bug 1853389 in OpenStack Compute (nova) "The ipv6 address in network_data.json within configdrive is wrong when using IronicDriver to deploy BM" [Undecided,Incomplete] - Assigned to Eric Lei (leiyashuai) | |
| 15:51:04 | mriedem | ok i guess they call https://docs.openstack.org/api-ref/baremetal/#attach-a-vif-to-a-node while creating a server | |
| 15:51:13 | mriedem | i'd think ironic would disallow attaching a vif while a node is being provisioned | |
| 15:51:26 | mriedem | oh i can't add ironic to this....storyboard | |
| 15:58:04 | mriedem | eandersson: i've left some comments in https://bugs.launchpad.net/nova/+bug/1854992 but marked it incomplete | |
| 15:58:04 | openstack | Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,Incomplete] | |
| 15:58:22 | mriedem | it's not clear to me what you think nova could do besides changing rpc casts to calls and i'm not sure what that would help if rabbit is dead | |
| 15:58:49 | mriedem | e.g. let's say rabbit is f'ed and the compute gets an error back, it needs to use rabbit to update the instance status to ERROR, so you have a chicken and egg problem | |
| 15:59:12 | mriedem | if it's conductor that notices it has direct access to the db so then we could update, but i don't think we're going to be changing the build_and_run_instances cast to a call... | |
| 15:59:19 | mriedem | dansmith: maybe you have ideas ^ | |
| 16:00:35 | mriedem | conductor could grow a threaded poll per instance that's building or something with a hard timeout? idk | |
| 16:00:52 | dansmith | conductor should definitely not imply some instance state like that | |
| 16:01:09 | dansmith | if rabbit is effed then all bets are off | |
| 16:01:16 | sean-k-mooney | mriedem: woudl a reset-state command not just update the statis in the db. i did not think that need to do an rpc to set the instance state | |
| 16:01:21 | mriedem | right any attempts here seem like hacks | |
| 16:01:33 | mriedem | reset state from the api is direct to the db | |
| 16:01:45 | mriedem | so yeah you can reset the state to error or just delete the instance | |
| 16:01:55 | mriedem | but they want the instances to automatically go to error when rabbit is down | |
| 16:01:59 | sean-k-mooney | ya so woudl that not bypass your chicken and egg problem | |
| 16:02:00 | mriedem | or at least have nova log something | |
| 16:02:17 | sean-k-mooney | oh right | |
| 16:02:25 | mriedem | sean-k-mooney: the admin would still have to manually do the needful | |
| 16:02:51 | sean-k-mooney | did i look at this last week. i this for blizzard? | |
| 16:02:58 | mriedem | they could have an external background service/script/whatever that polls the servers that are building and if they pass some timeout they just reset to error or nuke them and try again | |
| 16:03:01 | mriedem | yes | |
| 16:03:23 | sean-k-mooney | ok so we were talking about the mandatory flag for rabbitmq | |
| 16:03:30 | mriedem | if rabbit is dead b/c it can't keep up though, constantly hammering it is probably not the solution | |