Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-12
15:25:10 mriedem i guess there are some old migrations that dropped really old tables
15:25:10 stephenfin in 'nova/tests/unit/db/test_migrations.py'
15:25:19 mriedem 232, 233
15:25:25 dansmith mriedem: what were the tables?
15:25:30 stephenfin 346
15:25:40 stephenfin https://review.opendev.org/#/c/698705/1/nova/tests/unit/db/test_migrations.py@234
15:25:44 mriedem dump_*, compute_node_stats, instance_group_metadata
15:26:14 mriedem iscsi_targets, volumes
15:26:30 mriedem 292 was the most recent table drop migration
15:26:31 dansmith oh, gosh of course, from glance and cinder splits
15:26:49 mriedem https://review.opendev.org/#/c/166214/ being the most recent
15:26:50 dansmith stephenfin: 346 was a column
15:27:25 mriedem liberty, 2015
15:27:28 stephenfin ah, that explains why it told me I wasn't allowed drop a column when I was dropping a table. Same heuristic, I guess
15:27:30 mriedem the wild west of yore
15:28:02 stephenfin I assume there's a reason not to drop tables?
15:28:13 dansmith mriedem: liberty was the 292?
15:28:18 mriedem dansmith: yeah
15:28:26 dansmith mriedem: if so, those tables must've been unused for longer than that
15:28:32 stephenfin The big scary comments say don't do it but not why
15:28:41 mriedem dansmith: likely since grizzly/folsom after the cinder split
15:29:05 dansmith stephenfin: for the same reasons as not dropping columns, but if the code is gone it's different
15:29:25 dansmith stephenfin: so the only thing that worries me is doing the models and the migrations at the same time
15:30:08 dansmith i.e. I worry about applying the migrations from U to T at runtime (which we should be able to do) and then introspection being mad
15:30:16 mriedem https://review.opendev.org/#/c/698705/ (snapshots table) should have likely gone with the 292 migration since it's related to volume snapshots
15:30:45 dansmith stephenfin: so if you can do the migrations first and the model drops after and not have anything complain, that'd be a good test
15:31:08 stephenfin I'll give that a lash
15:31:14 stephenfin Need some TLC anyway, clearly
15:31:17 stephenfin *needs
15:31:19 mriedem stephenfin: in case you're taking notes, quota tables are in the api db too https://review.opendev.org/#/c/698706/1
15:31:24 dansmith stephenfin: also, I know you're just going for stats, but there's no reason to add a bunch of new migrations each to drop a thing
15:31:59 dansmith stephenfin: so, it'd be better if you put the drops all in one patch ahead of these, and then you can remove the models character by character in the following patches if you want
15:32:25 dansmith each migration slows down the process of building the schema on a new machine
15:32:47 stephenfin Kewl. I was trying to separate them logically
15:32:54 stephenfin didn't think about perf
15:33:18 stephenfin do we need to still care about placement-related tables?
15:33:42 stephenfin i.e. will the separate placement service ever use nova's DB?
15:33:53 dansmith well, I was going to say no,
15:34:08 dansmith but... people could easily have just pointed placement at nova and kept going, until some point
15:34:16 dansmith has placement had a db migration since they left?
15:34:16 mriedem the cutover was stein, you'd use the nova api db during the upgrade, but yeah..that
15:34:23 mriedem i think so
15:34:36 dansmith either way, probably worth a healthy warning
15:35:06 mriedem placement added a couple of blocker migrations
15:35:07 dansmith and tbh, if we're being very responsible, we might want a nova-manage delete-old-placement-stuff, which empties the tables and then drop the tables in the next release, contingent on them being empty
15:35:31 stephenfin your call
15:36:02 TheJulia melwitt: to your cpu arch question, I suspect you can rip out shipping the cpu arch from nova to ironic. Most of the code seems to rely upon stated properties of the machine's cpuarch and not the instance_info's cpu arch which makes sense from a deployment standpoint.
15:36:34 TheJulia melwitt: worth trying and making sure ironic doesn't explode with a test change?
15:36:45 mriedem dansmith: and what happens if someone doesn't run delete-old-placement-stuff? does the drop migration in V become a blocker migration until delete-old-placement-stuff is run?
15:37:09 dansmith mriedem: yeah, the drop just fails because we check for emptiness and raise
15:37:25 mriedem and is delete-old-placement-stuff checking for anything or just deleting records?
15:37:47 dansmith mriedem: no, just deleting.. figure if you're running a command like that you're opting in
15:38:50 mriedem so it's purely a signal to say yes i've migrated the placement tables from nova_api to placement back in stein (or whenever i upgraded) and i'm good to go,
15:38:54 mriedem and if not, i can't get to V
15:40:26 dansmith right
15:40:38 dansmith we could just nuke and drop and say "well, you should have known"
15:40:55 mriedem and in V+3 releases someone will be like, "wtf bro!"
15:40:57 dansmith but given the original prescription was to use this db as the way to migrate, it'd be super easy for people to have not moved
15:41:27 mriedem though thankfully you can rebuild the basic placement db stuff pretty easily as long as you're not using qos ports of cyborg thingies
15:41:38 mriedem *or cyborg thingies
15:41:56 mriedem ...or blazar...
15:42:15 mriedem or whatever VIO hacks into it for out of tree unicorns
15:42:18 mriedem it's all very stable
15:43:57 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Revert "docs: Highlight the current broken state of SEV" (partially) https://review.opendev.org/698746
15:49:47 mriedem TheJulia: while you're about, i've marked this as incomplete but does it make any sense to you? https://bugs.launchpad.net/nova/+bug/1853389
15:49:47 openstack Launchpad bug 1853389 in OpenStack Compute (nova) "The ipv6 address in network_data.json within configdrive is wrong when using IronicDriver to deploy BM" [Undecided,Incomplete] - Assigned to Eric Lei (leiyashuai)
15:51:04 mriedem ok i guess they call https://docs.openstack.org/api-ref/baremetal/#attach-a-vif-to-a-node while creating a server
15:51:13 mriedem i'd think ironic would disallow attaching a vif while a node is being provisioned
15:51:26 mriedem oh i can't add ironic to this....storyboard
15:58:04 mriedem eandersson: i've left some comments in https://bugs.launchpad.net/nova/+bug/1854992 but marked it incomplete
15:58:04 openstack Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,Incomplete]
15:58:22 mriedem it's not clear to me what you think nova could do besides changing rpc casts to calls and i'm not sure what that would help if rabbit is dead
15:58:49 mriedem e.g. let's say rabbit is f'ed and the compute gets an error back, it needs to use rabbit to update the instance status to ERROR, so you have a chicken and egg problem
15:59:12 mriedem if it's conductor that notices it has direct access to the db so then we could update, but i don't think we're going to be changing the build_and_run_instances cast to a call...
15:59:19 mriedem dansmith: maybe you have ideas ^
16:00:35 mriedem conductor could grow a threaded poll per instance that's building or something with a hard timeout? idk
16:00:52 dansmith conductor should definitely not imply some instance state like that
16:01:09 dansmith if rabbit is effed then all bets are off
16:01:16 sean-k-mooney mriedem: woudl a reset-state command not just update the statis in the db. i did not think that need to do an rpc to set the instance state
16:01:21 mriedem right any attempts here seem like hacks
16:01:33 mriedem reset state from the api is direct to the db
16:01:45 mriedem so yeah you can reset the state to error or just delete the instance
16:01:55 mriedem but they want the instances to automatically go to error when rabbit is down
16:01:59 sean-k-mooney ya so woudl that not bypass your chicken and egg problem
16:02:00 mriedem or at least have nova log something
16:02:17 sean-k-mooney oh right
16:02:25 mriedem sean-k-mooney: the admin would still have to manually do the needful
16:02:51 sean-k-mooney did i look at this last week. i this for blizzard?
16:02:58 mriedem they could have an external background service/script/whatever that polls the servers that are building and if they pass some timeout they just reset to error or nuke them and try again
16:03:01 mriedem yes
16:03:23 sean-k-mooney ok so we were talking about the mandatory flag for rabbitmq
16:03:30 mriedem if rabbit is dead b/c it can't keep up though, constantly hammering it is probably not the solution
16:04:40 sean-k-mooney right so the mandatory flag woudl raise an error on our side if we could not enqueue the packet which woudl allows us to detect the failure and then either retry with a different host if rabbit was up but the compute node queue was not found or error
16:04:41 dansmith trying to work around dead rabbit is like saying nova should be able to do something smart if the db is down
16:05:21 sean-k-mooney dansmith: right but there are cases where rabbit is fine but the compute agent topic queue is not present
16:05:43 dansmith sean-k-mooney: how is that not a rabbit failure?
16:05:59 mriedem dansmith: hear me out, if the db is down, auto switch to etcd and then when the db is back up, just like merge all the etcd stuff back into mysql
16:06:14 dansmith I don't mean a failure where we can point to rabbit as the sole responsible party, but for something that causes us to not be able to use rabbit for whatever reason
16:06:22 dansmith mriedem: obviously that's the proper solution
16:06:32 mriedem \o/

Earlier   Later