Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-12
15:33:53 dansmith well, I was going to say no,
15:34:08 dansmith but... people could easily have just pointed placement at nova and kept going, until some point
15:34:16 dansmith has placement had a db migration since they left?
15:34:16 mriedem the cutover was stein, you'd use the nova api db during the upgrade, but yeah..that
15:34:23 mriedem i think so
15:34:36 dansmith either way, probably worth a healthy warning
15:35:06 mriedem placement added a couple of blocker migrations
15:35:07 dansmith and tbh, if we're being very responsible, we might want a nova-manage delete-old-placement-stuff, which empties the tables and then drop the tables in the next release, contingent on them being empty
15:35:31 stephenfin your call
15:36:02 TheJulia melwitt: to your cpu arch question, I suspect you can rip out shipping the cpu arch from nova to ironic. Most of the code seems to rely upon stated properties of the machine's cpuarch and not the instance_info's cpu arch which makes sense from a deployment standpoint.
15:36:34 TheJulia melwitt: worth trying and making sure ironic doesn't explode with a test change?
15:36:45 mriedem dansmith: and what happens if someone doesn't run delete-old-placement-stuff? does the drop migration in V become a blocker migration until delete-old-placement-stuff is run?
15:37:09 dansmith mriedem: yeah, the drop just fails because we check for emptiness and raise
15:37:25 mriedem and is delete-old-placement-stuff checking for anything or just deleting records?
15:37:47 dansmith mriedem: no, just deleting.. figure if you're running a command like that you're opting in
15:38:50 mriedem so it's purely a signal to say yes i've migrated the placement tables from nova_api to placement back in stein (or whenever i upgraded) and i'm good to go,
15:38:54 mriedem and if not, i can't get to V
15:40:26 dansmith right
15:40:38 dansmith we could just nuke and drop and say "well, you should have known"
15:40:55 mriedem and in V+3 releases someone will be like, "wtf bro!"
15:40:57 dansmith but given the original prescription was to use this db as the way to migrate, it'd be super easy for people to have not moved
15:41:27 mriedem though thankfully you can rebuild the basic placement db stuff pretty easily as long as you're not using qos ports of cyborg thingies
15:41:38 mriedem *or cyborg thingies
15:41:56 mriedem ...or blazar...
15:42:15 mriedem or whatever VIO hacks into it for out of tree unicorns
15:42:18 mriedem it's all very stable
15:43:57 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Revert "docs: Highlight the current broken state of SEV" (partially) https://review.opendev.org/698746
15:49:47 mriedem TheJulia: while you're about, i've marked this as incomplete but does it make any sense to you? https://bugs.launchpad.net/nova/+bug/1853389
15:49:47 openstack Launchpad bug 1853389 in OpenStack Compute (nova) "The ipv6 address in network_data.json within configdrive is wrong when using IronicDriver to deploy BM" [Undecided,Incomplete] - Assigned to Eric Lei (leiyashuai)
15:51:04 mriedem ok i guess they call https://docs.openstack.org/api-ref/baremetal/#attach-a-vif-to-a-node while creating a server
15:51:13 mriedem i'd think ironic would disallow attaching a vif while a node is being provisioned
15:51:26 mriedem oh i can't add ironic to this....storyboard
15:58:04 mriedem eandersson: i've left some comments in https://bugs.launchpad.net/nova/+bug/1854992 but marked it incomplete
15:58:04 openstack Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,Incomplete]
15:58:22 mriedem it's not clear to me what you think nova could do besides changing rpc casts to calls and i'm not sure what that would help if rabbit is dead
15:58:49 mriedem e.g. let's say rabbit is f'ed and the compute gets an error back, it needs to use rabbit to update the instance status to ERROR, so you have a chicken and egg problem
15:59:12 mriedem if it's conductor that notices it has direct access to the db so then we could update, but i don't think we're going to be changing the build_and_run_instances cast to a call...
15:59:19 mriedem dansmith: maybe you have ideas ^
16:00:35 mriedem conductor could grow a threaded poll per instance that's building or something with a hard timeout? idk
16:00:52 dansmith conductor should definitely not imply some instance state like that
16:01:09 dansmith if rabbit is effed then all bets are off
16:01:16 sean-k-mooney mriedem: woudl a reset-state command not just update the statis in the db. i did not think that need to do an rpc to set the instance state
16:01:21 mriedem right any attempts here seem like hacks
16:01:33 mriedem reset state from the api is direct to the db
16:01:45 mriedem so yeah you can reset the state to error or just delete the instance
16:01:55 mriedem but they want the instances to automatically go to error when rabbit is down
16:01:59 sean-k-mooney ya so woudl that not bypass your chicken and egg problem
16:02:00 mriedem or at least have nova log something
16:02:17 sean-k-mooney oh right
16:02:25 mriedem sean-k-mooney: the admin would still have to manually do the needful
16:02:51 sean-k-mooney did i look at this last week. i this for blizzard?
16:02:58 mriedem they could have an external background service/script/whatever that polls the servers that are building and if they pass some timeout they just reset to error or nuke them and try again
16:03:01 mriedem yes
16:03:23 sean-k-mooney ok so we were talking about the mandatory flag for rabbitmq
16:03:30 mriedem if rabbit is dead b/c it can't keep up though, constantly hammering it is probably not the solution
16:04:40 sean-k-mooney right so the mandatory flag woudl raise an error on our side if we could not enqueue the packet which woudl allows us to detect the failure and then either retry with a different host if rabbit was up but the compute node queue was not found or error
16:04:41 dansmith trying to work around dead rabbit is like saying nova should be able to do something smart if the db is down
16:05:21 sean-k-mooney dansmith: right but there are cases where rabbit is fine but the compute agent topic queue is not present
16:05:43 dansmith sean-k-mooney: how is that not a rabbit failure?
16:05:59 mriedem dansmith: hear me out, if the db is down, auto switch to etcd and then when the db is back up, just like merge all the etcd stuff back into mysql
16:06:14 dansmith I don't mean a failure where we can point to rabbit as the sole responsible party, but for something that causes us to not be able to use rabbit for whatever reason
16:06:22 dansmith mriedem: obviously that's the proper solution
16:06:32 mriedem \o/
16:06:47 mriedem now we're cooking with robusto!
16:07:04 sean-k-mooney dansmith: it has to do with how oslo.messaging works. if rabbit dide or was restarted the way we curretly do a retry of sending the message to rabbit will recreate teh connection to the exchange but not the topic queue
16:07:16 sean-k-mooney so when the compute agent reconnect it does not always recreate it queue
16:07:29 sean-k-mooney you can fix it by restarting the compute agent
16:07:47 dansmith sean-k-mooney: I'm all for fixing bugs in oslo.messaging, but that has nothing to do with us trying to hack around a stuck instance state
16:07:56 sean-k-mooney the mandaotry flag woudl allow the conductor to detect that the queue was not present and we could make the compute node as down
16:08:22 dansmith jesus lord, pelase no
16:08:24 sean-k-mooney dansmith: https://bugs.launchpad.net/oslo.messaging/+bug/1661510
16:08:24 openstack Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio)
16:08:47 sean-k-mooney dansmith: well we dont have to mark it as down but it would be nice to at least log the fact
16:08:47 dansmith rabbit has split brain, one conductor thinks the compute queue is gone, and starts marking computes down?
16:09:12 sean-k-mooney ya marking as down is proably an over reaction
16:09:19 dansmith log what we know, not what we guess.. so sure, "hmm, seems like there should be a compute queue here..."
16:11:06 sean-k-mooney anyway the oslo change have merged but we are not using them in nova yet
16:27:59 mriedem oof https://review.opendev.org/#/c/697694/ ruins my life
16:29:29 gibi there is no progress without pain ;)
16:30:00 mriedem yeah yeah
16:30:27 gibi I will feel the pain when I have to backport functional tests
16:30:40 mriedem this is the cross cell thing, so no pain for you there
16:30:49 mriedem but i've got a lot of patches to sift through and update
16:31:10 gibi yeah I figured out that you now have a nice big merge conflict
16:31:13 efried dansmith: we finally got a grenade fail here https://review.opendev.org/692402
16:31:18 efried It was my magic touch
16:31:43 mriedem gibi: the merge conflict is trivial, it's the behavior/interface change
16:31:43 efried (hopefully it was for the issue we care about)
16:33:59 melwitt sean-k-mooney: I read the backscroll about the rabbit thing and I'm wondering, was there not a way that oslo.messaging could have recreated the missing queue? I wonder why restarting our service is the only way?
16:34:54 mriedem gibi: after https://review.opendev.org/#/c/695905/ isn't there supposed to be a patch to enable live migration with qos ports in the api?
16:35:01 mriedem with a release note and docs update and such?
16:35:36 mriedem oh nvm i see, "subsequent patches will add support for migration with target host and other edge case like reschedule."
16:35:38 gibi mriedem: that patch only supporst the happy path. I'm working on the re-schedule cases
16:35:43 mriedem ack
16:35:46 gibi mriedem: yeah
16:35:51 gibi stay tuned :)
16:36:18 sean-k-mooney melwitt: it proably could although we coudl also proably try to do it if we got the error that it was not deliverable
16:36:58 sean-k-mooney melwitt: i think they would prefer if we as the client of oslo.messaging did that since they dont want to assume it is correct
16:37:55 dansmith efried: okay, did you look for any juiciness?
16:38:06 dansmith efried: I can try to look through there later, but I barely even remember what the deal was

Earlier   Later