Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-12
15:42:18 mriedem it's all very stable
15:43:57 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Revert "docs: Highlight the current broken state of SEV" (partially) https://review.opendev.org/698746
15:49:47 mriedem TheJulia: while you're about, i've marked this as incomplete but does it make any sense to you? https://bugs.launchpad.net/nova/+bug/1853389
15:49:47 openstack Launchpad bug 1853389 in OpenStack Compute (nova) "The ipv6 address in network_data.json within configdrive is wrong when using IronicDriver to deploy BM" [Undecided,Incomplete] - Assigned to Eric Lei (leiyashuai)
15:51:04 mriedem ok i guess they call https://docs.openstack.org/api-ref/baremetal/#attach-a-vif-to-a-node while creating a server
15:51:13 mriedem i'd think ironic would disallow attaching a vif while a node is being provisioned
15:51:26 mriedem oh i can't add ironic to this....storyboard
15:58:04 mriedem eandersson: i've left some comments in https://bugs.launchpad.net/nova/+bug/1854992 but marked it incomplete
15:58:04 openstack Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,Incomplete]
15:58:22 mriedem it's not clear to me what you think nova could do besides changing rpc casts to calls and i'm not sure what that would help if rabbit is dead
15:58:49 mriedem e.g. let's say rabbit is f'ed and the compute gets an error back, it needs to use rabbit to update the instance status to ERROR, so you have a chicken and egg problem
15:59:12 mriedem if it's conductor that notices it has direct access to the db so then we could update, but i don't think we're going to be changing the build_and_run_instances cast to a call...
15:59:19 mriedem dansmith: maybe you have ideas ^
16:00:35 mriedem conductor could grow a threaded poll per instance that's building or something with a hard timeout? idk
16:00:52 dansmith conductor should definitely not imply some instance state like that
16:01:09 dansmith if rabbit is effed then all bets are off
16:01:16 sean-k-mooney mriedem: woudl a reset-state command not just update the statis in the db. i did not think that need to do an rpc to set the instance state
16:01:21 mriedem right any attempts here seem like hacks
16:01:33 mriedem reset state from the api is direct to the db
16:01:45 mriedem so yeah you can reset the state to error or just delete the instance
16:01:55 mriedem but they want the instances to automatically go to error when rabbit is down
16:01:59 sean-k-mooney ya so woudl that not bypass your chicken and egg problem
16:02:00 mriedem or at least have nova log something
16:02:17 sean-k-mooney oh right
16:02:25 mriedem sean-k-mooney: the admin would still have to manually do the needful
16:02:51 sean-k-mooney did i look at this last week. i this for blizzard?
16:02:58 mriedem they could have an external background service/script/whatever that polls the servers that are building and if they pass some timeout they just reset to error or nuke them and try again
16:03:01 mriedem yes
16:03:23 sean-k-mooney ok so we were talking about the mandatory flag for rabbitmq
16:03:30 mriedem if rabbit is dead b/c it can't keep up though, constantly hammering it is probably not the solution
16:04:40 sean-k-mooney right so the mandatory flag woudl raise an error on our side if we could not enqueue the packet which woudl allows us to detect the failure and then either retry with a different host if rabbit was up but the compute node queue was not found or error
16:04:41 dansmith trying to work around dead rabbit is like saying nova should be able to do something smart if the db is down
16:05:21 sean-k-mooney dansmith: right but there are cases where rabbit is fine but the compute agent topic queue is not present
16:05:43 dansmith sean-k-mooney: how is that not a rabbit failure?
16:05:59 mriedem dansmith: hear me out, if the db is down, auto switch to etcd and then when the db is back up, just like merge all the etcd stuff back into mysql
16:06:14 dansmith I don't mean a failure where we can point to rabbit as the sole responsible party, but for something that causes us to not be able to use rabbit for whatever reason
16:06:22 dansmith mriedem: obviously that's the proper solution
16:06:32 mriedem \o/
16:06:47 mriedem now we're cooking with robusto!
16:07:04 sean-k-mooney dansmith: it has to do with how oslo.messaging works. if rabbit dide or was restarted the way we curretly do a retry of sending the message to rabbit will recreate teh connection to the exchange but not the topic queue
16:07:16 sean-k-mooney so when the compute agent reconnect it does not always recreate it queue
16:07:29 sean-k-mooney you can fix it by restarting the compute agent
16:07:47 dansmith sean-k-mooney: I'm all for fixing bugs in oslo.messaging, but that has nothing to do with us trying to hack around a stuck instance state
16:07:56 sean-k-mooney the mandaotry flag woudl allow the conductor to detect that the queue was not present and we could make the compute node as down
16:08:22 dansmith jesus lord, pelase no
16:08:24 sean-k-mooney dansmith: https://bugs.launchpad.net/oslo.messaging/+bug/1661510
16:08:24 openstack Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio)
16:08:47 sean-k-mooney dansmith: well we dont have to mark it as down but it would be nice to at least log the fact
16:08:47 dansmith rabbit has split brain, one conductor thinks the compute queue is gone, and starts marking computes down?
16:09:12 sean-k-mooney ya marking as down is proably an over reaction
16:09:19 dansmith log what we know, not what we guess.. so sure, "hmm, seems like there should be a compute queue here..."
16:11:06 sean-k-mooney anyway the oslo change have merged but we are not using them in nova yet
16:27:59 mriedem oof https://review.opendev.org/#/c/697694/ ruins my life
16:29:29 gibi there is no progress without pain ;)
16:30:00 mriedem yeah yeah
16:30:27 gibi I will feel the pain when I have to backport functional tests
16:30:40 mriedem this is the cross cell thing, so no pain for you there
16:30:49 mriedem but i've got a lot of patches to sift through and update
16:31:10 gibi yeah I figured out that you now have a nice big merge conflict
16:31:13 efried dansmith: we finally got a grenade fail here https://review.opendev.org/692402
16:31:18 efried It was my magic touch
16:31:43 mriedem gibi: the merge conflict is trivial, it's the behavior/interface change
16:31:43 efried (hopefully it was for the issue we care about)
16:33:59 melwitt sean-k-mooney: I read the backscroll about the rabbit thing and I'm wondering, was there not a way that oslo.messaging could have recreated the missing queue? I wonder why restarting our service is the only way?
16:34:54 mriedem gibi: after https://review.opendev.org/#/c/695905/ isn't there supposed to be a patch to enable live migration with qos ports in the api?
16:35:01 mriedem with a release note and docs update and such?
16:35:36 mriedem oh nvm i see, "subsequent patches will add support for migration with target host and other edge case like reschedule."
16:35:38 gibi mriedem: that patch only supporst the happy path. I'm working on the re-schedule cases
16:35:43 mriedem ack
16:35:46 gibi mriedem: yeah
16:35:51 gibi stay tuned :)
16:36:18 sean-k-mooney melwitt: it proably could although we coudl also proably try to do it if we got the error that it was not deliverable
16:36:58 sean-k-mooney melwitt: i think they would prefer if we as the client of oslo.messaging did that since they dont want to assume it is correct
16:37:55 dansmith efried: okay, did you look for any juiciness?
16:38:06 dansmith efried: I can try to look through there later, but I barely even remember what the deal was
16:38:32 efried dansmith: I didn't look at all, just continue to be very interested in making that f'in bug go away.
16:38:51 efried but I never really understood the original issue, so not sure it would do anybody much good for me to dig.
16:52:17 openstackgerrit Eric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804
16:53:06 efried stephenfin, gibi: That's ready for your (hopefully final) nod ^
16:53:15 gibi efried: ack
16:53:19 stephenfin cool
16:53:21 efried This iteration is simpler in terms of weirdnesses
16:53:33 efried and fleshes out some of the interplay necessary for object store
16:53:39 efried jroll: ^
16:53:59 efried TxGirlGeek: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804
17:00:58 TxGirlGeek @efried Check
17:13:18 mriedem stephenfin: so your change to _build_minimal_create_server_request
17:13:24 mriedem to remove the API that's passed in,
17:13:33 mriedem that's a problem for any testing doing multi-tenant testing
17:13:38 mriedem like https://review.opendev.org/#/c/697331/4/nova/tests/functional/test_scheduler.py
17:13:44 mriedem *any test
17:14:14 mriedem so to avoid temporarily mutating the self.api in these types of tests we're going to have to add an api kwarg or something
17:14:33 stephenfin I thought I did, no?
17:14:38 stephenfin looking
17:15:17 mriedem nope https://review.opendev.org/#/c/697694/1/nova/tests/functional/integrated_helpers.py@a98
17:16:20 stephenfin okay, that's a good point. Can you -1 that and I'll respin with the api parameter
17:16:20 stephenfin I had it for some of the helpers to deal with that but clearly missed this one
17:16:20 stephenfin *missed some
17:17:13 mriedem same https://review.opendev.org/#/c/694179/2/nova/tests/functional/test_external_networks.py
17:17:26 mriedem your change is merged,

Earlier   Later