Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-12
16:02:51 sean-k-mooney did i look at this last week. i this for blizzard?
16:02:58 mriedem they could have an external background service/script/whatever that polls the servers that are building and if they pass some timeout they just reset to error or nuke them and try again
16:03:01 mriedem yes
16:03:23 sean-k-mooney ok so we were talking about the mandatory flag for rabbitmq
16:03:30 mriedem if rabbit is dead b/c it can't keep up though, constantly hammering it is probably not the solution
16:04:40 sean-k-mooney right so the mandatory flag woudl raise an error on our side if we could not enqueue the packet which woudl allows us to detect the failure and then either retry with a different host if rabbit was up but the compute node queue was not found or error
16:04:41 dansmith trying to work around dead rabbit is like saying nova should be able to do something smart if the db is down
16:05:21 sean-k-mooney dansmith: right but there are cases where rabbit is fine but the compute agent topic queue is not present
16:05:43 dansmith sean-k-mooney: how is that not a rabbit failure?
16:05:59 mriedem dansmith: hear me out, if the db is down, auto switch to etcd and then when the db is back up, just like merge all the etcd stuff back into mysql
16:06:14 dansmith I don't mean a failure where we can point to rabbit as the sole responsible party, but for something that causes us to not be able to use rabbit for whatever reason
16:06:22 dansmith mriedem: obviously that's the proper solution
16:06:32 mriedem \o/
16:06:47 mriedem now we're cooking with robusto!
16:07:04 sean-k-mooney dansmith: it has to do with how oslo.messaging works. if rabbit dide or was restarted the way we curretly do a retry of sending the message to rabbit will recreate teh connection to the exchange but not the topic queue
16:07:16 sean-k-mooney so when the compute agent reconnect it does not always recreate it queue
16:07:29 sean-k-mooney you can fix it by restarting the compute agent
16:07:47 dansmith sean-k-mooney: I'm all for fixing bugs in oslo.messaging, but that has nothing to do with us trying to hack around a stuck instance state
16:07:56 sean-k-mooney the mandaotry flag woudl allow the conductor to detect that the queue was not present and we could make the compute node as down
16:08:22 dansmith jesus lord, pelase no
16:08:24 sean-k-mooney dansmith: https://bugs.launchpad.net/oslo.messaging/+bug/1661510
16:08:24 openstack Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio)
16:08:47 sean-k-mooney dansmith: well we dont have to mark it as down but it would be nice to at least log the fact
16:08:47 dansmith rabbit has split brain, one conductor thinks the compute queue is gone, and starts marking computes down?
16:09:12 sean-k-mooney ya marking as down is proably an over reaction
16:09:19 dansmith log what we know, not what we guess.. so sure, "hmm, seems like there should be a compute queue here..."
16:11:06 sean-k-mooney anyway the oslo change have merged but we are not using them in nova yet
16:27:59 mriedem oof https://review.opendev.org/#/c/697694/ ruins my life
16:29:29 gibi there is no progress without pain ;)
16:30:00 mriedem yeah yeah
16:30:27 gibi I will feel the pain when I have to backport functional tests
16:30:40 mriedem this is the cross cell thing, so no pain for you there
16:30:49 mriedem but i've got a lot of patches to sift through and update
16:31:10 gibi yeah I figured out that you now have a nice big merge conflict
16:31:13 efried dansmith: we finally got a grenade fail here https://review.opendev.org/692402
16:31:18 efried It was my magic touch
16:31:43 mriedem gibi: the merge conflict is trivial, it's the behavior/interface change
16:31:43 efried (hopefully it was for the issue we care about)
16:33:59 melwitt sean-k-mooney: I read the backscroll about the rabbit thing and I'm wondering, was there not a way that oslo.messaging could have recreated the missing queue? I wonder why restarting our service is the only way?
16:34:54 mriedem gibi: after https://review.opendev.org/#/c/695905/ isn't there supposed to be a patch to enable live migration with qos ports in the api?
16:35:01 mriedem with a release note and docs update and such?
16:35:36 mriedem oh nvm i see, "subsequent patches will add support for migration with target host and other edge case like reschedule."
16:35:38 gibi mriedem: that patch only supporst the happy path. I'm working on the re-schedule cases
16:35:43 mriedem ack
16:35:46 gibi mriedem: yeah
16:35:51 gibi stay tuned :)
16:36:18 sean-k-mooney melwitt: it proably could although we coudl also proably try to do it if we got the error that it was not deliverable
16:36:58 sean-k-mooney melwitt: i think they would prefer if we as the client of oslo.messaging did that since they dont want to assume it is correct
16:37:55 dansmith efried: okay, did you look for any juiciness?
16:38:06 dansmith efried: I can try to look through there later, but I barely even remember what the deal was
16:38:32 efried dansmith: I didn't look at all, just continue to be very interested in making that f'in bug go away.
16:38:51 efried but I never really understood the original issue, so not sure it would do anybody much good for me to dig.
16:52:17 openstackgerrit Eric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804
16:53:06 efried stephenfin, gibi: That's ready for your (hopefully final) nod ^
16:53:15 gibi efried: ack
16:53:19 stephenfin cool
16:53:21 efried This iteration is simpler in terms of weirdnesses
16:53:33 efried and fleshes out some of the interplay necessary for object store
16:53:39 efried jroll: ^
16:53:59 efried TxGirlGeek: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804
17:00:58 TxGirlGeek @efried Check
17:13:18 mriedem stephenfin: so your change to _build_minimal_create_server_request
17:13:24 mriedem to remove the API that's passed in,
17:13:33 mriedem that's a problem for any testing doing multi-tenant testing
17:13:38 mriedem like https://review.opendev.org/#/c/697331/4/nova/tests/functional/test_scheduler.py
17:13:44 mriedem *any test
17:14:14 mriedem so to avoid temporarily mutating the self.api in these types of tests we're going to have to add an api kwarg or something
17:14:33 stephenfin I thought I did, no?
17:14:38 stephenfin looking
17:15:17 mriedem nope https://review.opendev.org/#/c/697694/1/nova/tests/functional/integrated_helpers.py@a98
17:16:20 stephenfin okay, that's a good point. Can you -1 that and I'll respin with the api parameter
17:16:20 stephenfin I had it for some of the helpers to deal with that but clearly missed this one
17:16:20 stephenfin *missed some
17:17:13 mriedem same https://review.opendev.org/#/c/694179/2/nova/tests/functional/test_external_networks.py
17:17:26 mriedem your change is merged,
17:17:29 mriedem which is why i'm in rebase hell
17:18:04 mriedem i'll add it when i move onto reworking one of these
17:18:16 mriedem https://review.opendev.org/#/q/path:%255Enova/tests/functional/.*+status:open+owner:self+branch:master :(
17:18:24 mriedem i guess that won't work for others...
17:18:35 mriedem there we go https://review.opendev.org/#/q/path:%255Enova/tests/functional/.*+status:open+owner:mriedem+branch:master
17:19:16 stephenfin /o\ sorry
17:19:20 stephenfin you're gonna hate https://review.opendev.org/#/c/697537/ so
17:19:29 stephenfin I can respin those for you, if you want
17:19:32 stephenfin seeing as I broke em
17:19:39 mriedem i'll do it
17:19:52 mriedem i realize to make an omelette you have to break a few eggs
17:20:06 mriedem but these are my faberge eggs
17:26:54 pmatulis how do i know what command will provide a known attribute, such as 'OS-EXT-SRV-ATTR:kernel_id'? the 'server show' command does not show this
17:27:53 mriedem you're likely not using a high enough microversion
17:28:04 mriedem you need >= 2.3 for that
17:28:12 mriedem openstack --os-compute-api-version 2.3 server show <server>
17:28:20 mriedem https://docs.openstack.org/api-ref/compute/?expanded=show-server-details-detail#show-server-details
17:28:32 mriedem tells you the response params and required min versions
17:30:09 pmatulis mriedem, wow ok, i've never encountered that option before. thank you
17:30:49 melwitt TheJulia: sorry, what did you mean about the cpu_arch? it
17:32:56 melwitt TheJulia: *it's the opposite, we take the cpu_arch from the ironic node properties to use as an advertised capability of a compute, for scheduling. what is the correct thing for us to do if an ironic node has _no_ cpu_arch in it's node properties? today we are advertising no cpu_arch in that case, which means any glance image with arch specified will not match an arch-less host
17:34:05 sean-k-mooney melwitt: do you need a filter to get that behavior?
17:34:26 melwitt sean-k-mooney: yeah it's the ImagePropertiesFilter
17:34:42 sean-k-mooney oh i assumeed the computecapablitys filter
17:34:43 melwitt which is enabled by default

Earlier   Later