Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-28
15:42:30 dansmith which makes me wonder if there's some garbage on the bus, or some messages being echoed because of bad HA or something weird like that
15:42:38 diliprenkila Hi all, I keep getting these errors on my compute nodes "ERROR oslo_service.service [req-0c500027-fc8f-4a24-b1c1-9714b3f248e6 - - - - -] Error starting thread.: AttributeError: '_TransactionContextManager' object has no attribute 'async_'" fuul log is at https://etherpad.openstack.org/p/i2kJvQ4s4o
15:42:45 sean-k-mooney Tahvok: well rabbit is usually the first thing to melt as you scale out
15:43:45 Tahvok sean-k-mooney: I know that, but it's working completely fine apart from the high cpu usage (around 100%~400% on 32 core machine) .. Everything is green
15:43:50 dansmith Tahvok: conductor does not send messages except as replies, so if you think some service is generating load on rabbit, it'd be something else
15:44:23 dansmith based on what you've said, I would not suspect a nova bug, but a misconfiguration or something else acting up
15:44:31 Tahvok dansmith: so it's trying to reply to some unknown messages? Is it only computes it's talking to?
15:44:33 stephenfin diliprenkila: You've got a package version mismatch
15:44:54 stephenfin diliprenkila: Also, for usage questions like that, please use #openstack
15:44:55 dansmith Tahvok: no I don't think garbage will cause it to try to reply to things with "unknown" like that
15:45:00 diliprenkila <stephenfin> How did u find that ?
15:45:24 dansmith Tahvok: the only nova-related wrinkle is that if you have something hammering rabbit causing some timeouts between compute and conductor, the retries on compute may exacerbate the problem, causing more load on conductor as the backlog grows
15:45:57 dansmith Tahvok: if you think there's a nova bug you should file a bug with complete logs (not just single lines like you have provided here) and someone can look, but like I said, I suspect something non-nova as the root cause
15:47:26 stephenfin diliprenkila: https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7
15:49:53 Tahvok dansmith: I have sorted 500 messages from the log, and all replies had different id's, which made me think it's conductor itself doing some stuff, and not simply retrying to answer to same timedout calls
15:50:18 sean-k-mooney Tahvok: one think you could try is deploying a seperate rabitmq instance for nova. that would help you isolate the issue. that said i know you may not want to do that on a running cloud
15:50:30 mriedem Tahvok: what versions of oslo.db and oslo.messaging are you using?
15:50:44 mriedem "AttributeError: '_TransactionContextManager' object has no attribute 'async_'"" suggests you're using an old oslo.db
15:50:48 sean-k-mooney mriedem: Tahvok mentioned it was an ocata cloud this morning
15:51:05 mriedem that doesn't tell me what i'd need to konw
15:51:12 mriedem you can be using min or max versions of oslo.db from ocata
15:51:13 Tahvok mriedem: sec, looking
15:51:15 mriedem or something completely different
15:51:36 stephenfin mriedem: diliprenkila had the same issue, unless you're mixing them up
15:51:37 dansmith mriedem: are you confusing Tahvok and diliprenkila ?
15:51:45 stephenfin dansmith: Yeah :)
15:52:07 mriedem yeah sorry
15:52:20 stephenfin mriedem: I think we've an issue there though. The patch I linked was released in 4.40.0, but we pin on a lower version https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7
15:52:36 stephenfin I assume we should be handling the older version of bumping our minimum
15:52:56 Tahvok All versions of nova-conductor container: http://paste.openstack.org/show/728972/
15:52:57 stephenfin ...and I also assume we don't see this in tests because we don't run a lower-constraints functional test
15:53:24 sean-k-mooney stephenfin: well is it on master or stable. we cant bump stable miniums
15:53:56 stephenfin sean-k-mooney: Hmm, lemme check
15:54:31 stephenfin sean-k-mooney: Master (git branch --contains 2d532963fa2e013e16cc403f2674a4488c4170ab)
15:55:18 Tahvok sean-k-mooney: I think deploying a separate cloud would not replicate the problem (I simply don't have 160 compute hosts sitting around)
15:55:19 sean-k-mooney stephenfin: i ment the issue you thing bumping the min verion would fix
15:55:35 stephenfin diliprenkila: Looks like that's a bug. Want to open one with that log and I'll fix that quickly?
15:55:47 stephenfin sean-k-mooney: Yeah, that's what I'm referring to
15:56:12 sean-k-mooney Tahvok: i was not suggesting deploying a seperate cloud. just 1 more rabbitmq node and pointing your existing nodes to use it.
15:56:29 mriedem stephenfin: sorry, is diliprenkila on master?
15:56:32 diliprenkila <mriedem> diliprenkila oslo.db==4.40.0 ,oslo.messaging==8.1.0 on the compute nodes and oslo.db==4.38.0, oslo.messaging==6.4.1 on the nova controller node
15:57:18 sean-k-mooney diliprenkila: so you are running an older controler then compute node?
15:57:20 openstackgerrit Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170
15:57:54 diliprenkila I have already opened a bug , https://bugs.launchpad.net/nova/+bug/1788833
15:57:55 openstack Launchpad bug 1788833 in OpenStack Compute (nova) "Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_" [Undecided,New]
15:57:59 Tahvok sean-k-mooney: yeah, doing that on production env is a bit risky, at least we would need lots of time to prepare such change.. I was hoping there is some better way to check what conductor was replying to
15:58:14 jaypipes dansmith, kosamara: pls review https://review.openstack.org/597170
15:58:29 stephenfin sean-k-mooney: This is the issue https://github.com/openstack/nova/commit/2d532963fa2e013e16cc403f2674a4488c4170ab#diff-8fec546e4c39f78d233f8e21dadaa3ff
15:58:51 sean-k-mooney Tahvok: am you could maybe dump the contence of the message queues from rabbitmq but other then that im not sure
15:59:01 stephenfin sean-k-mooney: We want that change but only when oslo.db>=4.40.0
15:59:06 diliprenkila <sean-k-mooney> Yes i am using old ones, but i did installed all nova packages from ubuntu cloud archive rocky
15:59:28 dansmith Tahvok: you probably need to sniff the bus or something like that
15:59:46 dansmith Tahvok: or talk to oslo.messaging people about how that "unknown" thing can even happen
15:59:53 Tahvok I was thinking maybe simply stopping all conductor services, and then looking which queue was accumulating the most?
15:59:56 stephenfin diliprenkila: You're using master nova on the broken node though, I imagine
16:00:08 stephenfin Because the code that's broken isn't on stable/rocky, from what I can see
16:00:17 diliprenkila <stephenfin> Yes i am using master
16:00:18 sean-k-mooney stephenfin: chaning from async to async_
16:00:55 efried kosamara: Are you still around?
16:01:10 dansmith Tahvok: the only time conductor initiates a message (not a reply) is during a build or resize type operation, that I can think of
16:01:15 sean-k-mooney stephenfin: it looks like that is in RC3. we cant bump miniums for rocky rc3 at this point
16:01:29 stephenfin sean-k-mooney: sec. We don't need to
16:01:34 dansmith Tahvok: sniffing the bus, you may find that one node is going crazy, spewing messages or something
16:02:01 stephenfin sean-k-mooney: But I don't think it is. 'git branch --contains 2d532963fa2e013e16cc403f2674a4488c4170ab' doesn't show it
16:02:27 Tahvok dansmith: we actually have some compute nodes with errors like: [instance: 631c2697-1cdf-4d97-8ae9-006cc5ed6e35] Instance not resizing, skipping migration.
16:02:43 Tahvok But this happen every sync operation (once a minute)
16:02:55 Tahvok Which is a separate issue we are trying to investigate..
16:03:00 dansmith Tahvok: I mean during the active part of a migration, in response to an api call
16:03:25 Tahvok dansmith: then it's not the case.. As nothing is migrating right now
16:03:33 dansmith Tahvok: right
16:03:35 sean-k-mooney stephenfin: if it was backported the commit id would have changed but the chaing id would still be in the git log. ill check
16:04:41 sean-k-mooney stephenfin: ya its in rc3
16:05:11 sean-k-mooney stephenfin: actully one second
16:05:18 openstack Launchpad bug 1788833 in OpenStack Compute (nova) "Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_" [Undecided,New]
16:05:18 mriedem stephenfin: did i summarize this correctly? https://bugs.launchpad.net/nova/+bug/1788833/comments/1
16:05:47 stephenfin mriedem: Yup
16:06:07 Tahvok Just realized I didn't paste the messages I'm actually seeing that conductor is handling: http://paste.openstack.org/show/nqYc4ghBYpLAFvinYYAp/
16:06:31 openstackgerrit Stephen Finucane proposed openstack/nova master: Don't use '_TransactionContextManager._async' https://review.openstack.org/597173
16:06:48 sean-k-mooney stephenfin: ya its in rc3
16:06:50 stephenfin mriedem, sean-k-mooney: And there's the fix (tl;dr: this can has some kicking left in it)
16:07:32 Tahvok dansmith: all of them are actually replies as you can see
16:07:34 stephenfin sean-k-mooney: How'd you figure that out?
16:08:03 dansmith Tahvok: that doesn't really tell us anything
16:08:06 sean-k-mooney git fetch --tags && git checkout 18.0.0.0rc3
16:08:17 sean-k-mooney then git log and search for change id
16:08:45 Tahvok dansmith: doesn't a reply message indicate it's a sync from compute nodes?
16:08:56 stephenfin sean-k-mooney: Ah, 'git branch -a --contains 2d532963fa2e013e16cc403f2674a4488c4170ab'
16:09:09 stephenfin The '-a' is important. I don't have stable/rocky locally yet
16:09:12 sean-k-mooney stephenfin: also if you look at https://github.com/openstack/nova/commit/2d532963fa2e013e16cc403f2674a4488c4170ab#diff-8fec546e4c39f78d233f8e21dadaa3ff it shows what branches have it
16:09:43 dansmith Tahvok: I don't think those lines are telling you that they're replies, they're telling you what queue will be used for the reply
16:11:01 sean-k-mooney stephenfin: mriedem in anycase https://review.openstack.org/#/c/597173/1 will need to get applied to stable/rocky
16:11:25 mriedem yes i know
16:11:27 mriedem i left a comment in there
16:11:45 mriedem we can't do an rc4 so i guess this is just going to be a known broken issue for anyone not using oslo.db 4.40 which is at least in upper-constraints for stable/rocky
16:12:03 mriedem and i'd think/hope most deployments should be using what's in upper-constraints for dependent libraries since those are the versions we test against
16:13:52 melwitt .
16:14:20 sean-k-mooney mriedem: yeah thats unfortunate but at least as you said upper constratins allows 4.4
16:14:42 openstackgerrit Stephen Finucane proposed openstack/nova master: Don't use '_TransactionContextManager._async' https://review.openstack.org/597173
16:14:44 sean-k-mooney *4.40
16:16:04 efried stephenfin: Getting late for you, you want me to propose the fup?

Earlier   Later