Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-28
15:05:46 dansmith I mean, just MHO of course
15:06:45 mriedem we have insanely bigger fish to fry so yes
15:06:50 mriedem like the port thing i already mentioned
15:07:33 dansmith how much bigger is insanely bigger?
15:07:40 dansmith like bigger^2?
15:10:30 mriedem hyperbole sized bigger
15:19:10 dansmith tssurya: mriedem: don't we want this to be under the big stack of down-cell patches so we can merge and backport it? https://review.openstack.org/#/c/592428/2
15:20:17 dansmith oh sorry I guess it is
15:20:18 dansmith nevermind
15:24:54 openstackgerrit Merged openstack/nova stable/ocata: Default embedded instance.flavor.disabled attribute https://review.openstack.org/580525
15:27:28 dansmith melwitt: if you want to hit this last patch in my series, it'll make the down cell stuff soon able to be based on master: https://review.openstack.org/#/c/594577/11
15:33:33 sean-k-mooney dansmith: Tahvok was asking about periodic spike in nova-conductor cpu usage in a cloud of ~160 compute nodes eairler. bejond the periodic jobs i was not sure what would be likely to cause the condoctor to be processing ~100 rpc messages a second. any toughts?
15:33:58 dansmith sean-k-mooney: sounds like a support (not dev) question
15:34:27 dansmith sean-k-mooney: but yeah, conductor pretty much just answers to nova-compute on an idle cloud, so it'd be periodics from compute nodes
15:34:54 sean-k-mooney dansmith: perhapes but do we expect 18 nova-condoctor worker treads to spike to 100% usage every 1-2 seconds http://paste.openstack.org/show/728925/
15:35:43 dansmith we should totally call it condoctor
15:35:59 sean-k-mooney :)
15:37:09 Tahvok lol
15:37:13 dansmith I assume that was a rhetorical question, but 160 computes configured to run some periodics every minute could certainly generate a fair bit of traffic
15:37:39 sean-k-mooney i was wondering if we should consider introducing intetional jitter in the perodic jobs to maybe spread when the jobs are running on each node
15:37:50 dansmith that is already done
15:37:52 Tahvok dansmith: but 100 messages a second? Don't they sync every minute or so? That should generate around 2.5 message a second.. Not a 100..
15:38:04 openstackgerrit Merged openstack/nova master: Make monkey patch work in uWSGI mode https://review.openstack.org/592285
15:38:27 dansmith Tahvok: they sync as often as you have them configured for.. without knowing what the messages are, who is sending them, etc, it's hard to say what the problem is
15:40:03 diliprenkila Hi all, I keep getting these errors on my compute nodes "ERROR oslo_service.service [req-0c500027-fc8f-4a24-b1c1-9714b3f248e6 - - - - -] Error starting thread.: AttributeError: '_TransactionContextManager' object has no attribute 'async_'"
15:40:06 Tahvok dansmith: I have tried to debug the rpc calls.. I've tried setting default_log_levels = oslo,messaging=DEBUG, and got lots of messages like this:
15:40:07 Tahvok And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout
15:40:42 Tahvok No idea what this unknowns are...
15:40:54 dansmith Tahvok: I have no idea what would cause that kind of debug with unknown calls and topics
15:41:07 dansmith Tahvok: I would strongly suspect something is majorly broken with your setup
15:41:20 Tahvok dansmith: almost all the messages were followed with this unknown message
15:41:31 sean-k-mooney Tahvok: didnt you say the cloud was fuctionlaly and able to boot vms?
15:41:40 Tahvok sean-k-mooney: yep, everything works
15:42:06 dansmith timeouts generally come from overwhelmed services, rabbit or conductor or db, but I've never seen unknown timeouts like that
15:42:07 Tahvok The reason we got to conductor, is because we are trying to investigate a high cpu usage from rabbit service
15:42:30 dansmith which makes me wonder if there's some garbage on the bus, or some messages being echoed because of bad HA or something weird like that
15:42:38 diliprenkila Hi all, I keep getting these errors on my compute nodes "ERROR oslo_service.service [req-0c500027-fc8f-4a24-b1c1-9714b3f248e6 - - - - -] Error starting thread.: AttributeError: '_TransactionContextManager' object has no attribute 'async_'" fuul log is at https://etherpad.openstack.org/p/i2kJvQ4s4o
15:42:45 sean-k-mooney Tahvok: well rabbit is usually the first thing to melt as you scale out
15:43:45 Tahvok sean-k-mooney: I know that, but it's working completely fine apart from the high cpu usage (around 100%~400% on 32 core machine) .. Everything is green
15:43:50 dansmith Tahvok: conductor does not send messages except as replies, so if you think some service is generating load on rabbit, it'd be something else
15:44:23 dansmith based on what you've said, I would not suspect a nova bug, but a misconfiguration or something else acting up
15:44:31 Tahvok dansmith: so it's trying to reply to some unknown messages? Is it only computes it's talking to?
15:44:33 stephenfin diliprenkila: You've got a package version mismatch
15:44:54 stephenfin diliprenkila: Also, for usage questions like that, please use #openstack
15:44:55 dansmith Tahvok: no I don't think garbage will cause it to try to reply to things with "unknown" like that
15:45:00 diliprenkila <stephenfin> How did u find that ?
15:45:24 dansmith Tahvok: the only nova-related wrinkle is that if you have something hammering rabbit causing some timeouts between compute and conductor, the retries on compute may exacerbate the problem, causing more load on conductor as the backlog grows
15:45:57 dansmith Tahvok: if you think there's a nova bug you should file a bug with complete logs (not just single lines like you have provided here) and someone can look, but like I said, I suspect something non-nova as the root cause
15:47:26 stephenfin diliprenkila: https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7
15:49:53 Tahvok dansmith: I have sorted 500 messages from the log, and all replies had different id's, which made me think it's conductor itself doing some stuff, and not simply retrying to answer to same timedout calls
15:50:18 sean-k-mooney Tahvok: one think you could try is deploying a seperate rabitmq instance for nova. that would help you isolate the issue. that said i know you may not want to do that on a running cloud
15:50:30 mriedem Tahvok: what versions of oslo.db and oslo.messaging are you using?
15:50:44 mriedem "AttributeError: '_TransactionContextManager' object has no attribute 'async_'"" suggests you're using an old oslo.db
15:50:48 sean-k-mooney mriedem: Tahvok mentioned it was an ocata cloud this morning
15:51:05 mriedem that doesn't tell me what i'd need to konw
15:51:12 mriedem you can be using min or max versions of oslo.db from ocata
15:51:13 Tahvok mriedem: sec, looking
15:51:15 mriedem or something completely different
15:51:36 stephenfin mriedem: diliprenkila had the same issue, unless you're mixing them up
15:51:37 dansmith mriedem: are you confusing Tahvok and diliprenkila ?
15:51:45 stephenfin dansmith: Yeah :)
15:52:07 mriedem yeah sorry
15:52:20 stephenfin mriedem: I think we've an issue there though. The patch I linked was released in 4.40.0, but we pin on a lower version https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7
15:52:36 stephenfin I assume we should be handling the older version of bumping our minimum
15:52:56 Tahvok All versions of nova-conductor container: http://paste.openstack.org/show/728972/
15:52:57 stephenfin ...and I also assume we don't see this in tests because we don't run a lower-constraints functional test
15:53:24 sean-k-mooney stephenfin: well is it on master or stable. we cant bump stable miniums
15:53:56 stephenfin sean-k-mooney: Hmm, lemme check
15:54:31 stephenfin sean-k-mooney: Master (git branch --contains 2d532963fa2e013e16cc403f2674a4488c4170ab)
15:55:18 Tahvok sean-k-mooney: I think deploying a separate cloud would not replicate the problem (I simply don't have 160 compute hosts sitting around)
15:55:19 sean-k-mooney stephenfin: i ment the issue you thing bumping the min verion would fix
15:55:35 stephenfin diliprenkila: Looks like that's a bug. Want to open one with that log and I'll fix that quickly?
15:55:47 stephenfin sean-k-mooney: Yeah, that's what I'm referring to
15:56:12 sean-k-mooney Tahvok: i was not suggesting deploying a seperate cloud. just 1 more rabbitmq node and pointing your existing nodes to use it.
15:56:29 mriedem stephenfin: sorry, is diliprenkila on master?
15:56:32 diliprenkila <mriedem> diliprenkila oslo.db==4.40.0 ,oslo.messaging==8.1.0 on the compute nodes and oslo.db==4.38.0, oslo.messaging==6.4.1 on the nova controller node
15:57:18 sean-k-mooney diliprenkila: so you are running an older controler then compute node?
15:57:20 openstackgerrit Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170
15:57:54 diliprenkila I have already opened a bug , https://bugs.launchpad.net/nova/+bug/1788833
15:57:55 openstack Launchpad bug 1788833 in OpenStack Compute (nova) "Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_" [Undecided,New]
15:57:59 Tahvok sean-k-mooney: yeah, doing that on production env is a bit risky, at least we would need lots of time to prepare such change.. I was hoping there is some better way to check what conductor was replying to
15:58:14 jaypipes dansmith, kosamara: pls review https://review.openstack.org/597170
15:58:29 stephenfin sean-k-mooney: This is the issue https://github.com/openstack/nova/commit/2d532963fa2e013e16cc403f2674a4488c4170ab#diff-8fec546e4c39f78d233f8e21dadaa3ff
15:58:51 sean-k-mooney Tahvok: am you could maybe dump the contence of the message queues from rabbitmq but other then that im not sure
15:59:01 stephenfin sean-k-mooney: We want that change but only when oslo.db>=4.40.0
15:59:06 diliprenkila <sean-k-mooney> Yes i am using old ones, but i did installed all nova packages from ubuntu cloud archive rocky
15:59:28 dansmith Tahvok: you probably need to sniff the bus or something like that
15:59:46 dansmith Tahvok: or talk to oslo.messaging people about how that "unknown" thing can even happen
15:59:53 Tahvok I was thinking maybe simply stopping all conductor services, and then looking which queue was accumulating the most?
15:59:56 stephenfin diliprenkila: You're using master nova on the broken node though, I imagine
16:00:08 stephenfin Because the code that's broken isn't on stable/rocky, from what I can see
16:00:17 diliprenkila <stephenfin> Yes i am using master
16:00:18 sean-k-mooney stephenfin: chaning from async to async_
16:00:55 efried kosamara: Are you still around?
16:01:10 dansmith Tahvok: the only time conductor initiates a message (not a reply) is during a build or resize type operation, that I can think of
16:01:15 sean-k-mooney stephenfin: it looks like that is in RC3. we cant bump miniums for rocky rc3 at this point
16:01:29 stephenfin sean-k-mooney: sec. We don't need to
16:01:34 dansmith Tahvok: sniffing the bus, you may find that one node is going crazy, spewing messages or something
16:02:01 stephenfin sean-k-mooney: But I don't think it is. 'git branch --contains 2d532963fa2e013e16cc403f2674a4488c4170ab' doesn't show it
16:02:27 Tahvok dansmith: we actually have some compute nodes with errors like: [instance: 631c2697-1cdf-4d97-8ae9-006cc5ed6e35] Instance not resizing, skipping migration.

Earlier   Later