| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 15:03:17 | dansmith | but the only entity that could do that already owns the thing | |
| 15:03:24 | mriedem | note: that doesn't block the server delete, | |
| 15:03:26 | dansmith | so, who cares? | |
| 15:03:34 | mriedem | and we already fail to delete the volume if it has snapshots | |
| 15:03:40 | mriedem | so yeah, this is like the lowest of priorities | |
| 15:04:01 | dansmith | or you could say it's working as designed and thus not a bug, which would be my preference | |
| 15:04:12 | tommylikehu | lol | |
| 15:04:39 | mriedem | that works for me | |
| 15:04:48 | mriedem | as i said above, "(9:52:21 AM) mriedem: tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?" | |
| 15:05:11 | dansmith | this would be like complaining that something that is able to hardlink the image file on disk before nova deletes it can still read the data | |
| 15:05:20 | dansmith | and the response to that is "yup. that's how that works" | |
| 15:05:46 | dansmith | I mean, just MHO of course | |
| 15:06:45 | mriedem | we have insanely bigger fish to fry so yes | |
| 15:06:50 | mriedem | like the port thing i already mentioned | |
| 15:07:33 | dansmith | how much bigger is insanely bigger? | |
| 15:07:40 | dansmith | like bigger^2? | |
| 15:10:30 | mriedem | hyperbole sized bigger | |
| 15:19:10 | dansmith | tssurya: mriedem: don't we want this to be under the big stack of down-cell patches so we can merge and backport it? https://review.openstack.org/#/c/592428/2 | |
| 15:20:17 | dansmith | oh sorry I guess it is | |
| 15:20:18 | dansmith | nevermind | |
| 15:24:54 | openstackgerrit | Merged openstack/nova stable/ocata: Default embedded instance.flavor.disabled attribute https://review.openstack.org/580525 | |
| 15:27:28 | dansmith | melwitt: if you want to hit this last patch in my series, it'll make the down cell stuff soon able to be based on master: https://review.openstack.org/#/c/594577/11 | |
| 15:33:33 | sean-k-mooney | dansmith: Tahvok was asking about periodic spike in nova-conductor cpu usage in a cloud of ~160 compute nodes eairler. bejond the periodic jobs i was not sure what would be likely to cause the condoctor to be processing ~100 rpc messages a second. any toughts? | |
| 15:33:58 | dansmith | sean-k-mooney: sounds like a support (not dev) question | |
| 15:34:27 | dansmith | sean-k-mooney: but yeah, conductor pretty much just answers to nova-compute on an idle cloud, so it'd be periodics from compute nodes | |
| 15:34:54 | sean-k-mooney | dansmith: perhapes but do we expect 18 nova-condoctor worker treads to spike to 100% usage every 1-2 seconds http://paste.openstack.org/show/728925/ | |
| 15:35:43 | dansmith | we should totally call it condoctor | |
| 15:35:59 | sean-k-mooney | :) | |
| 15:37:09 | Tahvok | lol | |
| 15:37:13 | dansmith | I assume that was a rhetorical question, but 160 computes configured to run some periodics every minute could certainly generate a fair bit of traffic | |
| 15:37:39 | sean-k-mooney | i was wondering if we should consider introducing intetional jitter in the perodic jobs to maybe spread when the jobs are running on each node | |
| 15:37:50 | dansmith | that is already done | |
| 15:37:52 | Tahvok | dansmith: but 100 messages a second? Don't they sync every minute or so? That should generate around 2.5 message a second.. Not a 100.. | |
| 15:38:04 | openstackgerrit | Merged openstack/nova master: Make monkey patch work in uWSGI mode https://review.openstack.org/592285 | |
| 15:38:27 | dansmith | Tahvok: they sync as often as you have them configured for.. without knowing what the messages are, who is sending them, etc, it's hard to say what the problem is | |
| 15:40:03 | diliprenkila | Hi all, I keep getting these errors on my compute nodes "ERROR oslo_service.service [req-0c500027-fc8f-4a24-b1c1-9714b3f248e6 - - - - -] Error starting thread.: AttributeError: '_TransactionContextManager' object has no attribute 'async_'" | |
| 15:40:06 | Tahvok | dansmith: I have tried to debug the rpc calls.. I've tried setting default_log_levels = oslo,messaging=DEBUG, and got lots of messages like this: | |
| 15:40:07 | Tahvok | And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout | |
| 15:40:42 | Tahvok | No idea what this unknowns are... | |
| 15:40:54 | dansmith | Tahvok: I have no idea what would cause that kind of debug with unknown calls and topics | |
| 15:41:07 | dansmith | Tahvok: I would strongly suspect something is majorly broken with your setup | |
| 15:41:20 | Tahvok | dansmith: almost all the messages were followed with this unknown message | |
| 15:41:31 | sean-k-mooney | Tahvok: didnt you say the cloud was fuctionlaly and able to boot vms? | |
| 15:41:40 | Tahvok | sean-k-mooney: yep, everything works | |
| 15:42:06 | dansmith | timeouts generally come from overwhelmed services, rabbit or conductor or db, but I've never seen unknown timeouts like that | |
| 15:42:07 | Tahvok | The reason we got to conductor, is because we are trying to investigate a high cpu usage from rabbit service | |
| 15:42:30 | dansmith | which makes me wonder if there's some garbage on the bus, or some messages being echoed because of bad HA or something weird like that | |
| 15:42:38 | diliprenkila | ||
| 15:42:45 | sean-k-mooney | Tahvok: well rabbit is usually the first thing to melt as you scale out | |
| 15:43:45 | Tahvok | sean-k-mooney: I know that, but it's working completely fine apart from the high cpu usage (around 100%~400% on 32 core machine) .. Everything is green | |
| 15:43:50 | dansmith | Tahvok: conductor does not send messages except as replies, so if you think some service is generating load on rabbit, it'd be something else | |
| 15:44:23 | dansmith | based on what you've said, I would not suspect a nova bug, but a misconfiguration or something else acting up | |
| 15:44:31 | Tahvok | dansmith: so it's trying to reply to some unknown messages? Is it only computes it's talking to? | |
| 15:44:33 | stephenfin | diliprenkila: You've got a package version mismatch | |
| 15:44:54 | stephenfin | diliprenkila: Also, for usage questions like that, please use #openstack | |
| 15:44:55 | dansmith | Tahvok: no I don't think garbage will cause it to try to reply to things with "unknown" like that | |
| 15:45:00 | diliprenkila | <stephenfin> How did u find that ? | |
| 15:45:24 | dansmith | Tahvok: the only nova-related wrinkle is that if you have something hammering rabbit causing some timeouts between compute and conductor, the retries on compute may exacerbate the problem, causing more load on conductor as the backlog grows | |
| 15:45:57 | dansmith | Tahvok: if you think there's a nova bug you should file a bug with complete logs (not just single lines like you have provided here) and someone can look, but like I said, I suspect something non-nova as the root cause | |
| 15:47:26 | stephenfin | diliprenkila: https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7 | |
| 15:49:53 | Tahvok | dansmith: I have sorted 500 messages from the log, and all replies had different id's, which made me think it's conductor itself doing some stuff, and not simply retrying to answer to same timedout calls | |
| 15:50:18 | sean-k-mooney | Tahvok: one think you could try is deploying a seperate rabitmq instance for nova. that would help you isolate the issue. that said i know you may not want to do that on a running cloud | |
| 15:50:30 | mriedem | Tahvok: what versions of oslo.db and oslo.messaging are you using? | |
| 15:50:44 | mriedem | "AttributeError: '_TransactionContextManager' object has no attribute 'async_'"" suggests you're using an old oslo.db | |
| 15:50:48 | sean-k-mooney | mriedem: Tahvok mentioned it was an ocata cloud this morning | |
| 15:51:05 | mriedem | that doesn't tell me what i'd need to konw | |
| 15:51:12 | mriedem | you can be using min or max versions of oslo.db from ocata | |
| 15:51:13 | Tahvok | mriedem: sec, looking | |
| 15:51:15 | mriedem | or something completely different | |
| 15:51:36 | stephenfin | mriedem: diliprenkila had the same issue, unless you're mixing them up | |
| 15:51:37 | dansmith | mriedem: are you confusing Tahvok and diliprenkila ? | |
| 15:51:45 | stephenfin | dansmith: Yeah :) | |
| 15:52:07 | mriedem | yeah sorry | |
| 15:52:20 | stephenfin | mriedem: I think we've an issue there though. The patch I linked was released in 4.40.0, but we pin on a lower version https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7 | |
| 15:52:36 | stephenfin | I assume we should be handling the older version of bumping our minimum | |
| 15:52:56 | Tahvok | All versions of nova-conductor container: http://paste.openstack.org/show/728972/ | |
| 15:52:57 | stephenfin | ...and I also assume we don't see this in tests because we don't run a lower-constraints functional test | |
| 15:53:24 | sean-k-mooney | stephenfin: well is it on master or stable. we cant bump stable miniums | |
| 15:53:56 | stephenfin | sean-k-mooney: Hmm, lemme check | |
| 15:54:31 | stephenfin | sean-k-mooney: Master (git branch --contains 2d532963fa2e013e16cc403f2674a4488c4170ab) | |
| 15:55:18 | Tahvok | sean-k-mooney: I think deploying a separate cloud would not replicate the problem (I simply don't have 160 compute hosts sitting around) | |
| 15:55:19 | sean-k-mooney | stephenfin: i ment the issue you thing bumping the min verion would fix | |
| 15:55:35 | stephenfin | diliprenkila: Looks like that's a bug. Want to open one with that log and I'll fix that quickly? | |
| 15:55:47 | stephenfin | sean-k-mooney: Yeah, that's what I'm referring to | |
| 15:56:12 | sean-k-mooney | Tahvok: i was not suggesting deploying a seperate cloud. just 1 more rabbitmq node and pointing your existing nodes to use it. | |
| 15:56:29 | mriedem | stephenfin: sorry, is diliprenkila on master? | |
| 15:56:32 | diliprenkila | <mriedem> diliprenkila oslo.db==4.40.0 ,oslo.messaging==8.1.0 on the compute nodes and oslo.db==4.38.0, oslo.messaging==6.4.1 on the nova controller node | |
| 15:57:18 | sean-k-mooney | diliprenkila: so you are running an older controler then compute node? | |
| 15:57:20 | openstackgerrit | Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170 | |
| 15:57:54 | diliprenkila | ||
| 15:57:55 | openstack | Launchpad bug 1788833 in OpenStack Compute (nova) "Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_" [Undecided,New] | |
| 15:57:59 | Tahvok | sean-k-mooney: yeah, doing that on production env is a bit risky, at least we would need lots of time to prepare such change.. I was hoping there is some better way to check what conductor was replying to | |
| 15:58:14 | jaypipes | dansmith, kosamara: pls review https://review.openstack.org/597170 | |
| 15:58:29 | stephenfin | sean-k-mooney: This is the issue https://github.com/openstack/nova/commit/2d532963fa2e013e16cc403f2674a4488c4170ab#diff-8fec546e4c39f78d233f8e21dadaa3ff | |
| 15:58:51 | sean-k-mooney | Tahvok: am you could maybe dump the contence of the message queues from rabbitmq but other then that im not sure | |
| 15:59:01 | stephenfin | sean-k-mooney: We want that change but only when oslo.db>=4.40.0 | |
| 15:59:06 | diliprenkila | <sean-k-mooney> Yes i am using old ones, but i did installed all nova packages from ubuntu cloud archive rocky | |
| 15:59:28 | dansmith | Tahvok: you probably need to sniff the bus or something like that | |
| 15:59:46 | dansmith | Tahvok: or talk to oslo.messaging people about how that "unknown" thing can even happen | |
| 15:59:53 | Tahvok | I was thinking maybe simply stopping all conductor services, and then looking which queue was accumulating the most? | |