| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 14:52:19 | mriedem | tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah? | |
| 14:52:32 | kosamara | efried: I've also got pending changes on the "done" things. I'll post them tomorrow. | |
| 14:53:16 | efried | kosamara: Okay. How about I just compose content and post it to a pastebin for you to pull in? | |
| 14:53:22 | kosamara | cool | |
| 14:53:31 | mriedem | tommylikehu: unless you're aware of some other way to atomically delete a volume, this is just a known issue. the only thing i can see nova doing is using force_delete with cinder admin creds (if nova is configured for those) to delete the volume while it's attached to the server | |
| 14:54:04 | tommylikehu | mriedem: can we do something to protect this process? | |
| 14:54:23 | tommylikehu | mriedem: something like what we do when attaching volume | |
| 14:54:25 | mriedem | tommylikehu: is this an actual issue someone ran into or they are just doing some kind of audit? | |
| 14:55:04 | mriedem | and what client software is waiting a millisecond to attach a volume while we're deleting it from another server? | |
| 14:55:31 | mriedem | this seems extremely low priority | |
| 14:55:37 | tommylikehu | mriedem: personally I think it's the second case | |
| 14:55:49 | mriedem | ok; never tell them about ports then :) | |
| 14:56:04 | mriedem | because you can attach/detach those to servers out of band all you like | |
| 14:56:48 | mriedem | PUT /v2.0/ports/{port_id} with a new device_id - now it's my port yay! | |
| 14:57:37 | mriedem | tommylikehu: so like i said above, nova could use the force_delete API if we're configured with cinder admin creds, but i'd consider it very low priority | |
| 14:58:18 | tommylikehu | mriedem: oooook, thanks:) | |
| 15:00:30 | mriedem | tommylikehu: i guess start by reporting a bug to nova | |
| 15:00:33 | mriedem | so it's on the books | |
| 15:01:27 | dansmith | mriedem: tommylikehu the concern is that a volume nova is going to delete becomes available for a second before being deleted? | |
| 15:02:18 | tommylikehu | dansmith: yes | |
| 15:02:41 | mriedem | trump could get his grubby hands all of ma volumes | |
| 15:02:47 | mriedem | *all over | |
| 15:02:55 | dansmith | and what, something could attach those and block the delete? | |
| 15:03:02 | mriedem | yeah i guess | |
| 15:03:17 | dansmith | but the only entity that could do that already owns the thing | |
| 15:03:24 | mriedem | note: that doesn't block the server delete, | |
| 15:03:26 | dansmith | so, who cares? | |
| 15:03:34 | mriedem | and we already fail to delete the volume if it has snapshots | |
| 15:03:40 | mriedem | so yeah, this is like the lowest of priorities | |
| 15:04:01 | dansmith | or you could say it's working as designed and thus not a bug, which would be my preference | |
| 15:04:12 | tommylikehu | lol | |
| 15:04:39 | mriedem | that works for me | |
| 15:04:48 | mriedem | as i said above, "(9:52:21 AM) mriedem: tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?" | |
| 15:05:11 | dansmith | this would be like complaining that something that is able to hardlink the image file on disk before nova deletes it can still read the data | |
| 15:05:20 | dansmith | and the response to that is "yup. that's how that works" | |
| 15:05:46 | dansmith | I mean, just MHO of course | |
| 15:06:45 | mriedem | we have insanely bigger fish to fry so yes | |
| 15:06:50 | mriedem | like the port thing i already mentioned | |
| 15:07:33 | dansmith | how much bigger is insanely bigger? | |
| 15:07:40 | dansmith | like bigger^2? | |
| 15:10:30 | mriedem | hyperbole sized bigger | |
| 15:19:10 | dansmith | tssurya: mriedem: don't we want this to be under the big stack of down-cell patches so we can merge and backport it? https://review.openstack.org/#/c/592428/2 | |
| 15:20:17 | dansmith | oh sorry I guess it is | |
| 15:20:18 | dansmith | nevermind | |
| 15:24:54 | openstackgerrit | Merged openstack/nova stable/ocata: Default embedded instance.flavor.disabled attribute https://review.openstack.org/580525 | |
| 15:27:28 | dansmith | melwitt: if you want to hit this last patch in my series, it'll make the down cell stuff soon able to be based on master: https://review.openstack.org/#/c/594577/11 | |
| 15:33:33 | sean-k-mooney | dansmith: Tahvok was asking about periodic spike in nova-conductor cpu usage in a cloud of ~160 compute nodes eairler. bejond the periodic jobs i was not sure what would be likely to cause the condoctor to be processing ~100 rpc messages a second. any toughts? | |
| 15:33:58 | dansmith | sean-k-mooney: sounds like a support (not dev) question | |
| 15:34:27 | dansmith | sean-k-mooney: but yeah, conductor pretty much just answers to nova-compute on an idle cloud, so it'd be periodics from compute nodes | |
| 15:34:54 | sean-k-mooney | dansmith: perhapes but do we expect 18 nova-condoctor worker treads to spike to 100% usage every 1-2 seconds http://paste.openstack.org/show/728925/ | |
| 15:35:43 | dansmith | we should totally call it condoctor | |
| 15:35:59 | sean-k-mooney | :) | |
| 15:37:09 | Tahvok | lol | |
| 15:37:13 | dansmith | I assume that was a rhetorical question, but 160 computes configured to run some periodics every minute could certainly generate a fair bit of traffic | |
| 15:37:39 | sean-k-mooney | i was wondering if we should consider introducing intetional jitter in the perodic jobs to maybe spread when the jobs are running on each node | |
| 15:37:50 | dansmith | that is already done | |
| 15:37:52 | Tahvok | dansmith: but 100 messages a second? Don't they sync every minute or so? That should generate around 2.5 message a second.. Not a 100.. | |
| 15:38:04 | openstackgerrit | Merged openstack/nova master: Make monkey patch work in uWSGI mode https://review.openstack.org/592285 | |
| 15:38:27 | dansmith | Tahvok: they sync as often as you have them configured for.. without knowing what the messages are, who is sending them, etc, it's hard to say what the problem is | |
| 15:40:03 | diliprenkila | Hi all, I keep getting these errors on my compute nodes "ERROR oslo_service.service [req-0c500027-fc8f-4a24-b1c1-9714b3f248e6 - - - - -] Error starting thread.: AttributeError: '_TransactionContextManager' object has no attribute 'async_'" | |
| 15:40:06 | Tahvok | dansmith: I have tried to debug the rpc calls.. I've tried setting default_log_levels = oslo,messaging=DEBUG, and got lots of messages like this: | |
| 15:40:07 | Tahvok | And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout | |
| 15:40:42 | Tahvok | No idea what this unknowns are... | |
| 15:40:54 | dansmith | Tahvok: I have no idea what would cause that kind of debug with unknown calls and topics | |
| 15:41:07 | dansmith | Tahvok: I would strongly suspect something is majorly broken with your setup | |
| 15:41:20 | Tahvok | dansmith: almost all the messages were followed with this unknown message | |
| 15:41:31 | sean-k-mooney | Tahvok: didnt you say the cloud was fuctionlaly and able to boot vms? | |
| 15:41:40 | Tahvok | sean-k-mooney: yep, everything works | |
| 15:42:06 | dansmith | timeouts generally come from overwhelmed services, rabbit or conductor or db, but I've never seen unknown timeouts like that | |
| 15:42:07 | Tahvok | The reason we got to conductor, is because we are trying to investigate a high cpu usage from rabbit service | |
| 15:42:30 | dansmith | which makes me wonder if there's some garbage on the bus, or some messages being echoed because of bad HA or something weird like that | |
| 15:42:38 | diliprenkila | ||
| 15:42:45 | sean-k-mooney | Tahvok: well rabbit is usually the first thing to melt as you scale out | |
| 15:43:45 | Tahvok | sean-k-mooney: I know that, but it's working completely fine apart from the high cpu usage (around 100%~400% on 32 core machine) .. Everything is green | |
| 15:43:50 | dansmith | Tahvok: conductor does not send messages except as replies, so if you think some service is generating load on rabbit, it'd be something else | |
| 15:44:23 | dansmith | based on what you've said, I would not suspect a nova bug, but a misconfiguration or something else acting up | |
| 15:44:31 | Tahvok | dansmith: so it's trying to reply to some unknown messages? Is it only computes it's talking to? | |
| 15:44:33 | stephenfin | diliprenkila: You've got a package version mismatch | |
| 15:44:54 | stephenfin | diliprenkila: Also, for usage questions like that, please use #openstack | |
| 15:44:55 | dansmith | Tahvok: no I don't think garbage will cause it to try to reply to things with "unknown" like that | |
| 15:45:00 | diliprenkila | <stephenfin> How did u find that ? | |
| 15:45:24 | dansmith | Tahvok: the only nova-related wrinkle is that if you have something hammering rabbit causing some timeouts between compute and conductor, the retries on compute may exacerbate the problem, causing more load on conductor as the backlog grows | |
| 15:45:57 | dansmith | Tahvok: if you think there's a nova bug you should file a bug with complete logs (not just single lines like you have provided here) and someone can look, but like I said, I suspect something non-nova as the root cause | |
| 15:47:26 | stephenfin | diliprenkila: https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7 | |
| 15:49:53 | Tahvok | dansmith: I have sorted 500 messages from the log, and all replies had different id's, which made me think it's conductor itself doing some stuff, and not simply retrying to answer to same timedout calls | |
| 15:50:18 | sean-k-mooney | Tahvok: one think you could try is deploying a seperate rabitmq instance for nova. that would help you isolate the issue. that said i know you may not want to do that on a running cloud | |
| 15:50:30 | mriedem | Tahvok: what versions of oslo.db and oslo.messaging are you using? | |
| 15:50:44 | mriedem | "AttributeError: '_TransactionContextManager' object has no attribute 'async_'"" suggests you're using an old oslo.db | |
| 15:50:48 | sean-k-mooney | mriedem: Tahvok mentioned it was an ocata cloud this morning | |
| 15:51:05 | mriedem | that doesn't tell me what i'd need to konw | |
| 15:51:12 | mriedem | you can be using min or max versions of oslo.db from ocata | |
| 15:51:13 | Tahvok | mriedem: sec, looking | |
| 15:51:15 | mriedem | or something completely different | |
| 15:51:36 | stephenfin | mriedem: diliprenkila had the same issue, unless you're mixing them up | |
| 15:51:37 | dansmith | mriedem: are you confusing Tahvok and diliprenkila ? | |
| 15:51:45 | stephenfin | dansmith: Yeah :) | |
| 15:52:07 | mriedem | yeah sorry | |
| 15:52:20 | stephenfin | mriedem: I think we've an issue there though. The patch I linked was released in 4.40.0, but we pin on a lower version https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7 | |
| 15:52:36 | stephenfin | I assume we should be handling the older version of bumping our minimum | |
| 15:52:56 | Tahvok | All versions of nova-conductor container: http://paste.openstack.org/show/728972/ | |