| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 14:44:47 | mriedem | b/c you can't delete an in-use volume | |
| 14:45:05 | mriedem | i think you can force delete an in-use volume though.... | |
| 14:46:01 | mriedem | but nova doesn't force delete and it's an admin-only API, and historically nova just relies on the user context token to do the detach/delete of the volume, we don't use configured admin credentials for that | |
| 14:46:45 | tommylikehu | we got a bug report from our customers saying that that period could be dangerous since other operations are not prohibitted | |
| 14:47:26 | Tahvok | Hey guys! Asked here before, but sean-k-mooney told me to wait 'till nova conductor cores show up online. We see some spikes of cpu usage of nova-conductor processes (every second or two): http://paste.openstack.org/show/728925/. | |
| 14:47:40 | openstackgerrit | Merged openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111 | |
| 14:48:33 | efried | kosamara: Hi, I'm here now. Ack, I'll write some words for 2,3,4. | |
| 14:48:54 | efried | kosamara: Would you like me to upload new patch sets to the spec, or dump the content somewhere for you to pull in? | |
| 14:49:42 | tommylikehu | mriedem: :) | |
| 14:49:42 | Tahvok | We have a big environment, with 160 compute hosts, and 2 controller (adding a third by the end of this week). I've enabled debugging on nova-conductor service, and saw that it's receiving around 30~120 messages per second. So I'm trying to investigate what could be throwing so much messages each second.. Our cloud is not very active, we have around 1 instance coming up/deleted every hour or so | |
| 14:50:23 | kosamara | efried: either works for me, but I think many patch sets can tangle up the discussion. Of course, this is my first spec, so you know. | |
| 14:51:07 | efried | kosamara: Meh, patch sets are no big deal; often they can help provide history/context of the evolution of a thing. As long as comments aren't lost along the way, it's fine. | |
| 14:51:14 | mriedem | tommylikehu: "since other operations are not prohibitted" ? | |
| 14:51:45 | mriedem | tommylikehu: meaning, the customer thinks something could reserve the volume in that split second while we're deleting the instance and then fail to delete the volume? | |
| 14:51:46 | efried | kosamara: Just need to avoid stepping on each other. So we should just both check in with each other before posting a new patch set to make sure the other doesn't have local edits pending. | |
| 14:51:48 | mriedem | and thus leave it orphaned | |
| 14:52:10 | tommylikehu | mriedem: yeah | |
| 14:52:19 | mriedem | tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah? | |
| 14:52:32 | kosamara | efried: I've also got pending changes on the "done" things. I'll post them tomorrow. | |
| 14:53:16 | efried | kosamara: Okay. How about I just compose content and post it to a pastebin for you to pull in? | |
| 14:53:22 | kosamara | cool | |
| 14:53:31 | mriedem | tommylikehu: unless you're aware of some other way to atomically delete a volume, this is just a known issue. the only thing i can see nova doing is using force_delete with cinder admin creds (if nova is configured for those) to delete the volume while it's attached to the server | |
| 14:54:04 | tommylikehu | mriedem: can we do something to protect this process? | |
| 14:54:23 | tommylikehu | mriedem: something like what we do when attaching volume | |
| 14:54:25 | mriedem | tommylikehu: is this an actual issue someone ran into or they are just doing some kind of audit? | |
| 14:55:04 | mriedem | and what client software is waiting a millisecond to attach a volume while we're deleting it from another server? | |
| 14:55:31 | mriedem | this seems extremely low priority | |
| 14:55:37 | tommylikehu | mriedem: personally I think it's the second case | |
| 14:55:49 | mriedem | ok; never tell them about ports then :) | |
| 14:56:04 | mriedem | because you can attach/detach those to servers out of band all you like | |
| 14:56:48 | mriedem | PUT /v2.0/ports/{port_id} with a new device_id - now it's my port yay! | |
| 14:57:37 | mriedem | tommylikehu: so like i said above, nova could use the force_delete API if we're configured with cinder admin creds, but i'd consider it very low priority | |
| 14:58:18 | tommylikehu | mriedem: oooook, thanks:) | |
| 15:00:30 | mriedem | tommylikehu: i guess start by reporting a bug to nova | |
| 15:00:33 | mriedem | so it's on the books | |
| 15:01:27 | dansmith | mriedem: tommylikehu the concern is that a volume nova is going to delete becomes available for a second before being deleted? | |
| 15:02:18 | tommylikehu | dansmith: yes | |
| 15:02:41 | mriedem | trump could get his grubby hands all of ma volumes | |
| 15:02:47 | mriedem | *all over | |
| 15:02:55 | dansmith | and what, something could attach those and block the delete? | |
| 15:03:02 | mriedem | yeah i guess | |
| 15:03:17 | dansmith | but the only entity that could do that already owns the thing | |
| 15:03:24 | mriedem | note: that doesn't block the server delete, | |
| 15:03:26 | dansmith | so, who cares? | |
| 15:03:34 | mriedem | and we already fail to delete the volume if it has snapshots | |
| 15:03:40 | mriedem | so yeah, this is like the lowest of priorities | |
| 15:04:01 | dansmith | or you could say it's working as designed and thus not a bug, which would be my preference | |
| 15:04:12 | tommylikehu | lol | |
| 15:04:39 | mriedem | that works for me | |
| 15:04:48 | mriedem | as i said above, "(9:52:21 AM) mriedem: tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?" | |
| 15:05:11 | dansmith | this would be like complaining that something that is able to hardlink the image file on disk before nova deletes it can still read the data | |
| 15:05:20 | dansmith | and the response to that is "yup. that's how that works" | |
| 15:05:46 | dansmith | I mean, just MHO of course | |
| 15:06:45 | mriedem | we have insanely bigger fish to fry so yes | |
| 15:06:50 | mriedem | like the port thing i already mentioned | |
| 15:07:33 | dansmith | how much bigger is insanely bigger? | |
| 15:07:40 | dansmith | like bigger^2? | |
| 15:10:30 | mriedem | hyperbole sized bigger | |
| 15:19:10 | dansmith | tssurya: mriedem: don't we want this to be under the big stack of down-cell patches so we can merge and backport it? https://review.openstack.org/#/c/592428/2 | |
| 15:20:17 | dansmith | oh sorry I guess it is | |
| 15:20:18 | dansmith | nevermind | |
| 15:24:54 | openstackgerrit | Merged openstack/nova stable/ocata: Default embedded instance.flavor.disabled attribute https://review.openstack.org/580525 | |
| 15:27:28 | dansmith | melwitt: if you want to hit this last patch in my series, it'll make the down cell stuff soon able to be based on master: https://review.openstack.org/#/c/594577/11 | |
| 15:33:33 | sean-k-mooney | dansmith: Tahvok was asking about periodic spike in nova-conductor cpu usage in a cloud of ~160 compute nodes eairler. bejond the periodic jobs i was not sure what would be likely to cause the condoctor to be processing ~100 rpc messages a second. any toughts? | |
| 15:33:58 | dansmith | sean-k-mooney: sounds like a support (not dev) question | |
| 15:34:27 | dansmith | sean-k-mooney: but yeah, conductor pretty much just answers to nova-compute on an idle cloud, so it'd be periodics from compute nodes | |
| 15:34:54 | sean-k-mooney | dansmith: perhapes but do we expect 18 nova-condoctor worker treads to spike to 100% usage every 1-2 seconds http://paste.openstack.org/show/728925/ | |
| 15:35:43 | dansmith | we should totally call it condoctor | |
| 15:35:59 | sean-k-mooney | :) | |
| 15:37:09 | Tahvok | lol | |
| 15:37:13 | dansmith | I assume that was a rhetorical question, but 160 computes configured to run some periodics every minute could certainly generate a fair bit of traffic | |
| 15:37:39 | sean-k-mooney | i was wondering if we should consider introducing intetional jitter in the perodic jobs to maybe spread when the jobs are running on each node | |
| 15:37:50 | dansmith | that is already done | |
| 15:37:52 | Tahvok | dansmith: but 100 messages a second? Don't they sync every minute or so? That should generate around 2.5 message a second.. Not a 100.. | |
| 15:38:04 | openstackgerrit | Merged openstack/nova master: Make monkey patch work in uWSGI mode https://review.openstack.org/592285 | |
| 15:38:27 | dansmith | Tahvok: they sync as often as you have them configured for.. without knowing what the messages are, who is sending them, etc, it's hard to say what the problem is | |
| 15:40:03 | diliprenkila | Hi all, I keep getting these errors on my compute nodes "ERROR oslo_service.service [req-0c500027-fc8f-4a24-b1c1-9714b3f248e6 - - - - -] Error starting thread.: AttributeError: '_TransactionContextManager' object has no attribute 'async_'" | |
| 15:40:06 | Tahvok | dansmith: I have tried to debug the rpc calls.. I've tried setting default_log_levels = oslo,messaging=DEBUG, and got lots of messages like this: | |
| 15:40:07 | Tahvok | And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout | |
| 15:40:42 | Tahvok | No idea what this unknowns are... | |
| 15:40:54 | dansmith | Tahvok: I have no idea what would cause that kind of debug with unknown calls and topics | |
| 15:41:07 | dansmith | Tahvok: I would strongly suspect something is majorly broken with your setup | |
| 15:41:20 | Tahvok | dansmith: almost all the messages were followed with this unknown message | |
| 15:41:31 | sean-k-mooney | Tahvok: didnt you say the cloud was fuctionlaly and able to boot vms? | |
| 15:41:40 | Tahvok | sean-k-mooney: yep, everything works | |
| 15:42:06 | dansmith | timeouts generally come from overwhelmed services, rabbit or conductor or db, but I've never seen unknown timeouts like that | |
| 15:42:07 | Tahvok | The reason we got to conductor, is because we are trying to investigate a high cpu usage from rabbit service | |
| 15:42:30 | dansmith | which makes me wonder if there's some garbage on the bus, or some messages being echoed because of bad HA or something weird like that | |
| 15:42:38 | diliprenkila | ||
| 15:42:45 | sean-k-mooney | Tahvok: well rabbit is usually the first thing to melt as you scale out | |
| 15:43:45 | Tahvok | sean-k-mooney: I know that, but it's working completely fine apart from the high cpu usage (around 100%~400% on 32 core machine) .. Everything is green | |
| 15:43:50 | dansmith | Tahvok: conductor does not send messages except as replies, so if you think some service is generating load on rabbit, it'd be something else | |
| 15:44:23 | dansmith | based on what you've said, I would not suspect a nova bug, but a misconfiguration or something else acting up | |
| 15:44:31 | Tahvok | dansmith: so it's trying to reply to some unknown messages? Is it only computes it's talking to? | |
| 15:44:33 | stephenfin | diliprenkila: You've got a package version mismatch | |
| 15:44:54 | stephenfin | diliprenkila: Also, for usage questions like that, please use #openstack | |
| 15:44:55 | dansmith | Tahvok: no I don't think garbage will cause it to try to reply to things with "unknown" like that | |
| 15:45:00 | diliprenkila | <stephenfin> How did u find that ? | |
| 15:45:24 | dansmith | Tahvok: the only nova-related wrinkle is that if you have something hammering rabbit causing some timeouts between compute and conductor, the retries on compute may exacerbate the problem, causing more load on conductor as the backlog grows | |
| 15:45:57 | dansmith | Tahvok: if you think there's a nova bug you should file a bug with complete logs (not just single lines like you have provided here) and someone can look, but like I said, I suspect something non-nova as the root cause | |