| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 14:43:48 | mriedem | tommylikehu: you had a question about volumes? | |
| 14:44:05 | tommylikehu | yeah, I have a question regarding deleting operations, When deleting instance and its related volumes, there could be a period of time that those volumes' statues can be available, right? I mean right after the volume is detached | |
| 14:44:37 | mriedem | tommylikehu: i believe so yes | |
| 14:44:47 | mriedem | b/c you can't delete an in-use volume | |
| 14:45:05 | mriedem | i think you can force delete an in-use volume though.... | |
| 14:46:01 | mriedem | but nova doesn't force delete and it's an admin-only API, and historically nova just relies on the user context token to do the detach/delete of the volume, we don't use configured admin credentials for that | |
| 14:46:45 | tommylikehu | we got a bug report from our customers saying that that period could be dangerous since other operations are not prohibitted | |
| 14:47:26 | Tahvok | Hey guys! Asked here before, but sean-k-mooney told me to wait 'till nova conductor cores show up online. We see some spikes of cpu usage of nova-conductor processes (every second or two): http://paste.openstack.org/show/728925/. | |
| 14:47:40 | openstackgerrit | Merged openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111 | |
| 14:48:33 | efried | kosamara: Hi, I'm here now. Ack, I'll write some words for 2,3,4. | |
| 14:48:54 | efried | kosamara: Would you like me to upload new patch sets to the spec, or dump the content somewhere for you to pull in? | |
| 14:49:42 | Tahvok | We have a big environment, with 160 compute hosts, and 2 controller (adding a third by the end of this week). I've enabled debugging on nova-conductor service, and saw that it's receiving around 30~120 messages per second. So I'm trying to investigate what could be throwing so much messages each second.. Our cloud is not very active, we have around 1 instance coming up/deleted every hour or so | |
| 14:49:42 | tommylikehu | mriedem: :) | |
| 14:50:23 | kosamara | efried: either works for me, but I think many patch sets can tangle up the discussion. Of course, this is my first spec, so you know. | |
| 14:51:07 | efried | kosamara: Meh, patch sets are no big deal; often they can help provide history/context of the evolution of a thing. As long as comments aren't lost along the way, it's fine. | |
| 14:51:14 | mriedem | tommylikehu: "since other operations are not prohibitted" ? | |
| 14:51:45 | mriedem | tommylikehu: meaning, the customer thinks something could reserve the volume in that split second while we're deleting the instance and then fail to delete the volume? | |
| 14:51:46 | efried | kosamara: Just need to avoid stepping on each other. So we should just both check in with each other before posting a new patch set to make sure the other doesn't have local edits pending. | |
| 14:51:48 | mriedem | and thus leave it orphaned | |
| 14:52:10 | tommylikehu | mriedem: yeah | |
| 14:52:19 | mriedem | tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah? | |
| 14:52:32 | kosamara | efried: I've also got pending changes on the "done" things. I'll post them tomorrow. | |
| 14:53:16 | efried | kosamara: Okay. How about I just compose content and post it to a pastebin for you to pull in? | |
| 14:53:22 | kosamara | cool | |
| 14:53:31 | mriedem | tommylikehu: unless you're aware of some other way to atomically delete a volume, this is just a known issue. the only thing i can see nova doing is using force_delete with cinder admin creds (if nova is configured for those) to delete the volume while it's attached to the server | |
| 14:54:04 | tommylikehu | mriedem: can we do something to protect this process? | |
| 14:54:23 | tommylikehu | mriedem: something like what we do when attaching volume | |
| 14:54:25 | mriedem | tommylikehu: is this an actual issue someone ran into or they are just doing some kind of audit? | |
| 14:55:04 | mriedem | and what client software is waiting a millisecond to attach a volume while we're deleting it from another server? | |
| 14:55:31 | mriedem | this seems extremely low priority | |
| 14:55:37 | tommylikehu | mriedem: personally I think it's the second case | |
| 14:55:49 | mriedem | ok; never tell them about ports then :) | |
| 14:56:04 | mriedem | because you can attach/detach those to servers out of band all you like | |
| 14:56:48 | mriedem | PUT /v2.0/ports/{port_id} with a new device_id - now it's my port yay! | |
| 14:57:37 | mriedem | tommylikehu: so like i said above, nova could use the force_delete API if we're configured with cinder admin creds, but i'd consider it very low priority | |
| 14:58:18 | tommylikehu | mriedem: oooook, thanks:) | |
| 15:00:30 | mriedem | tommylikehu: i guess start by reporting a bug to nova | |
| 15:00:33 | mriedem | so it's on the books | |
| 15:01:27 | dansmith | mriedem: tommylikehu the concern is that a volume nova is going to delete becomes available for a second before being deleted? | |
| 15:02:18 | tommylikehu | dansmith: yes | |
| 15:02:41 | mriedem | trump could get his grubby hands all of ma volumes | |
| 15:02:47 | mriedem | *all over | |
| 15:02:55 | dansmith | and what, something could attach those and block the delete? | |
| 15:03:02 | mriedem | yeah i guess | |
| 15:03:17 | dansmith | but the only entity that could do that already owns the thing | |
| 15:03:24 | mriedem | note: that doesn't block the server delete, | |
| 15:03:26 | dansmith | so, who cares? | |
| 15:03:34 | mriedem | and we already fail to delete the volume if it has snapshots | |
| 15:03:40 | mriedem | so yeah, this is like the lowest of priorities | |
| 15:04:01 | dansmith | or you could say it's working as designed and thus not a bug, which would be my preference | |
| 15:04:12 | tommylikehu | lol | |
| 15:04:39 | mriedem | that works for me | |
| 15:04:48 | mriedem | as i said above, "(9:52:21 AM) mriedem: tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?" | |
| 15:05:11 | dansmith | this would be like complaining that something that is able to hardlink the image file on disk before nova deletes it can still read the data | |
| 15:05:20 | dansmith | and the response to that is "yup. that's how that works" | |
| 15:05:46 | dansmith | I mean, just MHO of course | |
| 15:06:45 | mriedem | we have insanely bigger fish to fry so yes | |
| 15:06:50 | mriedem | like the port thing i already mentioned | |
| 15:07:33 | dansmith | how much bigger is insanely bigger? | |
| 15:07:40 | dansmith | like bigger^2? | |
| 15:10:30 | mriedem | hyperbole sized bigger | |
| 15:19:10 | dansmith | tssurya: mriedem: don't we want this to be under the big stack of down-cell patches so we can merge and backport it? https://review.openstack.org/#/c/592428/2 | |
| 15:20:17 | dansmith | oh sorry I guess it is | |
| 15:20:18 | dansmith | nevermind | |
| 15:24:54 | openstackgerrit | Merged openstack/nova stable/ocata: Default embedded instance.flavor.disabled attribute https://review.openstack.org/580525 | |
| 15:27:28 | dansmith | melwitt: if you want to hit this last patch in my series, it'll make the down cell stuff soon able to be based on master: https://review.openstack.org/#/c/594577/11 | |
| 15:33:33 | sean-k-mooney | dansmith: Tahvok was asking about periodic spike in nova-conductor cpu usage in a cloud of ~160 compute nodes eairler. bejond the periodic jobs i was not sure what would be likely to cause the condoctor to be processing ~100 rpc messages a second. any toughts? | |
| 15:33:58 | dansmith | sean-k-mooney: sounds like a support (not dev) question | |
| 15:34:27 | dansmith | sean-k-mooney: but yeah, conductor pretty much just answers to nova-compute on an idle cloud, so it'd be periodics from compute nodes | |
| 15:34:54 | sean-k-mooney | dansmith: perhapes but do we expect 18 nova-condoctor worker treads to spike to 100% usage every 1-2 seconds http://paste.openstack.org/show/728925/ | |
| 15:35:43 | dansmith | we should totally call it condoctor | |
| 15:35:59 | sean-k-mooney | :) | |
| 15:37:09 | Tahvok | lol | |
| 15:37:13 | dansmith | I assume that was a rhetorical question, but 160 computes configured to run some periodics every minute could certainly generate a fair bit of traffic | |
| 15:37:39 | sean-k-mooney | i was wondering if we should consider introducing intetional jitter in the perodic jobs to maybe spread when the jobs are running on each node | |
| 15:37:50 | dansmith | that is already done | |
| 15:37:52 | Tahvok | dansmith: but 100 messages a second? Don't they sync every minute or so? That should generate around 2.5 message a second.. Not a 100.. | |
| 15:38:04 | openstackgerrit | Merged openstack/nova master: Make monkey patch work in uWSGI mode https://review.openstack.org/592285 | |
| 15:38:27 | dansmith | Tahvok: they sync as often as you have them configured for.. without knowing what the messages are, who is sending them, etc, it's hard to say what the problem is | |
| 15:40:03 | diliprenkila | Hi all, I keep getting these errors on my compute nodes "ERROR oslo_service.service [req-0c500027-fc8f-4a24-b1c1-9714b3f248e6 - - - - -] Error starting thread.: AttributeError: '_TransactionContextManager' object has no attribute 'async_'" | |
| 15:40:06 | Tahvok | dansmith: I have tried to debug the rpc calls.. I've tried setting default_log_levels = oslo,messaging=DEBUG, and got lots of messages like this: | |
| 15:40:07 | Tahvok | And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout | |
| 15:40:42 | Tahvok | No idea what this unknowns are... | |
| 15:40:54 | dansmith | Tahvok: I have no idea what would cause that kind of debug with unknown calls and topics | |
| 15:41:07 | dansmith | Tahvok: I would strongly suspect something is majorly broken with your setup | |
| 15:41:20 | Tahvok | dansmith: almost all the messages were followed with this unknown message | |
| 15:41:31 | sean-k-mooney | Tahvok: didnt you say the cloud was fuctionlaly and able to boot vms? | |
| 15:41:40 | Tahvok | sean-k-mooney: yep, everything works | |
| 15:42:06 | dansmith | timeouts generally come from overwhelmed services, rabbit or conductor or db, but I've never seen unknown timeouts like that | |
| 15:42:07 | Tahvok | The reason we got to conductor, is because we are trying to investigate a high cpu usage from rabbit service | |
| 15:42:30 | dansmith | which makes me wonder if there's some garbage on the bus, or some messages being echoed because of bad HA or something weird like that | |
| 15:42:38 | diliprenkila | ||
| 15:42:45 | sean-k-mooney | Tahvok: well rabbit is usually the first thing to melt as you scale out | |
| 15:43:45 | Tahvok | sean-k-mooney: I know that, but it's working completely fine apart from the high cpu usage (around 100%~400% on 32 core machine) .. Everything is green | |
| 15:43:50 | dansmith | Tahvok: conductor does not send messages except as replies, so if you think some service is generating load on rabbit, it'd be something else | |
| 15:44:23 | dansmith | based on what you've said, I would not suspect a nova bug, but a misconfiguration or something else acting up | |
| 15:44:31 | Tahvok | dansmith: so it's trying to reply to some unknown messages? Is it only computes it's talking to? | |
| 15:44:33 | stephenfin | diliprenkila: You've got a package version mismatch | |
| 15:44:54 | stephenfin | diliprenkila: Also, for usage questions like that, please use #openstack | |
| 15:44:55 | dansmith | Tahvok: no I don't think garbage will cause it to try to reply to things with "unknown" like that | |