Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-28
10:46:12 Tahvok Hey guys! I'm trying to understand why nova conductor is receiving a lot of messages.. I mean I get from 30 to 120 messages a second. The environment is not small, we have around 160 compute hosts, but it's not very active. We have a new instance coming up/deleted every hour or so
10:47:08 Tahvok Apart from it, we have our rabbit service working with 100% up to 400% cpu all the time, and along with it, we see the nova-conductor service processes taking 10% cpu each
10:53:12 sean-k-mooney Tahvok: i could be wrong but it think all database acess from the computenodes is relayed via the conductor so the periodic jobs that update the compute node resouces and health will be a portion of those messages
10:53:50 Tahvok sean-k-mooney: thanks.. I
10:54:12 Tahvok I've just tried something else: default_log_levels = oslo,messaging=DEBUG
10:54:35 Tahvok And I see lots of messages like this: 2018-08-28 05:54:26.497 23601 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" _raise_timeout /openstack/venvs/nova-15.1.25/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:1052
10:55:16 Tahvok What are this unknown calls? Also, rabbit seems to be working fine, as we don't have any timeout issues when creating new instances
10:55:26 sean-k-mooney hum that not very desciptive. we probaly should have better logging
10:55:49 Tahvok Btw, I'm on Ocata if that matters
10:57:00 sean-k-mooney stephenfin: any RPC people around that you can think of. i would ping dansmith but he should be sleeping for another few hours
10:57:34 Tahvok What I'm trying to fix is basically this periodic (every second or 2) spikes: http://paste.openstack.org/show/728925/
10:58:23 Tahvok Each infra host has 32 thread cores, I think it should be enough to handle 160 compute hosts
10:58:50 sean-k-mooney Tahvok: so every few seconds the condoctor is taking 100% cpu across all cores?
10:59:01 Tahvok I might be wrong thought, and we need to increase our resources, that's why I'm consulting with you
10:59:22 sean-k-mooney Tahvok: do you have 1 controler or several
10:59:39 sean-k-mooney Tahvok: also no you should be fine
11:00:13 Tahvok sean-k-mooney: not all cores, as there are only 18 nova-conductor processes running, at least according to this: http://paste.openstack.org/show/728926/
11:00:43 Tahvok sean-k-mooney: 2 controllers, and we plan to add another one by the end of this week
11:00:44 sean-k-mooney 1 contoler should be eaislly able to handle 160 nodes that are more or less idel in terms of vm lifcyle events
11:02:20 sean-k-mooney Tahvok: my guess is the spikes are caused by the periodic jobs. if all the clocks are synced all 160 nodes will submit the there updates around the same time
11:03:07 sean-k-mooney we proables should be intoducing some spread in when they run. that said without any logs/error that is jsut a guess
11:03:29 Tahvok We have other network issues that network team is handling right now (ksoftirqd is taking lots of cpu (80%+)), but we are trying to fix the rabbit issue. That's what we see on the active router controller: http://paste.openstack.org/show/728928/
11:04:11 Tahvok sean-k-mooney: don't they submit the sync every minute? It doesn't explain ~100 messages every second
11:05:20 Tahvok And we don't have ceilometer, so rabbit handles only the basic openstack services
11:09:23 sean-k-mooney they do. but that does not seam that high to me for a cloud of your size.
11:13:10 Tahvok sean-k-mooney: so 100+ messages a second is normal for this size?
11:18:09 sean-k-mooney Tahvok: i unfortuetly dont have that data to hand.
11:24:32 sean-k-mooney Tahvok: we did some sacle testing back 2 years ago https://review.openstack.org/#/c/352101/ that im quickly checking.
11:25:59 sean-k-mooney Tahvok: that was done with a cloud with 3 contolers and about 230 compute nodes. if i remember correctly the total cpu usage on the contolers was in low 10% for the majority of the testing
12:02:08 stephenfin sean-k-mooney: bauzas would be my other suggestion but he's still on vacation
12:07:51 sean-k-mooney stephenfin: ya he came to mind. Tahvok i would suggest asking again in an hour or so. the us based cores that work on the conductor will be online then and perhaps can give a better answer
12:08:28 Tahvok sean-k-mooney: ok, thanks!
12:42:04 openstackgerrit Konstantinos Samaras-Tsakiris proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
12:47:38 openstackgerrit Merged openstack/nova master: Deprecate Core/Ram/DiskFilter https://review.openstack.org/596502
13:12:36 gibi mriedem: hi! Do you have topics for the today's notification subteam meeting?
13:13:17 mriedem nope
13:15:40 gibi mriedem: cool, then I will cancel
13:15:48 mriedem wfm
13:17:56 gibi mriedem: we got a report that not just the flavor.disabled is missing in some old embedded falvors but flavor.is_public as well. https://bugs.launchpad.net/nova/+bug/1739325 I will look at it a bit later but I guess a solution will be similar than for flavor.disabled
13:17:56 openstack Launchpad bug 1739325 in OpenStack Compute (nova) ocata "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
13:19:23 mriedem gibi: i suppose,
13:19:34 mriedem difference between disabled and is_public is disabled isn't in any API but is_public is
13:20:23 gibi so I have to check that what happens on the API if such old flavor is present and fix possible failures there as well
13:20:46 mriedem well,
13:21:08 mriedem this isn't a failure on flavor resources, or shouldn't be, it's embedded flavors in the instance which were originally migrated from the instance system_metadata
13:21:57 mriedem this is the problem http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/flavors.py#n52
13:22:09 mriedem is_public isn't in there, so it wasn't stored in the embeded instance.flavor
13:22:15 mriedem so we'll have to default to is_public=True
13:22:20 mriedem if it's not in the embedded flavor
13:24:32 mriedem i left a comment on the bug
13:24:40 gibi mriedem: OK, I have to look at the API code showing the embeded flavor to see how is_public is handled there
13:24:50 mriedem i'll report a new bug for is_public since this other one is already closed
13:25:27 mriedem we don't show that field from the instance.flavor
13:25:58 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/views/servers.py#n349
13:26:56 gibi mriedem: then the API is not broken. cool. You are way faster to find these things in the code than me.
13:27:07 gibi mriedem: I agree that it needs a separate bug
13:27:24 gibi mriedem: thanks for reporting it
13:27:36 mriedem https://bugs.launchpad.net/nova/+bug/1789423
13:27:36 openstack Launchpad bug 1789423 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset is_public field" [Undecided,New]
13:28:17 mriedem lyarwood: speaking of which, can you hit this? https://review.openstack.org/#/c/580525/
13:32:02 lyarwood mriedem: yup looking
13:34:22 gibi jaypipes: thanks for the mail about the consumer gen handling in nova I think efried_doc's and your comment together helps me redo the patch series (now I have to find the time to do it).
13:42:04 jaypipes gibi: np
14:07:24 kosamara Hi efried, are you around?
14:09:59 openstackgerrit Jay Pipes proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:10:32 jaypipes stephenfin: can you +2/W https://review.openstack.org/#/c/597111/ please?
14:11:11 stephenfin jaypipes: Sure, done
14:11:36 jaypipes stephenfin: danke
14:11:48 jaypipes kosamara: ^^
14:11:54 jaypipes kosamara: thx mate
14:20:26 kosamara jaypipes: thanks!
14:23:48 kosamara efried: I'd like to take up your offer :) From the 4 major things to address on the spec, could you focus on 2,3,4 so that I can focus on 1 for libvirt?
14:24:54 kosamara efried: I'll also see if I can carry over content from your OOT spec for powervm there. And I think alex_xu's questions are part of the "cyborg intersection".
14:43:48 mriedem tommylikehu: you had a question about volumes?
14:44:05 tommylikehu yeah, I have a question regarding deleting operations, When deleting instance and its related volumes, there could be a period of time that those volumes' statues can be available, right? I mean right after the volume is detached
14:44:37 mriedem tommylikehu: i believe so yes
14:44:47 mriedem b/c you can't delete an in-use volume
14:45:05 mriedem i think you can force delete an in-use volume though....
14:46:01 mriedem but nova doesn't force delete and it's an admin-only API, and historically nova just relies on the user context token to do the detach/delete of the volume, we don't use configured admin credentials for that
14:46:45 tommylikehu we got a bug report from our customers saying that that period could be dangerous since other operations are not prohibitted
14:47:26 Tahvok Hey guys! Asked here before, but sean-k-mooney told me to wait 'till nova conductor cores show up online. We see some spikes of cpu usage of nova-conductor processes (every second or two): http://paste.openstack.org/show/728925/.
14:47:40 openstackgerrit Merged openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:48:33 efried kosamara: Hi, I'm here now. Ack, I'll write some words for 2,3,4.
14:48:54 efried kosamara: Would you like me to upload new patch sets to the spec, or dump the content somewhere for you to pull in?
14:49:42 tommylikehu mriedem: :)
14:49:42 Tahvok We have a big environment, with 160 compute hosts, and 2 controller (adding a third by the end of this week). I've enabled debugging on nova-conductor service, and saw that it's receiving around 30~120 messages per second. So I'm trying to investigate what could be throwing so much messages each second.. Our cloud is not very active, we have around 1 instance coming up/deleted every hour or so
14:50:23 kosamara efried: either works for me, but I think many patch sets can tangle up the discussion. Of course, this is my first spec, so you know.
14:51:07 efried kosamara: Meh, patch sets are no big deal; often they can help provide history/context of the evolution of a thing. As long as comments aren't lost along the way, it's fine.
14:51:14 mriedem tommylikehu: "since other operations are not prohibitted" ?
14:51:45 mriedem tommylikehu: meaning, the customer thinks something could reserve the volume in that split second while we're deleting the instance and then fail to delete the volume?
14:51:46 efried kosamara: Just need to avoid stepping on each other. So we should just both check in with each other before posting a new patch set to make sure the other doesn't have local edits pending.
14:51:48 mriedem and thus leave it orphaned
14:52:10 tommylikehu mriedem: yeah
14:52:19 mriedem tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?
14:52:32 kosamara efried: I've also got pending changes on the "done" things. I'll post them tomorrow.
14:53:16 efried kosamara: Okay. How about I just compose content and post it to a pastebin for you to pull in?
14:53:22 kosamara cool
14:53:31 mriedem tommylikehu: unless you're aware of some other way to atomically delete a volume, this is just a known issue. the only thing i can see nova doing is using force_delete with cinder admin creds (if nova is configured for those) to delete the volume while it's attached to the server
14:54:04 tommylikehu mriedem: can we do something to protect this process?
14:54:23 tommylikehu mriedem: something like what we do when attaching volume

Earlier   Later