Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-07
15:01:07 openstackgerrit Dan Smith proposed openstack/nova master: Add xenapi driver image type capabilities https://review.opendev.org/655731
15:01:08 openstackgerrit Dan Smith proposed openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903
15:01:08 openstackgerrit Dan Smith proposed openstack/nova master: Add image type request filter https://review.opendev.org/656413
15:01:09 openstackgerrit Dan Smith proposed openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025
15:03:10 dansmith mriedem: I've been thinking about the tracing thing too
15:05:14 dansmith more about making the existing filters log what they're doing, but the timer thing is legit too
15:08:42 mriedem i'd think that osprofiler should be handling the time spent in each filter, but not sure how granular that is
15:10:42 dansmith meaning we need some sort of hook thing?
15:12:06 mriedem i'm not sure if the @profiler.trace_cls decorator on the SchedulerAPI (rpc) gives us what we'd want there
15:12:15 mriedem since select_destinations is a call it might, but not sure
15:12:41 openstackgerrit Surya Seetharaman proposed openstack/nova master: [Docs] Change the server query parameter display into a list docs. https://review.opendev.org/657624
15:14:42 mriedem there was a recent ML thread about nicer osprofiler visualizations but i can't find it
15:16:24 mriedem oh right, something like this http://logs.openstack.org/69/617269/3/check/tempest-smoke-py3-osprofiler-redis/d5563c0/osprofiler-traces/trace-58ed6da6-f2c8-418c-9c60-337bf00ad86a.html.gz
15:16:29 openstackgerrit Surya Seetharaman proposed openstack/nova master: [Docs] Change the server query parameter display into a list. https://review.opendev.org/657624
15:16:33 mriedem not sure how to easily find a server create trace in there
15:21:37 dansmith mriedem: well, I'll start something and you can add to it if you want
15:22:08 dansmith it'd be really nice if we could see what the reqspec looks like before/after in a generic way, but that's probably too detailed and too fragile, vs. each one just logging what it's doing
15:22:39 mriedem i guess http://logs.openstack.org/69/617269/3/check/tempest-smoke-py3-osprofiler-redis/d5563c0/osprofiler-traces/trace-46016681-2a3d-4ca2-adf7-c4941649d753.html.gz and filter on nova-scheduler
15:23:08 mriedem doesn't give me the time per filter though
15:23:39 mriedem just says that like select_destinations took 302ms
15:29:47 mriedem ah timeutils already has something i was thinking to use
15:29:55 dansmith I was just using stopwatch
15:30:35 mriedem looks like you can just use oslo_utils.timeutils.time_it
15:31:27 dansmith yeah, but I want to not log if the thing appears disabled
15:32:14 mriedem the decorator takes an enabled kwarg
15:32:43 dansmith but we don't know until after it's run, unless we change something
15:32:58 dansmith I was just going to have them return a boolean if they're enabled/effective
15:33:05 dansmith like, if we're is_bfv=True, no point in logging 0.0
15:33:23 dansmith I could only log if above a threshold, but then you don't know if something is running or running fastly
15:33:24 mriedem ok that's more granular than i was thinking at first, like this is pretty easy:
15:33:25 mriedem http://paste.openstack.org/show/750876/
15:33:50 dansmith yeah, that works for the two simple ones, but not for my new one
15:34:09 dansmith because it has three reasons to not log
15:34:23 dansmith give me a few and I'll push up what I was thinking
15:34:40 mriedem map_az_to_placement_aggregate also has several reasons to not do anything
15:34:45 mriedem so depends on how granular you want this
15:38:49 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629
15:38:52 dansmith mriedem: ^
15:39:11 dansmith generic timer and contextual logs from each filter about what change it's making
15:39:20 dansmith which I think is what I'd want to validate or diagnose
15:50:10 mriedem i think your unit tests in https://review.opendev.org/#/c/656413/ are going to blow up
15:51:36 dansmith I thought is_bfv was usage safe so it would just return false
15:51:43 dansmith but I didn't run them
15:52:31 dansmith or maybe we set it somewhere before we get too far, but I won't hit that in the unit tests
15:52:56 mriedem conductor sets it before the scheduler for a move, and api for a new server, but your unit tests won't lazy-load the value
15:53:25 dansmith yeah
15:54:33 dansmith I need to remove the empty image from the volume case anyway
15:58:39 openstackgerrit Dan Smith proposed openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903
15:58:40 openstackgerrit Dan Smith proposed openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025
16:24:14 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629
16:40:20 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
17:23:36 dansmith ah crap
17:24:57 openstackgerrit Dan Smith proposed openstack/nova master: Add image type request filter https://review.opendev.org/656413
17:24:58 openstackgerrit Dan Smith proposed openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025
17:24:58 openstackgerrit Dan Smith proposed openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903
17:24:59 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629
17:34:28 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif stable/rocky: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655692
17:50:38 dansmith mriedem: have you been following the helm and health check stuff that wants to throw a non-nova tool on our bus to ping all our services all the time for health checks?
17:51:34 dansmith I've been arguing for going the real http-based healthcheck route to avoid extra message bus load, the upgrade concerns, and to, you know, allow health checks to work in the absence of a working message bus, which is the most likely failure in any deployment :)
17:52:49 artom <drive by> I'd argue that you'd still need to check message queue health, but that it's a separate thing and shouldn't be confused with Nova
17:53:16 artom Cuz if your message queue is down, not much is going to work :)
17:53:25 dansmith nova does this already, and, the http based check can report "hey I can't talk to rabbit"
17:53:46 dansmith the http check returns a json blob of "things I know about" and "are they working or not"
17:54:16 dansmith so, "can I talk to rabbit" "am I able to report my service status" "can I talk to libvirt right now" etc
17:55:07 dansmith doing that over the message bus adds load to the most loaded thing, and doesn't work if the bus is down,
17:55:27 dansmith but also won't work if conductor is down after a few minutes, because nova-compute is likely to have exhausted its threadpool trying to check in
17:55:33 dansmith so it's really a fragile expectation I think
18:15:32 artom TIL
18:15:32 artom dansmith, oh, we have a specific health check "API"?
18:16:00 dansmith no, but kube has a standard format that people have proposed adding to our services
18:16:06 dansmith and I'm saying we should do _that_
18:16:21 artom Seems sensible to me
18:16:32 artom Better than doing it through the back door
18:16:51 artom What, that I agreed with you?
18:17:21 dansmith no, nevermind :)
18:17:29 artom Oh I know.
18:17:42 artom But... this being a professional channel and all.
18:22:30 cdent "we should do _that_"++
18:23:54 mriedem dansmith: no i haven't been following it
18:24:26 dansmith they already have an agent that sits on our bus and pings services, and unsurprisingly, have found that the version matters and things break if you don't have that right
18:25:11 dansmith and I just want to make sure we're not getting into a situation where we bless that kind of activity, and/or start trying to worry about external consumers of our RPC APIs and hold versions around because of what might be out in the wild
18:25:30 dansmith https://review.opendev.org/#/c/651140/
18:28:16 artom I mean, we could be evil and just remove ping()
18:28:38 dansmith artom: we need it for our own use
18:28:51 dansmith and note that that error dump was on the *nova* server side, as a result of pinging with an old version,
18:29:01 dansmith so that tool potentially generates bug reports for us
18:29:29 dansmith artom: compute pings conductor at startup and waits until it replies, in case it gets started earlier, since it'll fail without conductor being up
18:29:51 dansmith I told them about ping because they wanted to add their own ping-like thing to our rpcapi that they would use to do that,
18:30:07 dansmith so I was like "we already have this, so please don't add another one, but also... about that approach..."
18:31:11 artom Wait, ping() is in the conductor rpcapi, right?
18:31:19 artom So... how are they going to check for Nova API health?
18:31:32 artom Which I assume is what they most care about...
18:31:53 openstackgerrit Chris Dent proposed openstack/nova master: Make all functional tests reusable by other projects https://review.opendev.org/657659
18:33:27 artom Ah, it's in the base class
18:33:31 artom So all our services have it
18:34:18 dansmith they can't check api that way anyway
18:34:22 dansmith because api doesn't listen
18:34:32 dansmith but yeah, it's baked in for the services that do
18:35:24 artom So that's a good argument: you can only use ping() on services that listen to the message queue, which API doesn't.
18:35:32 artom And API is kinda important ;)

Earlier   Later