Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-07
15:34:09 dansmith because it has three reasons to not log
15:34:23 dansmith give me a few and I'll push up what I was thinking
15:34:40 mriedem map_az_to_placement_aggregate also has several reasons to not do anything
15:34:45 mriedem so depends on how granular you want this
15:38:49 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629
15:38:52 dansmith mriedem: ^
15:39:11 dansmith generic timer and contextual logs from each filter about what change it's making
15:39:20 dansmith which I think is what I'd want to validate or diagnose
15:50:10 mriedem i think your unit tests in https://review.opendev.org/#/c/656413/ are going to blow up
15:51:36 dansmith I thought is_bfv was usage safe so it would just return false
15:51:43 dansmith but I didn't run them
15:52:31 dansmith or maybe we set it somewhere before we get too far, but I won't hit that in the unit tests
15:52:56 mriedem conductor sets it before the scheduler for a move, and api for a new server, but your unit tests won't lazy-load the value
15:53:25 dansmith yeah
15:54:33 dansmith I need to remove the empty image from the volume case anyway
15:58:39 openstackgerrit Dan Smith proposed openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903
15:58:40 openstackgerrit Dan Smith proposed openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025
16:24:14 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629
16:40:20 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
17:23:36 dansmith ah crap
17:24:57 openstackgerrit Dan Smith proposed openstack/nova master: Add image type request filter https://review.opendev.org/656413
17:24:58 openstackgerrit Dan Smith proposed openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025
17:24:58 openstackgerrit Dan Smith proposed openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903
17:24:59 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629
17:34:28 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif stable/rocky: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655692
17:50:38 dansmith mriedem: have you been following the helm and health check stuff that wants to throw a non-nova tool on our bus to ping all our services all the time for health checks?
17:51:34 dansmith I've been arguing for going the real http-based healthcheck route to avoid extra message bus load, the upgrade concerns, and to, you know, allow health checks to work in the absence of a working message bus, which is the most likely failure in any deployment :)
17:52:49 artom <drive by> I'd argue that you'd still need to check message queue health, but that it's a separate thing and shouldn't be confused with Nova
17:53:16 artom Cuz if your message queue is down, not much is going to work :)
17:53:25 dansmith nova does this already, and, the http based check can report "hey I can't talk to rabbit"
17:53:46 dansmith the http check returns a json blob of "things I know about" and "are they working or not"
17:54:16 dansmith so, "can I talk to rabbit" "am I able to report my service status" "can I talk to libvirt right now" etc
17:55:07 dansmith doing that over the message bus adds load to the most loaded thing, and doesn't work if the bus is down,
17:55:27 dansmith but also won't work if conductor is down after a few minutes, because nova-compute is likely to have exhausted its threadpool trying to check in
17:55:33 dansmith so it's really a fragile expectation I think
18:15:32 artom TIL
18:15:32 artom dansmith, oh, we have a specific health check "API"?
18:16:00 dansmith no, but kube has a standard format that people have proposed adding to our services
18:16:06 dansmith and I'm saying we should do _that_
18:16:21 artom Seems sensible to me
18:16:32 artom Better than doing it through the back door
18:16:51 artom What, that I agreed with you?
18:17:21 dansmith no, nevermind :)
18:17:29 artom Oh I know.
18:17:42 artom But... this being a professional channel and all.
18:22:30 cdent "we should do _that_"++
18:23:54 mriedem dansmith: no i haven't been following it
18:24:26 dansmith they already have an agent that sits on our bus and pings services, and unsurprisingly, have found that the version matters and things break if you don't have that right
18:25:11 dansmith and I just want to make sure we're not getting into a situation where we bless that kind of activity, and/or start trying to worry about external consumers of our RPC APIs and hold versions around because of what might be out in the wild
18:25:30 dansmith https://review.opendev.org/#/c/651140/
18:28:16 artom I mean, we could be evil and just remove ping()
18:28:38 dansmith artom: we need it for our own use
18:28:51 dansmith and note that that error dump was on the *nova* server side, as a result of pinging with an old version,
18:29:01 dansmith so that tool potentially generates bug reports for us
18:29:29 dansmith artom: compute pings conductor at startup and waits until it replies, in case it gets started earlier, since it'll fail without conductor being up
18:29:51 dansmith I told them about ping because they wanted to add their own ping-like thing to our rpcapi that they would use to do that,
18:30:07 dansmith so I was like "we already have this, so please don't add another one, but also... about that approach..."
18:31:11 artom Wait, ping() is in the conductor rpcapi, right?
18:31:19 artom So... how are they going to check for Nova API health?
18:31:32 artom Which I assume is what they most care about...
18:31:53 openstackgerrit Chris Dent proposed openstack/nova master: Make all functional tests reusable by other projects https://review.opendev.org/657659
18:33:27 artom Ah, it's in the base class
18:33:31 artom So all our services have it
18:34:18 dansmith they can't check api that way anyway
18:34:22 dansmith because api doesn't listen
18:34:32 dansmith but yeah, it's baked in for the services that do
18:35:24 artom So that's a good argument: you can only use ping() on services that listen to the message queue, which API doesn't.
18:35:32 artom And API is kinda important ;)
18:36:04 dansmith one could argue that you can test nova-api externally already ... via the api
18:36:21 dansmith but yeah, it's an argument.. the others I have are more important to me though :)
18:36:39 artom Right, I'm trying to think of what's important to *them* :)
18:36:47 artom Since you know, it's how you actually convince people ;)
18:39:01 artom Reading aspiers's reply on https://review.opendev.org/#/c/653707/2/specs/rpc-health-checks.rst@43, I can see where the self-healing SIG people are coming from. You want a think that tells you directly, with no proxy, whether a single service is healthy or not.
18:39:33 artom You then have your own intelligence on what to do. But you don't need further analysis to see where the actual failure is - well, less of it, anyways
18:39:49 artom Doing it through the RPC back door is one less layer to think about
18:40:24 dansmith it's dependent on the network and rabbit working though
18:40:45 dansmith and hides the visibility of what might be wrong on the network
18:41:06 dansmith like, if you get no route, you know the computer is down. if you get refused, you know the service is down
18:41:26 dansmith via rpc, you'll just not hear back, which could mean pretty much anything, including "the bus is overloaded right now"
18:42:03 artom Also true.
18:42:04 dansmith regardless, I don't want them to be on our bus as a matter of principle, aside from the fact that I think they'll get bad data from doing so
18:42:27 artom Anyways, I need to step out of this philosophical discussion, things need doing :)
18:42:31 artom Was fun thinking about, though
18:43:08 openstackgerrit Surya Seetharaman proposed openstack/nova master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648662
18:44:58 tssurya mriedem: thanks again for the detailed review ^. I totally appreciate it.
18:49:51 aspiers dansmith: thanks for the quick and helpful reply. I think I'm totally fine with your suggestion of opening up new dedicated HTTP endpoints for health-checking RPC-only services, and the advantages you cite make sense
18:53:22 aspiers dansmith, artom: FWIW, the problem the OSH folks encountered was not due to pinging with an old version - it was due to not knowing ping() exists and therefore deliberately invoking a non-existent RPC call, which (unsurprisingly) caused errors in the server logs
18:53:43 aspiers The errors were actually masked for a while, but surfaced when something in oslo logging changed
18:54:10 artom aspiers, ah, yeah that makes more sense with "Attempted method: pod_health_probe_method_ignore_errors"
18:54:13 dansmith aspiers: it's the same thing that will happen if you do use ping, but with the wrong version
18:54:19 artom That's definitely not a method we have ')
18:54:20 dansmith which was my point
18:54:24 aspiers dansmith: gotcha
18:55:07 artom It does add an external thing to the upgrade process. Nova internally can support N/N+1
18:55:36 artom With the proposed health check, suddenly this new external thing has be to upgraded at the same time as part of the Nova upgrade
18:55:41 dansmith artom: we actually only support N/N+1 internally for major versions on bridge releases
18:56:06 dansmith we support N.0-N.x on regular releases
18:56:22 aspiers artom: You lost me - what has to be upgraded?
18:56:29 dansmith this ^ :)
18:56:30 artom aspiers, and dansmith lost me :)

Earlier   Later