Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-07
17:10:39 stephenfin who's actually going to work on this extended health check?
17:11:05 stephenfin I mean, without someone signed up to do that work, isn't this all effectively bikeshedding?
17:11:34 artom bauzas, 💣
17:11:34 artom bauzas, 💣
17:11:38 bauzas \o/
17:12:05 artom stephenfin, look like gmann wil try it? I kinda wanna have a whack at it too
17:13:46 melwitt I'm also willing to help work on it (if need be) since I know at least some operators would want it to be a real health check
17:14:11 dansmith me too, I have some ideas on how to make it useful without requiring auth
17:14:20 bauzas again, a good healthcheck is something about L7 checks
17:14:21 bnemec Not a tldr, but most of the previous discussions are linked on https://storyboard.openstack.org/#!/story/2001439
17:14:22 gmann +1
17:14:26 bauzas what dansmith said
17:14:39 zigo There's already some extra parameters to give to /healthcheck with the Accept: header, so we could extend that and add more headers to do deeper checks...
17:15:12 bnemec There's also the detailed opt in the middleware that already allows operators to choose how much they want checked.
17:15:21 zigo X-OpenStack-Check-Type: full-check
17:15:22 artom So we're all just going to have a massive healthcheck party? :)
17:15:30 bauzas this being said, hitting the WSGI nova v2 app could be sufficient IMHO
17:15:51 artom bauzas, that's a '300 multiple choices', not '200 OK', and is different from other projects
17:15:51 bauzas and AFAIK, we can hit it without being auth'd
17:16:19 bauzas artom: I'm just talking of the healthcheck being part of the WSGI app
17:16:24 bauzas and returning 200
17:16:35 bauzas not operators using the existing app call
17:16:44 artom bauzas, you mean not using the middleware at all?
17:16:53 artom It'd be easier to make it extensible, yah
17:17:02 bauzas the middleware just verifies that the WSGI server runs, honestly
17:17:15 bauzas and even worst, it's now a new app with another endpoint
17:17:39 bauzas which means that the wsgi main app can silently drop and the healthcheck would still tell 'heh, fine enough'
17:18:09 bauzas having it as a separate app or a middleware just means (again) that you verify uswgi, that's it
17:25:11 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372
17:26:49 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372
17:32:07 bauzas gibi: won't be there tomorrow, yet again bank holiday
17:49:58 sean-k-mooney artom: some service allwo the middelware to be customisable other hardcode it
17:50:02 sean-k-mooney im not sure what nova does
17:50:16 artom sean-k-mooney, nova doesn't do anything yet.
17:50:27 artom (And with how things are going, probably never will :P )
17:50:35 sean-k-mooney artom: i mean in terms of middelware enabling
17:50:56 sean-k-mooney i cant rememebre if we hard code the list or if you can custimise it by a file
17:51:53 melwitt I don't think we need to be so negative about it, just want there to be some thought put into what we're doing before pulling the trigger
17:52:03 sean-k-mooney artom: we do some of it via https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini
17:53:15 sean-k-mooney im not entirly shure how that works but i think we can add more midelware or customise it via that file
17:53:20 melwitt sean-k-mooney: do you know a lot about the middleware? the question I have is would it in anyway preclude us from doing a real health check in the future, such as accessing a cache built lazily in the background containing db/mq etc healthy kind of info?
17:53:44 sean-k-mooney melwitt: i do not
17:53:55 sean-k-mooney i assume we coudl swap out the implemenation in the future
17:54:00 sean-k-mooney with our own version
17:54:25 sean-k-mooney provided it still returned the correct info
17:55:03 sean-k-mooney melwitt: i assume you would want to check if we can still acess the db and if the conductors are alive
17:55:10 sean-k-mooney or something like that
17:55:41 melwitt yeah, something like that. dansmith idea was to gather those lazily while normal requests are going, and an unauth'ed healthcheck could check the cache
17:55:49 sean-k-mooney that sound like it shoudl also be doable i think the current healthcheck midelware would jsut serve to say hay the api is running
17:56:18 melwitt right, as-is it checks whether wsgi is working
17:56:24 sean-k-mooney melwitt: dont we have heartbeats for each service in service groups or something
17:56:33 sean-k-mooney and the status is stored in the db
17:56:47 melwitt and we were talking about having that be the first version, then next we'd extend it to do real health checks
17:57:34 melwitt and we're looking to have a plan for this and the first thing to know is whether going forward with the middleware way of doing it would lock us in a corner in a way we couldn't extend it
17:57:49 melwitt or if we need to expose /healthcheck in a different way
17:58:15 melwitt we do have heartbeats for each service via the service groups stuff yes
17:59:15 sean-k-mooney melwitt: so this is how neturon enables it
17:59:17 sean-k-mooney https://opendev.org/openstack/neutron/src/branch/master/etc/api-paste.ini#L47-L48
17:59:23 melwitt gmann_afk is going to investigate next week but since you mentioned the middleware I was curious if you knew more about it already
17:59:26 sean-k-mooney so we could just copy past that
17:59:40 sean-k-mooney and we woudl get healtcheck endpoint
17:59:42 melwitt yeah that's what's proposed already
17:59:52 sean-k-mooney operator can already do that too
17:59:57 sean-k-mooney they just need to add it
18:00:12 sean-k-mooney but then we coudl cange the class to a nova one later
18:00:42 sean-k-mooney so paste.filter_factory = oslo_middleware:Healthcheck.factory would be paste.filter_factory = nova.middelware:Healthcheck.factory
18:00:45 sean-k-mooney or something like that
18:00:54 melwitt ok
18:01:21 melwitt and then present the same header/response options, same interface
18:01:52 melwitt I wonder if we can subclass the middleware, that might be likely. anyway, gmann_afk is gonna figure it out :)
18:02:32 sean-k-mooney cool it looks like this is the midealware code https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck
18:02:55 sean-k-mooney we could derive form the Healthcheck class i thikn
18:03:28 bnemec Healthcheck is pluggable.
18:03:33 sean-k-mooney then override process_request https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck/__init__.py#L544
18:04:02 sean-k-mooney bnemec: yes nova does not use the middelware at all by default
18:04:37 sean-k-mooney bnemec: but if we start enabling it we would like something more advanced that check if critical serivce like the scheduler and condcutoer were alive
18:04:45 sean-k-mooney rather then jsut saying yes the api is alive
18:05:03 bnemec Right, I'm saying you don't need to reimplement it, just write a plugin for those checks.
18:05:34 sean-k-mooney bnemec: right and we woudl do that by inheritinbg and then overriding process_request?
18:05:45 sean-k-mooney bnemec: do you have an example we could follow?
18:06:05 bnemec No, it's a stevedore-based plugin system. You just register your plugin in the appropriate namespace.
18:06:27 bnemec https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/healthcheck/__init__.py#L388
18:06:32 sean-k-mooney bnemec: sure but we dont kwno what the plugin should do
18:06:40 sean-k-mooney e.g. what i shoudl retrun
18:06:51 sean-k-mooney what the api for this is as a user
18:07:10 bnemec https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/healthcheck/pluginbase.py
18:07:34 sean-k-mooney ah perfect
18:07:42 sean-k-mooney so we have to retrun a HealthcheckResult
18:08:18 bnemec Right
18:08:37 sean-k-mooney there are some examples of the respocnes here too https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck/__init__.py#L68-L163
18:10:55 sean-k-mooney bnemec: anyway im sure gmann_afk will figure it out or be in touch
18:11:32 sean-k-mooney bnemec: what i was more interested was if there was a standard way to report erros and infomation when it not healty
18:12:25 sean-k-mooney if not and we ant it to be consumable in an automated fashion then we would have too treat it like a version notification or other api resonce and managen and document it in our api ref
18:13:31 sean-k-mooney https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck/disable_by_file.py i guess are the example plugins
18:13:35 bnemec Yep, versioning these checks is definitely something we should work out on the spec.
18:14:11 bnemec Looks like it, yeah.
18:15:20 sean-k-mooney yeah so other then establishing conventiosn this look like its not much code to add on our side to have a funtioning plugin
18:15:41 sean-k-mooney then we can just make the healt check more advanced and imporve the info we return iteritivly
18:19:20 dansmith I'm surprised that nobody thinks that exposing python version, gcc version, distro, version, kernel version, etc to everyone unauthenticated is not a huge problem
18:20:02 dansmith because all those things are included in the healthcheck response without any recourse, as far as I can tell

Earlier   Later