Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-07
17:25:11 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372
17:26:49 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372
17:32:07 bauzas gibi: won't be there tomorrow, yet again bank holiday
17:49:58 sean-k-mooney artom: some service allwo the middelware to be customisable other hardcode it
17:50:02 sean-k-mooney im not sure what nova does
17:50:16 artom sean-k-mooney, nova doesn't do anything yet.
17:50:27 artom (And with how things are going, probably never will :P )
17:50:35 sean-k-mooney artom: i mean in terms of middelware enabling
17:50:56 sean-k-mooney i cant rememebre if we hard code the list or if you can custimise it by a file
17:51:53 melwitt I don't think we need to be so negative about it, just want there to be some thought put into what we're doing before pulling the trigger
17:52:03 sean-k-mooney artom: we do some of it via https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini
17:53:15 sean-k-mooney im not entirly shure how that works but i think we can add more midelware or customise it via that file
17:53:20 melwitt sean-k-mooney: do you know a lot about the middleware? the question I have is would it in anyway preclude us from doing a real health check in the future, such as accessing a cache built lazily in the background containing db/mq etc healthy kind of info?
17:53:44 sean-k-mooney melwitt: i do not
17:53:55 sean-k-mooney i assume we coudl swap out the implemenation in the future
17:54:00 sean-k-mooney with our own version
17:54:25 sean-k-mooney provided it still returned the correct info
17:55:03 sean-k-mooney melwitt: i assume you would want to check if we can still acess the db and if the conductors are alive
17:55:10 sean-k-mooney or something like that
17:55:41 melwitt yeah, something like that. dansmith idea was to gather those lazily while normal requests are going, and an unauth'ed healthcheck could check the cache
17:55:49 sean-k-mooney that sound like it shoudl also be doable i think the current healthcheck midelware would jsut serve to say hay the api is running
17:56:18 melwitt right, as-is it checks whether wsgi is working
17:56:24 sean-k-mooney melwitt: dont we have heartbeats for each service in service groups or something
17:56:33 sean-k-mooney and the status is stored in the db
17:56:47 melwitt and we were talking about having that be the first version, then next we'd extend it to do real health checks
17:57:34 melwitt and we're looking to have a plan for this and the first thing to know is whether going forward with the middleware way of doing it would lock us in a corner in a way we couldn't extend it
17:57:49 melwitt or if we need to expose /healthcheck in a different way
17:58:15 melwitt we do have heartbeats for each service via the service groups stuff yes
17:59:15 sean-k-mooney melwitt: so this is how neturon enables it
17:59:17 sean-k-mooney https://opendev.org/openstack/neutron/src/branch/master/etc/api-paste.ini#L47-L48
17:59:23 melwitt gmann_afk is going to investigate next week but since you mentioned the middleware I was curious if you knew more about it already
17:59:26 sean-k-mooney so we could just copy past that
17:59:40 sean-k-mooney and we woudl get healtcheck endpoint
17:59:42 melwitt yeah that's what's proposed already
17:59:52 sean-k-mooney operator can already do that too
17:59:57 sean-k-mooney they just need to add it
18:00:12 sean-k-mooney but then we coudl cange the class to a nova one later
18:00:42 sean-k-mooney so paste.filter_factory = oslo_middleware:Healthcheck.factory would be paste.filter_factory = nova.middelware:Healthcheck.factory
18:00:45 sean-k-mooney or something like that
18:00:54 melwitt ok
18:01:21 melwitt and then present the same header/response options, same interface
18:01:52 melwitt I wonder if we can subclass the middleware, that might be likely. anyway, gmann_afk is gonna figure it out :)
18:02:32 sean-k-mooney cool it looks like this is the midealware code https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck
18:02:55 sean-k-mooney we could derive form the Healthcheck class i thikn
18:03:28 bnemec Healthcheck is pluggable.
18:03:33 sean-k-mooney then override process_request https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck/__init__.py#L544
18:04:02 sean-k-mooney bnemec: yes nova does not use the middelware at all by default
18:04:37 sean-k-mooney bnemec: but if we start enabling it we would like something more advanced that check if critical serivce like the scheduler and condcutoer were alive
18:04:45 sean-k-mooney rather then jsut saying yes the api is alive
18:05:03 bnemec Right, I'm saying you don't need to reimplement it, just write a plugin for those checks.
18:05:34 sean-k-mooney bnemec: right and we woudl do that by inheritinbg and then overriding process_request?
18:05:45 sean-k-mooney bnemec: do you have an example we could follow?
18:06:05 bnemec No, it's a stevedore-based plugin system. You just register your plugin in the appropriate namespace.
18:06:27 bnemec https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/healthcheck/__init__.py#L388
18:06:32 sean-k-mooney bnemec: sure but we dont kwno what the plugin should do
18:06:40 sean-k-mooney e.g. what i shoudl retrun
18:06:51 sean-k-mooney what the api for this is as a user
18:07:10 bnemec https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/healthcheck/pluginbase.py
18:07:34 sean-k-mooney ah perfect
18:07:42 sean-k-mooney so we have to retrun a HealthcheckResult
18:08:18 bnemec Right
18:08:37 sean-k-mooney there are some examples of the respocnes here too https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck/__init__.py#L68-L163
18:10:55 sean-k-mooney bnemec: anyway im sure gmann_afk will figure it out or be in touch
18:11:32 sean-k-mooney bnemec: what i was more interested was if there was a standard way to report erros and infomation when it not healty
18:12:25 sean-k-mooney if not and we ant it to be consumable in an automated fashion then we would have too treat it like a version notification or other api resonce and managen and document it in our api ref
18:13:31 sean-k-mooney https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck/disable_by_file.py i guess are the example plugins
18:13:35 bnemec Yep, versioning these checks is definitely something we should work out on the spec.
18:14:11 bnemec Looks like it, yeah.
18:15:20 sean-k-mooney yeah so other then establishing conventiosn this look like its not much code to add on our side to have a funtioning plugin
18:15:41 sean-k-mooney then we can just make the healt check more advanced and imporve the info we return iteritivly
18:19:20 dansmith I'm surprised that nobody thinks that exposing python version, gcc version, distro, version, kernel version, etc to everyone unauthenticated is not a huge problem
18:20:02 dansmith because all those things are included in the healthcheck response without any recourse, as far as I can tell
18:20:18 artom dansmith, only if you configure detailed=True in the paste file
18:20:26 artom Which nobody should ever do
18:20:45 dansmith ah, okay, I thought that was a query param
18:21:03 artom It isn't - at least, based on what I saw
18:22:07 mriedem if your healthcheck api would give me some credentials i could do some health checking of my own, thanks
18:22:10 dansmith so the details of the healthchecks aren't either, other than the reason thing
18:25:41 dansmith okay, so in that case, the config causes us to return different data types
18:25:50 artom My own personal healthcheck api is to down a pint of beer. Still awak = 200 OK, pass out = 503 service unavailable
18:26:04 dansmith which is unfortunate.. if nobody should ever turn on detailed=true, then hopefully nothing will ever see the conflicting structure, but..
18:26:38 artom dansmith, I suppose you could if you're absolutely certain your Nova API endpoints aren't accessible from any unsafe network
18:26:48 artom Like, you trust your firewalls 110%
18:27:04 dansmith hmm, I don't see the if detailed switch in the html version tough
18:28:09 dansmith so the html version is always detailed?
18:28:43 dansmith or maybe the default template doesn't include those things?
18:29:39 dansmith oh I see, conditionals in the template, although I can't eye-parse what that all looks like by default
18:30:30 dansmith ah, I see the if detailed at the top
18:33:24 sean-k-mooney dansmith: so yes detailed is a vaild security consern. im not sure if we are required to exose any of the detailed info you listed
18:33:46 sean-k-mooney dansmith: e.g. if we can retrun different detailed infor per service that is less scare form a security point of view
18:33:56 sean-k-mooney liek the equivalent of a service list
18:34:03 dansmith sean-k-mooney: no, it looks like that's all off unless show_details, but it's one switch to turn on details which then exposes a ton of internals, which may surprise someone
18:35:14 sean-k-mooney ok so other then the last hour or so i have never bother looking at this in detail other then being aware it was configurable via api-paste.ini
18:35:43 sean-k-mooney so im ok with the idea of a healt check api endpoint. but im not sure it we want to use teh default implemantion
18:35:48 dansmith right, well, the fact that it is configurable with api-paste means this is not an emergency :)
18:35:51 sean-k-mooney them more i lean about it the mor ei think no
18:36:02 sean-k-mooney yep
18:36:19 sean-k-mooney operatoer can add it today if they want. if the should is a different matter
18:36:37 dansmith so, it's totally fine to have a spec, get people on the same page here, and make sure we have a plan for how this happens now, soon, and later
18:36:39 sean-k-mooney i thikn with out detailed=true its ok but does not tell you much

Earlier   Later