Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-07
16:15:01 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Rewrite 'nova-manage db' docs to use proper directives https://review.opendev.org/677509
16:15:02 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Rewrite 'nova-manage placement' docs https://review.opendev.org/726193
16:15:02 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Rewrite 'nova-manage api_db', 'cell_v2' docs https://review.opendev.org/726192
16:15:03 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Document options for nova-*proxy https://review.opendev.org/726195
16:15:03 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Only use section headers for HTML https://review.opendev.org/726194
16:44:23 openstackgerrit Shi Yan proposed openstack/nova master: Fix aggregate placement sync issue https://review.opendev.org/726201
16:47:07 openstackgerrit Merged openstack/os-vif master: Remove six https://review.opendev.org/724966
16:47:08 openstackgerrit Merged openstack/os-vif master: Update master for stable/ussuri https://review.opendev.org/724714
16:47:09 openstackgerrit Merged openstack/os-vif master: Add Python3 victoria unit tests https://review.opendev.org/724715
17:01:56 artom gmann, continuing from #-meeting, yeah, sounds like more discussion would have been nice
17:02:26 gmann stephenfin: gibi py3.8 functional job as voting on check and gate pipeline - https://review.opendev.org/#/c/725372/
17:02:27 artom I just... I don't get the opposition. Like, yeah, the name could have been better chosen, and yeah, we may want to improve it in the future
17:02:56 artom But as long as we document what it does for the former, and make it evolve-able for the latter, why not do it?
17:03:14 stephenfin gmann: Oh, it works now? I thought there were issues with mock from Python 3.8
17:03:50 melwitt artom: well, I think we need to know whether it's extendable right? we don't fully understand that yet, do we?
17:04:07 gmann artom: i will say if old things have limit and need more clarity then documenting those is fine but any future thing and we can better design so better design is my preference than going with document-limits
17:04:44 artom gmann, agreed, but we were pre-empted by projects with lesser standards than ours ;)
17:05:08 gmann stephenfin: i think it worked. let me again check log that it is correctly doing py3.8
17:05:15 melwitt like can we build a health data cache in that middleware or is there something about it that would preclude doing that? I don't think basic investigation has been done
17:05:31 artom melwitt, so, I'd be OK with just 2 URLs (and I know dansmith will hate me for that) - /healthcheck for unauthed, unversioned HTTP alice checks, and something like /status that's authed and versioned and we can evolve
17:05:59 melwitt that's backward, /status should be the ping check
17:06:18 gmann artom: that is something to fix since starting of openstack :), making uniform things across all projects is still an issue :).
17:06:23 melwitt and so to me seems worse to do that, expose two things that are doing the opposite of their names
17:07:09 gmann stephenfin: it does and worked fine - https://zuul.opendev.org/t/openstack/build/bc80a0fe21354e10b81b1e4a4db35356/log/job-output.txt#1172
17:07:55 artom melwitt, so find a way for /healthcheck to do both?
17:08:48 melwitt yeah, at least do basic due diligence on it. look at the middleware enough to determine whether we could extend the health check to actually be a health check. or if we would need to expose it some other way in order to be able to extend it. that's mho
17:09:47 stephenfin gmann: okay. comments left
17:10:17 gmann let me do some poc for extend things.but may be next week i will be able to do. so that we can check if current proposed things can be done in compatible way with extendable.
17:10:39 stephenfin who's actually going to work on this extended health check?
17:11:05 stephenfin I mean, without someone signed up to do that work, isn't this all effectively bikeshedding?
17:11:34 artom bauzas, 💣
17:11:34 artom bauzas, 💣
17:11:38 bauzas \o/
17:12:05 artom stephenfin, look like gmann wil try it? I kinda wanna have a whack at it too
17:13:46 melwitt I'm also willing to help work on it (if need be) since I know at least some operators would want it to be a real health check
17:14:11 dansmith me too, I have some ideas on how to make it useful without requiring auth
17:14:20 bauzas again, a good healthcheck is something about L7 checks
17:14:21 bnemec Not a tldr, but most of the previous discussions are linked on https://storyboard.openstack.org/#!/story/2001439
17:14:22 gmann +1
17:14:26 bauzas what dansmith said
17:14:39 zigo There's already some extra parameters to give to /healthcheck with the Accept: header, so we could extend that and add more headers to do deeper checks...
17:15:12 bnemec There's also the detailed opt in the middleware that already allows operators to choose how much they want checked.
17:15:21 zigo X-OpenStack-Check-Type: full-check
17:15:22 artom So we're all just going to have a massive healthcheck party? :)
17:15:30 bauzas this being said, hitting the WSGI nova v2 app could be sufficient IMHO
17:15:51 bauzas and AFAIK, we can hit it without being auth'd
17:15:51 artom bauzas, that's a '300 multiple choices', not '200 OK', and is different from other projects
17:16:19 bauzas artom: I'm just talking of the healthcheck being part of the WSGI app
17:16:24 bauzas and returning 200
17:16:35 bauzas not operators using the existing app call
17:16:44 artom bauzas, you mean not using the middleware at all?
17:16:53 artom It'd be easier to make it extensible, yah
17:17:02 bauzas the middleware just verifies that the WSGI server runs, honestly
17:17:15 bauzas and even worst, it's now a new app with another endpoint
17:17:39 bauzas which means that the wsgi main app can silently drop and the healthcheck would still tell 'heh, fine enough'
17:18:09 bauzas having it as a separate app or a middleware just means (again) that you verify uswgi, that's it
17:25:11 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372
17:26:49 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372
17:32:07 bauzas gibi: won't be there tomorrow, yet again bank holiday
17:49:58 sean-k-mooney artom: some service allwo the middelware to be customisable other hardcode it
17:50:02 sean-k-mooney im not sure what nova does
17:50:16 artom sean-k-mooney, nova doesn't do anything yet.
17:50:27 artom (And with how things are going, probably never will :P )
17:50:35 sean-k-mooney artom: i mean in terms of middelware enabling
17:50:56 sean-k-mooney i cant rememebre if we hard code the list or if you can custimise it by a file
17:51:53 melwitt I don't think we need to be so negative about it, just want there to be some thought put into what we're doing before pulling the trigger
17:52:03 sean-k-mooney artom: we do some of it via https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini
17:53:15 sean-k-mooney im not entirly shure how that works but i think we can add more midelware or customise it via that file
17:53:20 melwitt sean-k-mooney: do you know a lot about the middleware? the question I have is would it in anyway preclude us from doing a real health check in the future, such as accessing a cache built lazily in the background containing db/mq etc healthy kind of info?
17:53:44 sean-k-mooney melwitt: i do not
17:53:55 sean-k-mooney i assume we coudl swap out the implemenation in the future
17:54:00 sean-k-mooney with our own version
17:54:25 sean-k-mooney provided it still returned the correct info
17:55:03 sean-k-mooney melwitt: i assume you would want to check if we can still acess the db and if the conductors are alive
17:55:10 sean-k-mooney or something like that
17:55:41 melwitt yeah, something like that. dansmith idea was to gather those lazily while normal requests are going, and an unauth'ed healthcheck could check the cache
17:55:49 sean-k-mooney that sound like it shoudl also be doable i think the current healthcheck midelware would jsut serve to say hay the api is running
17:56:18 melwitt right, as-is it checks whether wsgi is working
17:56:24 sean-k-mooney melwitt: dont we have heartbeats for each service in service groups or something
17:56:33 sean-k-mooney and the status is stored in the db
17:56:47 melwitt and we were talking about having that be the first version, then next we'd extend it to do real health checks
17:57:34 melwitt and we're looking to have a plan for this and the first thing to know is whether going forward with the middleware way of doing it would lock us in a corner in a way we couldn't extend it
17:57:49 melwitt or if we need to expose /healthcheck in a different way
17:58:15 melwitt we do have heartbeats for each service via the service groups stuff yes
17:59:15 sean-k-mooney melwitt: so this is how neturon enables it
17:59:17 sean-k-mooney https://opendev.org/openstack/neutron/src/branch/master/etc/api-paste.ini#L47-L48
17:59:23 melwitt gmann_afk is going to investigate next week but since you mentioned the middleware I was curious if you knew more about it already
17:59:26 sean-k-mooney so we could just copy past that
17:59:40 sean-k-mooney and we woudl get healtcheck endpoint
17:59:42 melwitt yeah that's what's proposed already
17:59:52 sean-k-mooney operator can already do that too
17:59:57 sean-k-mooney they just need to add it
18:00:12 sean-k-mooney but then we coudl cange the class to a nova one later
18:00:42 sean-k-mooney so paste.filter_factory = oslo_middleware:Healthcheck.factory would be paste.filter_factory = nova.middelware:Healthcheck.factory
18:00:45 sean-k-mooney or something like that
18:00:54 melwitt ok
18:01:21 melwitt and then present the same header/response options, same interface
18:01:52 melwitt I wonder if we can subclass the middleware, that might be likely. anyway, gmann_afk is gonna figure it out :)
18:02:32 sean-k-mooney cool it looks like this is the midealware code https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck
18:02:55 sean-k-mooney we could derive form the Healthcheck class i thikn

Earlier   Later