Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-05
19:59:37 artom *They
19:59:50 bnemec Basically the ability to do more than just HTTP API checks, say db, messaging, maybe libvirt for Nova.
19:59:51 gmann artom: doc SIG is there.
20:00:23 artom gmann, so they'd be the ones coming up with where to put this doc, if we get openstack-wide uniformity
20:00:35 gmann bnemec: yeah, libvirt specific can be done as nova defined backend. but yeah good to have db, messaging things on oslo.middleware
20:01:05 artom gmann, but... which DB? If the deployment is multicell, just the nova_api one?
20:01:21 artom It's a rathole that I'm not sure we want to go down into, for something this simple
20:01:30 artom /healthcheck is better than nothing, even if it's not perfect
20:01:38 gmann artom: is it technical writing SIG not doc sig - https://governance.openstack.org/sigs/
20:01:48 bnemec artom: That was our conclusion too last time we discussed it.
20:02:11 bnemec If we try to boil the ocean and healthcheck all the things this will never get done at all.
20:02:51 gmann yeah deployment model is one challenge for such checks
20:03:16 artom gmann, and presumably we can improve this in the future, no?
20:03:34 artom So maybe just version is (somehow) for now, and leave is as the simple middleware thing
20:03:48 artom And eventually add /healthcheck/v2 or whatever, which can be smarter
20:04:01 artom *version it
20:04:22 gmann url can be same and configured backends by operator can enhance the checks.
20:07:50 gmann zigo: i think we need separate new config on nova side if we want to disable it right? bnemec can correct me.
20:08:09 gmann otherwise if middleware is added then it is enabled by defualt
20:08:12 gmann default
20:10:01 bnemec If we want to make healthcheck runtime configurable then it would be preferrable if we could do it in oslo. Otherwise you end up with a half dozen differen opts in different projects designed to do the same thing.
20:11:08 gmann bnemec: yeah. i am worried to have enable by default because it is unauthenticated endpoint.
20:12:21 artom gmann, what's the harm though?
20:13:08 artom Like, OK, anyone can spam that, but so can anyone spam any other API and get a flood of 403s back
20:13:33 artom The disk of DOS (if that's your concern) is greater with the second one, because keystone has to actually do work
20:13:58 artom *risk of
20:14:42 bnemec My understanding was the unauthenticated healthchecks are generally okay. Most of the stuff calling them doesn't have auth capability anyway, so you don't have much choice.
20:15:07 gmann artom: well it is still security risk to know services running on that node
20:15:28 artom gmann, security by obscurity is Real Bad (TM) ;)
20:15:49 gmann yeah i am not suggesting to make it auth but disable by default and let operator to enable it by knowing this is unauth things.
20:15:50 artom If knowing that nova-api is running somewhere is a risk, we're doing something horribly wrong
20:16:48 bnemec How would you prevent them from knowing the service is there? A simple port scan would tell them that.
20:17:07 artom gmann, off by default works for me. But that only reinforces the doc problem - admins have to know how to enable it, and more fundamentally that it's available
20:18:18 gmann artom: doc anyways we might need to tell at least it is there now in nova and refer oslo doc for all details and config options.
20:19:56 gmann and there is no harm of doing those doc of 2-3 line in all projects start implementing that and point to oslo doc for details. because we do not have any central doc place now for openstack operators/admin etc as general
20:21:36 artom gmann, fair point
20:21:58 gmann knowing service can be hacked by multiple way. but is it fine to have this unatuh endpoint for operator without no way to disable it ? may be zigo can tell.
20:22:36 gmann i felt giving the enable/disable option for them does not harm. disable by default or enable that can be disucssed whatever is more common.
20:22:47 zigo The thing is, in a typical deployment, haproxy will be the one holding the API on a public IP.
20:23:16 zigo So, an operator could disable /healthcheck on haproxy because it's in front.
20:23:37 zigo The /healthcheck from the nova-api would typically *not* be reachable from the outside.
20:25:14 gmann but it can be reachable to anyone for other deployment which are not under haproxy or other layer of direction/load balancer etc
20:25:39 gmann disabling on haproxy side is all good i think.
20:26:04 zigo We have the / which needs to be enabled unauth anyways...
20:27:34 artom gmann, I still strongly disagree that it's a legit security risk - or if it is, we've messed up badly
20:28:01 artom Our security should be that nova-api is safe, not that we need to hide it :)
20:31:11 dansmith I haven't been paying attention here
20:31:24 dansmith but anything that generates db or mq traffic should be authenticated, I think
20:31:30 dansmith even if read-only
20:31:36 artom dansmith, it doesn't
20:32:03 dansmith okay, so a only a healthcheck of "can you respond to http requests" then?
20:32:12 artom dansmith, yep
20:32:18 dansmith because generally those expand to "can you talk to your database?" etc
20:32:49 artom dansmith, yeah, gmann wanted to do that, but IMO it's complexity we don't necessarily need (right now)
20:33:02 artom For instance, which database? api? cell1? What about multicell?
20:33:36 dansmith artom: right, so if it's going to grow to that (which most people will want it it, AFAIK) you should consider it needing to be auth'd
20:34:04 gmann dansmith: yeah, my idea is to expand that to DB checks also
20:34:06 artom dansmith, no argument there from me
20:34:07 dansmith because I would expect returning a dict of each db (api, cell1, cell2, etc) as well as each mq would be exactly what people want
20:34:16 dansmith gmann: right
20:34:38 dansmith artom: so are you arguing that if you have no auth, you get a "yes I'm here but that's all I will tell you" response/
20:36:04 gmann artom: and i will say we finalize it now what all things we want to do with healthcheck instead of expending it later which make lot of things change like auth, enable/disable things etc. but only https request checks we are like 'everything is fine as it is'
20:36:10 artom dansmith, it was more of a response to gmann's security risk argument - that just knowing a service is running somewhere is not a security risk, or if it is, we've done something horribly wrong
20:36:48 artom gmann, you mean future proofing it?
20:37:08 bnemec We've been discussing them for years and nobody has actually implemented anything.
20:37:16 gmann that is why i suggested to have spec and then discuss what all things we want to do with that.
20:37:49 dansmith gmann: we should never expect to design something like that once and forever, but agree we should not design it before making it useful, only to have to revise
20:37:55 artom gmann, yeah, I get that... It's just... looks to me like it'll get bogged down in the kind of discussion that we're having :)
20:38:27 gmann bnemec: yeah, it has more attention now as zigo also brought it up for doing it for all projects as community goal which is good idea.
20:38:42 artom So I'd be more in the campg of "do something easy and useful *now*, trying to future-proof it", instead "do something perfect later"
20:38:44 bnemec It's not the first time that has happened either. I proposed it in Berlin as a community goal.
20:38:46 dansmith artom: so you leave all your computer's ports open while connected directly to the internet and depend on windows/samba auth to keep people out of your files right? :)
20:39:23 dansmith expose ssh to everyone, relying on your password and the patched-ness of your sshd, etc
20:40:12 artom I mean, I've almost done the latter
20:40:34 gmann artom: but it can break things later say, it is enable by default and if we extend in future for things need more care then we may want to disable it by default.
20:40:54 gmann bnemec: ah, did not know that :)
20:40:59 artom gmann, I'm not arguing off by default :)
20:41:01 dansmith artom: but you wouldn't tell everyone to do that as matter of policy right? there're legit reasons to limit scope and reach, not all of which are "have done something terribly wrong"
20:42:09 artom dansmith, yes, I understand reducing target aread
20:42:11 artom *area
20:46:15 melwitt gmann: just a note that we recently made version discovery unauth'ed https://review.opendev.org/685181 if that is relevant at all
20:50:14 gmann melwitt: yeah that is fine but in case of healthcheck which can be extended on more deeper check of DB etc so keeping it unauth and enable by default with no way to disable is my concern. at least we can make it disable by default and let operator to enable it knowingly.
20:50:27 melwitt k
20:50:30 gmann with good help msg in config option.
20:50:57 dansmith agree
21:08:36 zigo artom: I think writing what you describe as the perfect /healthcheck should prevent enabling what we already have.
21:09:11 artom zigo, wait, why is that addressed to me? I'm agreeing with bnemec and you, somethign good now is better than something perfect later :)
21:09:19 zigo Don't get me wrong, I'd love to have what you described.
21:09:22 zigo Oh, wrong person? :)
21:09:31 artom dansmith I think
21:09:35 zigo Maybe ...
21:09:42 artom With this dict that gives cell and mq status for all cells and mqs
21:09:45 artom *his
21:09:47 zigo Everyone keeps disturbing me and I can't read IRC properly! :)
21:10:04 zigo Sorry then.
21:10:25 artom No worries :)
21:16:31 openstackgerrit Merged openstack/nova master: Remove future imports https://review.opendev.org/714675
23:45:22 openstackgerrit Merged openstack/nova master: Follow-up for NUMA live migration functional tests https://review.opendev.org/724903
23:57:35 openstackgerrit Artom Lifshitz proposed openstack/nova stable/ussuri: Functional tests for NUMA live migration https://review.opendev.org/725736
#openstack-nova - 2020-05-06
06:34:29 openstackgerrit Jie Li proposed openstack/nova master: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/722187

Earlier   Later