| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-13 | |||
| 13:50:38 | sfinucan | jaypipes: You don't get to escape either :P https://review.openstack.org/361140 | |
| 13:51:07 | cdent | ha | |
| 13:51:11 | artom | And I think the backend is basically a dict: https://github.com/openstack/oslo.cache/blob/master/oslo_cache/backends/dictionary.py | |
| 13:51:50 | artom | So unless the REST API request just happened to land on the same process that eventually handles the metadata request, we need some form of IPC | |
| 13:52:44 | mriedem | artom: default cache is just in-memory per process dict? | |
| 13:52:51 | mriedem | i think it's dogpile yeah? | |
| 13:52:59 | artom | mriedem, that's what I gather, yeah | |
| 13:55:02 | mriedem | well this is a sticky wicket | |
| 13:55:19 | mriedem | we have to look up an instance when it's not cache, or the cache is expired | |
| 13:55:48 | mriedem | i was thinking we could have set a flag in instance_extra that would tell metadata api to refresh the cache, but that defeats the purpose because you have to do the db lookup to find out | |
| 13:56:26 | mriedem | default cache expiration is 15 seconds right? | |
| 13:56:48 | snikitin_ | sfinucan: wow. Thank you for update! btw 'legacy' looks good because this behavior it really a little bit strange | |
| 13:56:57 | mriedem | does the guest need to see the tagged block device immediately? | |
| 13:57:10 | artom | mriedem, ah, you're saying just wait it out? | |
| 13:57:13 | frickler | mriedem: regarding your ML question, johnthetubaguy has a patch up to clarify things: https://review.openstack.org/#/c/438887/3 | |
| 13:57:17 | mriedem | artom: that's what i'm asking | |
| 13:57:33 | mriedem | artom: i.e. this is already a latent issue right? | |
| 13:57:39 | sfinucan | snikitin_: no problem | |
| 13:57:55 | mriedem | if your guest is relying on changes via metadata api, and it's already cached on a timer, then your guest logic probably already needs to poll for changes | |
| 13:58:03 | mriedem | i don't really know the use cases though | |
| 13:58:12 | sfinucan | snikitin_: Personally I'd like to drop that policy because it makes no sense. If you care about NUMA affinity, you should have 100% NUMA-aware nodes. If you don't, you clearly don't care enough :) | |
| 13:58:17 | johnthetubaguy | frickler: mriedem: thats right, I new that was familiar, I think bauzas said he would fix up that patch. | |
| 13:58:25 | sfinucan | But this is a good half way measure to avoid breaking existing deployments | |
| 13:58:27 | artom | mriedem, I suppose it is... but... well, did we ever have a user case where the metadata changed during an instance's lifetime? | |
| 13:59:00 | mriedem | artom: i'm sure there are all sorts of things that change | |
| 13:59:04 | snikitin_ | sfinucan: yea, me too. But this damn backward compatibility... | |
| 13:59:14 | mriedem | but let me check | |
| 13:59:38 | bauzas | johnthetubaguy: it just went out of my radar, because I'm not the Gerrit owner :( | |
| 13:59:51 | johnthetubaguy | bauzas: yeah, I have done that a few times also | |
| 13:59:54 | sfinucan | snikitin_: I know, right? How dare our customers ask for backwards compatibility :P Let's hope that naming clears up everything | |
| 13:59:54 | mriedem | artom: security groups could change, password could change | |
| 14:00:10 | mriedem | artom: network metadata could change couldn't it depending on attaching/detaching ports? | |
| 14:00:25 | mriedem | not to mention dynamic vendor data | |
| 14:00:27 | bauzas | mriedem: if your ML thread is related to my az spec, nevermind it, we could just wait for Queens | |
| 14:00:39 | mriedem | bauzas: it was the impetus yes | |
| 14:00:49 | mriedem | i never knew about default_scheduler_zone | |
| 14:00:54 | artom | mriedem, that's probably correct... | |
| 14:01:05 | bauzas | mriedem: default_schedul/e/_zone but yeah | |
| 14:01:15 | mriedem | artom: so if you accept as a limitation that you're not getting the latest changes immediately when using metadata api, then this is probably ok | |
| 14:01:16 | bauzas | mriedem: hope my reply made things clearer for you | |
| 14:01:32 | mriedem | bauzas: it made me feel like there was a purpose | |
| 14:01:43 | mriedem | bauzas: however, the help text on those options is still murky | |
| 14:01:43 | artom | mriedem, I think that's acceptable in the short term, at least | |
| 14:01:46 | mriedem | and should probably be updated | |
| 14:01:53 | bauzas | mriedem: yup, hence johnthetubaguy and frickler point | |
| 14:01:55 | mriedem | bauzas: plus, | |
| 14:01:58 | snikitin_ | sfinucan: :D | |
| 14:01:58 | artom | mriedem, and if someone complains we can always do a "real" cache later | |
| 14:02:01 | mriedem | this big fat scary warning https://docs.openstack.org/developer/nova/aggregates.html#availability-zones-azs | |
| 14:02:04 | mriedem | should probably be in the api-ref | |
| 14:02:07 | mriedem | where api users would see it | |
| 14:02:10 | mriedem | rather than way down in devref | |
| 14:02:11 | artom | Or they can run memcache and we can add a bit where we invalidate it | |
| 14:02:29 | bauzas | mriedem: well, when I wrote by that time, I wasn't seasoned with api-ref :) | |
| 14:02:39 | mriedem | artom: memcache doesn't help the timing problem right? just the consistency issue, | |
| 14:02:40 | bauzas | it should even be in the operators guide | |
| 14:02:51 | mriedem | i.e. depending on which metadata api worker i hit with my request, i could get different results | |
| 14:03:02 | bauzas | anyway, spec deadline day, I'm turned into review day for me | |
| 14:03:07 | mriedem | or would memcache be updated globally for all? i suppose it would | |
| 14:03:09 | mriedem | nevermind then | |
| 14:03:18 | artom | mriedem, I'm assuming memcache would be a global singleton | |
| 14:03:32 | artom | So it's much easier to invalidate an item from a different place in the code | |
| 14:03:36 | mriedem | bauzas: "Consequently, it is highly recommended for users to never ever ask for booting an instance by specifying an explicit AZ named nova and for operators to never set the AZ metadata for an aggregate to nova." | |
| 14:03:40 | mriedem | ^ is all about the end user | |
| 14:03:47 | mriedem | artom: yeah true | |
| 14:04:10 | mriedem | artom: so yeah if you were a deployment that heavily depended on metadata api, you're probably using memcahe | |
| 14:04:22 | bauzas | mriedem: setting AZ metadata is an operator thing, right? not sure I get your point | |
| 14:04:27 | mriedem | *memcache - that's why the dogpile per-process dict cache is the cheap and easy default | |
| 14:04:38 | mriedem | bauzas: read again "Consequently, it is highly recommended for users to never ever ask for booting an instance by specifying an explicit AZ named nova and for operators to never set the AZ metadata for an aggregate to nova." | |
| 14:04:42 | bauzas | mriedem: most of the bugs I see are related to people saying "I did nova boot --az nova" and now I'm stuck | |
| 14:04:52 | mriedem | bauzas: yes, that's exactly what that warning says | |
| 14:05:02 | mriedem | "don't do nova boot --availability-zone nova" | |
| 14:05:02 | bauzas | mriedem: or people saying "I did created aggregate with AZ=nova and now I'm broken" | |
| 14:05:18 | bauzas | mriedem: I know, *I* wrote that big fat warning :)à | |
| 14:05:30 | mriedem | the user doesn't read the operators guide | |
| 14:05:52 | mriedem | https://github.com/openstack/nova/blob/master/api-ref/source/parameters.yaml#L3499 | |
| 14:05:54 | bauzas | ah okay, get your point | |
| 14:06:20 | bauzas | gotcha | |
| 14:06:24 | bauzas | I can forge a patch | |
| 14:06:27 | johnthetubaguy | lbragstad: mriedem: sounds like we need to talk about "is_admin_project:True", I am assuming that basically means god_mode=True | |
| 14:07:35 | mriedem | johnthetubaguy: my understanding in talking with lbragstad was is_admin_project was assigning someone as admin to a project, not the deployment | |
| 14:07:41 | mriedem | but it's really confusing so i probably screwed it up | |
| 14:08:00 | mriedem | or is is_admin_project a flag on a project that says, this is the god project, if yo'ure a member, welcome to heaven | |
| 14:08:02 | lbragstad | mriedem that - but the other way around | |
| 14:08:33 | lbragstad | the admin_project is this super-duper special project within keystone and if you have admin on it, you have admin on the deployment | |
| 14:08:50 | lbragstad | (it's weird and a little strange) | |
| 14:09:09 | mriedem | so what is the case where i'm just admin on a non-admin project? | |
| 14:09:18 | mriedem | is that just, role:admin on project foo? | |
| 14:09:24 | mriedem | not role:admin on project admin? | |
| 14:09:26 | lbragstad | mriedem yeah | |
| 14:09:49 | lbragstad | mriedem the case would be that you'd be able to manage the project, but not the deployment | |
| 14:09:50 | mriedem | ok | |
| 14:09:56 | lbragstad | (think creating subprojects) | |
| 14:10:21 | mriedem | johnthetubaguy: i don't know if you need that explained in your spec or not, it would have helped me, and in simple terms | |
| 14:10:34 | openstackgerrit | Sergey Nikitin proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | |
| 14:10:37 | mriedem | * god-mode: user as admin role on the special admin project | |
| 14:10:38 | lbragstad | or if you wanted to manage role assignments for things within the project you have admin on | |
| 14:11:12 | johnthetubaguy | mriedem: I tried to describe that in there, I probably failed. Its why I spoke about a "special" admin project. | |
| 14:11:14 | mriedem | * fief lord: user has admin role on non-admin tenant project | |
| 14:11:14 | lbragstad | i mean - it's a step in the right direction, but i kinda think of it as a bandaid | |