| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-13 | |||
| 13:45:42 | artom | I'm trying to thinking of a good way to solve that, without suddenly requiring all deployment to run memcache.' | |
| 13:46:07 | bauzas | jaypipes: in case you have some brain time free, I'd love to get your insights on https://review.openstack.org/#/c/450122/15 | |
| 13:46:53 | bauzas | jaypipes: my biggest concern is how a virt driver can return a single dict for a common RC if it has two distinct devices behind it | |
| 13:47:04 | bauzas | unless it's not really a problem and then meh. | |
| 13:48:04 | mriedem | artom: i wonder if we could set a flag in the db to signal when the cache should be dumped | |
| 13:48:30 | mriedem | and only look it up when requesting a version new enough for the tagged attach stuff, | |
| 13:48:37 | mriedem | although i think with config drive that would be every time | |
| 13:48:38 | artom | mriedem, it's not in the DB anywhere... | |
| 13:48:47 | mriedem | artom: i know | |
| 13:48:52 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | |
| 13:48:59 | sfinucan | johnthetubaguy: & | |
| 13:49:03 | mriedem | artom: is the cache per instance? | |
| 13:49:11 | mriedem | i'm not terribly familiar with this | |
| 13:49:13 | artom | mriedem, I think it's per process | |
| 13:49:14 | artom | https://github.com/openstack/nova/blob/master/nova/cache_utils.py | |
| 13:49:14 | johnthetubaguy | sfinucan: cool, queued it | |
| 13:49:32 | sfinucan | Also: cdent, sean-k-mooney, snikitin_, sgordon - https://review.openstack.org/361140 | |
| 13:49:45 | sfinucan | I went with required, preferred and legacy | |
| 13:49:57 | artom | mriedem, the metadata handler calls get_client(), which, depending on conf, configures a cache region in a certain way | |
| 13:50:32 | artom | It's using dogpile.cache, in which a region is basically a frontend/client: https://dogpilecache.readthedocs.io/en/latest/usage.html | |
| 13:50:38 | sfinucan | jaypipes: You don't get to escape either :P https://review.openstack.org/361140 | |
| 13:51:07 | cdent | ha | |
| 13:51:11 | artom | And I think the backend is basically a dict: https://github.com/openstack/oslo.cache/blob/master/oslo_cache/backends/dictionary.py | |
| 13:51:50 | artom | So unless the REST API request just happened to land on the same process that eventually handles the metadata request, we need some form of IPC | |
| 13:52:44 | mriedem | artom: default cache is just in-memory per process dict? | |
| 13:52:51 | mriedem | i think it's dogpile yeah? | |
| 13:52:59 | artom | mriedem, that's what I gather, yeah | |
| 13:55:02 | mriedem | well this is a sticky wicket | |
| 13:55:19 | mriedem | we have to look up an instance when it's not cache, or the cache is expired | |
| 13:55:48 | mriedem | i was thinking we could have set a flag in instance_extra that would tell metadata api to refresh the cache, but that defeats the purpose because you have to do the db lookup to find out | |
| 13:56:26 | mriedem | default cache expiration is 15 seconds right? | |
| 13:56:48 | snikitin_ | sfinucan: wow. Thank you for update! btw 'legacy' looks good because this behavior it really a little bit strange | |
| 13:56:57 | mriedem | does the guest need to see the tagged block device immediately? | |
| 13:57:10 | artom | mriedem, ah, you're saying just wait it out? | |
| 13:57:13 | frickler | mriedem: regarding your ML question, johnthetubaguy has a patch up to clarify things: https://review.openstack.org/#/c/438887/3 | |
| 13:57:17 | mriedem | artom: that's what i'm asking | |
| 13:57:33 | mriedem | artom: i.e. this is already a latent issue right? | |
| 13:57:39 | sfinucan | snikitin_: no problem | |
| 13:57:55 | mriedem | if your guest is relying on changes via metadata api, and it's already cached on a timer, then your guest logic probably already needs to poll for changes | |
| 13:58:03 | mriedem | i don't really know the use cases though | |
| 13:58:12 | sfinucan | snikitin_: Personally I'd like to drop that policy because it makes no sense. If you care about NUMA affinity, you should have 100% NUMA-aware nodes. If you don't, you clearly don't care enough :) | |
| 13:58:17 | johnthetubaguy | frickler: mriedem: thats right, I new that was familiar, I think bauzas said he would fix up that patch. | |
| 13:58:25 | sfinucan | But this is a good half way measure to avoid breaking existing deployments | |
| 13:58:27 | artom | mriedem, I suppose it is... but... well, did we ever have a user case where the metadata changed during an instance's lifetime? | |
| 13:59:00 | mriedem | artom: i'm sure there are all sorts of things that change | |
| 13:59:04 | snikitin_ | sfinucan: yea, me too. But this damn backward compatibility... | |
| 13:59:14 | mriedem | but let me check | |
| 13:59:38 | bauzas | johnthetubaguy: it just went out of my radar, because I'm not the Gerrit owner :( | |
| 13:59:51 | johnthetubaguy | bauzas: yeah, I have done that a few times also | |
| 13:59:54 | mriedem | artom: security groups could change, password could change | |
| 13:59:54 | sfinucan | snikitin_: I know, right? How dare our customers ask for backwards compatibility :P Let's hope that naming clears up everything | |
| 14:00:10 | mriedem | artom: network metadata could change couldn't it depending on attaching/detaching ports? | |
| 14:00:25 | mriedem | not to mention dynamic vendor data | |
| 14:00:27 | bauzas | mriedem: if your ML thread is related to my az spec, nevermind it, we could just wait for Queens | |
| 14:00:39 | mriedem | bauzas: it was the impetus yes | |
| 14:00:49 | mriedem | i never knew about default_scheduler_zone | |
| 14:00:54 | artom | mriedem, that's probably correct... | |
| 14:01:05 | bauzas | mriedem: default_schedul/e/_zone but yeah | |
| 14:01:15 | mriedem | artom: so if you accept as a limitation that you're not getting the latest changes immediately when using metadata api, then this is probably ok | |
| 14:01:16 | bauzas | mriedem: hope my reply made things clearer for you | |
| 14:01:32 | mriedem | bauzas: it made me feel like there was a purpose | |
| 14:01:43 | artom | mriedem, I think that's acceptable in the short term, at least | |
| 14:01:43 | mriedem | bauzas: however, the help text on those options is still murky | |
| 14:01:46 | mriedem | and should probably be updated | |
| 14:01:53 | bauzas | mriedem: yup, hence johnthetubaguy and frickler point | |
| 14:01:55 | mriedem | bauzas: plus, | |
| 14:01:58 | artom | mriedem, and if someone complains we can always do a "real" cache later | |
| 14:01:58 | snikitin_ | sfinucan: :D | |
| 14:02:01 | mriedem | this big fat scary warning https://docs.openstack.org/developer/nova/aggregates.html#availability-zones-azs | |
| 14:02:04 | mriedem | should probably be in the api-ref | |
| 14:02:07 | mriedem | where api users would see it | |
| 14:02:10 | mriedem | rather than way down in devref | |
| 14:02:11 | artom | Or they can run memcache and we can add a bit where we invalidate it | |
| 14:02:29 | bauzas | mriedem: well, when I wrote by that time, I wasn't seasoned with api-ref :) | |
| 14:02:39 | mriedem | artom: memcache doesn't help the timing problem right? just the consistency issue, | |
| 14:02:40 | bauzas | it should even be in the operators guide | |
| 14:02:51 | mriedem | i.e. depending on which metadata api worker i hit with my request, i could get different results | |
| 14:03:02 | bauzas | anyway, spec deadline day, I'm turned into review day for me | |
| 14:03:07 | mriedem | or would memcache be updated globally for all? i suppose it would | |
| 14:03:09 | mriedem | nevermind then | |
| 14:03:18 | artom | mriedem, I'm assuming memcache would be a global singleton | |
| 14:03:32 | artom | So it's much easier to invalidate an item from a different place in the code | |
| 14:03:36 | mriedem | bauzas: "Consequently, it is highly recommended for users to never ever ask for booting an instance by specifying an explicit AZ named nova and for operators to never set the AZ metadata for an aggregate to nova." | |
| 14:03:40 | mriedem | ^ is all about the end user | |
| 14:03:47 | mriedem | artom: yeah true | |
| 14:04:10 | mriedem | artom: so yeah if you were a deployment that heavily depended on metadata api, you're probably using memcahe | |
| 14:04:22 | bauzas | mriedem: setting AZ metadata is an operator thing, right? not sure I get your point | |
| 14:04:27 | mriedem | *memcache - that's why the dogpile per-process dict cache is the cheap and easy default | |
| 14:04:38 | mriedem | bauzas: read again "Consequently, it is highly recommended for users to never ever ask for booting an instance by specifying an explicit AZ named nova and for operators to never set the AZ metadata for an aggregate to nova." | |
| 14:04:42 | bauzas | mriedem: most of the bugs I see are related to people saying "I did nova boot --az nova" and now I'm stuck | |
| 14:04:52 | mriedem | bauzas: yes, that's exactly what that warning says | |
| 14:05:02 | bauzas | mriedem: or people saying "I did created aggregate with AZ=nova and now I'm broken" | |
| 14:05:02 | mriedem | "don't do nova boot --availability-zone nova" | |
| 14:05:18 | bauzas | mriedem: I know, *I* wrote that big fat warning :)à | |
| 14:05:30 | mriedem | the user doesn't read the operators guide | |
| 14:05:52 | mriedem | https://github.com/openstack/nova/blob/master/api-ref/source/parameters.yaml#L3499 | |
| 14:05:54 | bauzas | ah okay, get your point | |
| 14:06:20 | bauzas | gotcha | |
| 14:06:24 | bauzas | I can forge a patch | |
| 14:06:27 | johnthetubaguy | lbragstad: mriedem: sounds like we need to talk about "is_admin_project:True", I am assuming that basically means god_mode=True | |