| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-01 | |||
| 19:31:17 | mriedem | anyway, that probably doesn't matter here, | |
| 19:31:27 | cfriesen | do we expect anyone to have wildly changing inventory? I would have thought inventory is relatively stable. | |
| 19:31:27 | mriedem | we get the inventory regardless to know if it changed so we can push updates back to placement | |
| 19:31:35 | mriedem | cfriesen: that's what i said about 2 hours ago | |
| 19:31:37 | sean-k-mooney | mriedem: if its somthing that is discoverd by the virt driver its not an issue if it chagnes | |
| 19:32:26 | mriedem | i also don't think we really need to worry about refreshing aggregate relationships in the compute, | |
| 19:32:29 | efried | mriedem: I contend it doesn't matter if powervm (or any driver) changes inventory every single periodic. Because update_provider_tree is getting whatever the previous state of placement was (because placement isn't changed yet) and then update_from_provider_tree is flushing that back to placement *and* updating the cache accordingly. | |
| 19:32:30 | mriedem | since we don't do anything with those yet | |
| 19:32:38 | efried | yeah, what sean-k-mooney said, only bigger. | |
| 19:33:38 | sean-k-mooney | so do we all agree we dont need to refesh the cache in any case that is atleas emitly obvious to us | |
| 19:33:53 | efried | tentatively yes | |
| 19:34:35 | mriedem | i would have thought a lot of prior discussion about why we even have a cache in the first place has happene | |
| 19:34:37 | mriedem | *happened | |
| 19:34:57 | mriedem | therefore it seems pretty severe to just all of a sudden say, "oh i guess we don't" | |
| 19:35:19 | mriedem | and if there are reasons, do those reasons justify us caching by default | |
| 19:35:35 | mriedem | anyway, those are questions for the ML, not irc | |
| 19:35:37 | sean-k-mooney | if so then should we start with a patch to default the refresh to off. and a mailing list post to see what operators think/other feedback | |
| 19:35:55 | mriedem | cern is the only deployment big enough and new enough that i've heard complain about that refresh | |
| 19:36:01 | mriedem | not sure if mnaser is doing anything about it | |
| 19:36:20 | mnaser | hi | |
| 19:36:42 | mriedem | mnaser: tl;dr do you turn this way down? https://docs.openstack.org/nova/latest/configuration/config.html#compute.resource_provider_association_refresh | |
| 19:37:01 | mriedem | to avoid computes DDoS'ing placement every 5 minutes | |
| 19:37:49 | mnaser | i didn't know about that but i do feel like placement gets waaaaay too much unnecessary traffic | |
| 19:38:05 | mnaser | an idle cloud (aka literally no new vms being created/deleted) will constantly hit placement | |
| 19:38:22 | efried | so mriedem, to do the thing you were talking about earlier, we would add is_new_compute_node as a kwarg from _update_available_resource into _update => _update_to_placement and then only call _get_provider_tree_and_ensure_root if it's true? | |
| 19:38:44 | mriedem | mnaser: inventory updates baby! | |
| 19:38:47 | mnaser | i'm all in favour of minimizing the amount of traffic that placement gets however a lot of times we end up seeing weird stuff happen in placement db | |
| 19:38:58 | mnaser | so if we have to edit stuff via the api | |
| 19:39:06 | mnaser | it'd be nice to just know they work | |
| 19:39:17 | mriedem | efried: that's what i was thinking yeah, and nearly starting writing it, but then you guys all said in_tree was mega importante | |
| 19:39:42 | efried | mnaser: True story. If you edit something by hand, and we've switched off all this cache refreshing, the only way you're going to pick it up again is to restart the compute service. | |
| 19:39:52 | efried | or maybe we can work a HUP in there or something. | |
| 19:40:11 | mriedem | HUP is how we refresh the enabled/disabled cell cache in the scheduler | |
| 19:40:34 | sean-k-mooney | and any of the mutable config stuff if we have it so HUP makes sense | |
| 19:40:36 | mnaser | but really the only editing ive had to do was delete/remove allocations | |
| 19:40:42 | mnaser | that were stale for $reasons | |
| 19:40:50 | sean-k-mooney | mnaser: oh if its allocation that fine | |
| 19:40:54 | efried | mriedem: based on having toggled some kind of setting, right? I.e. HUP is a no-op if you haven't changed anything. | |
| 19:40:58 | mnaser | so forgive me for my silly question but | |
| 19:41:22 | mnaser | why do computes care about that information | |
| 19:41:31 | efried | mnaser: which information specifically? | |
| 19:41:37 | efried | allocations? | |
| 19:41:46 | mnaser | well, whatever api hits all the time, i think its allocations? | |
| 19:41:57 | efried | what hits the API all the time is inventory, providers, aggregates. | |
| 19:42:18 | mnaser | but inventory can be a one time hit on start up because afaik the state of that is pretty darn static right | |
| 19:42:37 | sean-k-mooney | no this check aggretes and traits on the provider not the assoications | |
| 19:42:38 | efried | mnaser: And what we're discussing is, the virt drivers need to know what that stuff looks like so they can *modify* the provider layout, inventory, etc. | |
| 19:42:43 | sean-k-mooney | *allocations | |
| 19:42:50 | efried | BUT that stuff shouldn't change unless the virt driver changes it | |
| 19:42:55 | efried | ...or if you muck with it in the CLI :) | |
| 19:43:16 | mnaser | yeah usually my mucking around is around allocations, that's where things get out of sync usually | |
| 19:43:38 | efried | mnaser: Placement for dummies won't help you. And for placement-in-nova, there's no such thing as for-dummies. | |
| 19:43:49 | mnaser | but i mean the "traits" change dynamically? | |
| 19:43:59 | mnaser | anyways, i wont let your discusion diverge too much | |
| 19:44:09 | efried | no, traits is a good point, /me thinks... | |
| 19:44:15 | mriedem | mnaser: you are asking and saying the same thing i've been saying for an hour or so, | |
| 19:44:19 | mriedem | that inventory is pretty static | |
| 19:44:32 | sean-k-mooney | mnaser: we technicall propised that operators should be able to add traits to RPs but currently the virt dirver just overrite them i think | |
| 19:44:32 | mnaser | its 100% static.. things in inventory are | |
| 19:44:36 | efried | once again, if traits change due to external factors, you could HUP to get that flushed. | |
| 19:44:37 | mriedem | traits could be changed out of band if you're decorating capabilities on your compute node for scheduling, | |
| 19:44:41 | mnaser | memory.. disk.. vcpus.. | |
| 19:44:43 | mriedem | and aggregates aren't used in the compute service (yet) | |
| 19:45:00 | mnaser | yeah unless someone is hot plugging in memory/disk/cpu | |
| 19:45:05 | mnaser | i dont see inventory changing | |
| 19:45:22 | mnaser | and yes i agree traits make sense if you wanna say this compute node is special.. but also, does that compute node really care to know if its special? | |
| 19:45:27 | mnaser | only the scheduler cares that it's special.. | |
| 19:45:35 | sean-k-mooney | mriedem: well the only ones that are are the ones that are created form nova host aggreates(assuming jays stuff laned last cycle) | |
| 19:45:40 | efried | you could run into some interesting race conditions. If you muck with traits at the same time as the virt driver is mucking with traits, whoever gets there last will win. | |
| 19:45:40 | mriedem | mnaser: almost correct | |
| 19:45:59 | mriedem | right, we try to merge in what the compute is reporting for traits with what was put on the node resource provider for traits externally | |
| 19:46:13 | sean-k-mooney | mriedem: sorry you said compute service ignore that | |
| 19:46:31 | mnaser | ok so its like | |
| 19:46:34 | mriedem | the *only* thing on the compute that would care about aggregates in placement is shared storage providers, | |
| 19:46:36 | mriedem | which we don't support yet | |
| 19:46:37 | mnaser | self reported traits + "user" decorated traits | |
| 19:46:44 | mriedem | mnaser: yes | |
| 19:47:13 | mnaser | but the nova-compute reported traits.. i feel like those are pretty static right? maybe i just don't know any wild use cases | |
| 19:47:18 | mriedem | i think i've been saying since pike, at least queens, we don't need to refresh aggregate info in compute | |
| 19:47:26 | mriedem | mnaser: proably depends on the driver | |
| 19:47:30 | mnaser | but i feel like most of the time, if a system trait changes, you probably have nova-compute restart things | |
| 19:47:31 | mnaser | ah ok | |
| 19:47:44 | mriedem | vmware would love to be able to randomly proxy traits from vcenter through nova-compute to placement | |
| 19:47:45 | cfriesen | mnaser: the one exception would be something like vTPM where the driver uses the presence of the requested trait to decide to do something with the instance. | |
| 19:47:48 | mriedem | for changes in vcenter | |
| 19:47:57 | sean-k-mooney | mnaser: the intent with the user decorated traits was to let the operator tag node with stuff the virt dirver cant discover or to express policy | |
| 19:48:11 | cfriesen | mnaser: but that's really looking at the requested trait, not the trait on the resource provider | |
| 19:48:28 | sean-k-mooney | cfriesen: but that is in the instance request | |
| 19:48:40 | mnaser | yeah, i dunno, i feel like those will not change much, and i think just calling a method *once* when you make some changes rather than all the time isnt probelmatic | |
| 19:48:42 | sean-k-mooney | cfriesen: it does not need the RP info | |
| 19:48:43 | cfriesen | sean-k-mooney: yah, that's what I realized after typing the first sentence. :) | |
| 19:48:53 | mnaser | i mean lets be honest, we don't "refresh" allocations and those can go pretty stale | |
| 19:49:09 | mnaser | are "system" and "user" traits distingusable or in a race they can wipe each other out? | |
| 19:49:30 | mriedem | system traits might be 'standard' traits | |
| 19:49:35 | mriedem | user traits would be CUSTOM traits | |
| 19:49:42 | sean-k-mooney | efried: i assume we update the resouce provider generattion when updating traits? | |
| 19:49:52 | mriedem | but a user could put standard traits on a provider that the virt driver doesn't report | |
| 19:49:56 | mriedem | sean-k-mooney: yes | |
| 19:49:57 | efried | sean-k-mooney: Placement does | |
| 19:50:17 | cfriesen | sean-k-mooney: on a totally different topic, have you ever run into a scenario where qemu has a thread sitting at 100% cpu but not making any forward progress? I'm assuming it's a livelock somehow, just not sure how. | |