| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-22 | |||
| 18:17:54 | mriedem | why? | |
| 18:18:04 | sean-k-mooney | metadata in placement | |
| 18:18:10 | mriedem | that's what traits are | |
| 18:18:43 | dansmith | because it's not a capability | |
| 18:19:14 | efried | mriedem: That's a pretty big "bug fix" | |
| 18:19:20 | sean-k-mooney | well if a compute service is disable its not capable of booting a node | |
| 18:19:23 | sean-k-mooney | *vm | |
| 18:20:03 | sean-k-mooney | the other way to do it if traits were not used would be a aggregate | |
| 18:20:24 | sean-k-mooney | just add the root rps of all disabled nodes to an aggreate and frobid it | |
| 18:20:26 | mriedem | efried: see the todo | |
| 18:20:30 | sean-k-mooney | but a trait works too | |
| 18:20:57 | mriedem | so, this is a poc hacked together for discussion at the ptg | |
| 18:21:05 | mriedem | i would write a spec | |
| 18:21:11 | mriedem | but wanted to post this first after working on it one morning | |
| 18:21:26 | efried | dig | |
| 18:21:53 | sean-k-mooney | an active service aggrate + member of is likely less of a sticking point in that an aggrate is just a bunch of resouce providers | |
| 18:32:51 | mriedem | efried: i thought i had something in the ptg etherpad for this bug but can't find it now - what would you prefer? nova general etherpad or the xp one with placement? | |
| 18:33:20 | efried | mriedem: "this bug" - disabled CPU prefiltering? | |
| 18:33:32 | mriedem | yes | |
| 18:33:34 | efried | Doesn't require any placement changes (I don't count os-traits really) so nova etherpad. | |
| 18:33:55 | efried | but please ask yourself: does it need PTG time? | |
| 18:33:56 | mriedem | alright | |
| 18:34:17 | efried | I haven't read your patch yet | |
| 18:34:19 | mriedem | given the questions i just got after posting this... | |
| 18:34:48 | efried | would | |
| 18:34:48 | efried | sure, questions that would be answered in a spec; I guess I'm asking whether discussion in the spec wolud suffice. | |
| 18:35:14 | sean-k-mooney | am it might help to do some of it in real time too | |
| 18:35:30 | mriedem | yeah idk, i guess i was thinking i'd rather crank it out in 15 minutes rather than waffle on a spec for 2 months | |
| 18:35:33 | sean-k-mooney | but i think the idea of adressing this in placment in some way is a good one | |
| 18:36:12 | efried | sean-k-mooney: Do you mean you think there's potential placement API changes needed? | |
| 18:36:13 | mriedem | sean-k-mooney: it has to be if cern (and huawei public cloud) want to grab more allocation candidates from placement during scheduling, cern is currently hacking around this | |
| 18:36:47 | sean-k-mooney | efried: no this should not need any api changes in placment | |
| 18:37:25 | sean-k-mooney | but we need to either use a trait or aggrage of some kind to track it in placement so placment can use its existing apis to filter out disabled hosts | |
| 18:37:37 | sean-k-mooney | mriedem: right if you use limit its going to be a huge pain | |
| 18:38:24 | mriedem | cern limits their allocation candidate results to like 20 and as a result were only getting back disabled computes in some cases | |
| 18:38:25 | sean-k-mooney | i left a quick comment https://review.opendev.org/#/c/654596/1/nova/scheduler/request_filter.py@104 by the way ill review it properly tomorow | |
| 18:38:42 | efried | Yeah, so the concept seems pretty simple to me, but if you think we need PTG time... | |
| 18:39:00 | mriedem | efried: i just assume that like all things placement related there will be a holy war | |
| 18:39:48 | efried | I really hope we're getting close to having an established process for these GET /a_c-based filters | |
| 18:40:53 | efried | to the point where soon we'll be able to have specless bps for them. "Add trait called X. Set trait X on compute RP when Y. Add filter for X a) in flavor, b) in image, c) when condition Z is present. Done." | |
| 18:41:56 | sean-k-mooney | that would make my "port num instance filter to placemtn" personal backlog item simpler :) | |
| 18:42:05 | efried | melwitt, lbragstad: I seem to recall this session https://www.openstack.org/summit/denver-2019/summit-schedule/events/23712/migrating-nova-apis-to-keystone-scope-types being merged with another - link please? | |
| 18:43:15 | efried | d'oh, never mind. | |
| 18:45:54 | jaypipes | mriedem: https://review.opendev.org/#/c/623558/ addresses the *reason* why CERN limits to 20. That's a more appropriate solution IMHO than making placement into another servicegroup API where we will need to sync active status between the nova cell DB tables and placement. | |
| 18:46:54 | jaypipes | mriedem: it's not a holy war. it's just fundamentally you'll get into a situation where there are two sources of truth for whether a provider is active or not. | |
| 18:47:05 | jaypipes | I'd prefer to avoid that if possilbe. | |
| 18:47:42 | mriedem | we already have 2 sources of truth for aggregates yeah? | |
| 18:47:58 | mriedem | i thought one of the goals of placement was to do as much filtering in sql as possible to avoid the slower post-filtering in python we have | |
| 18:48:03 | mriedem | so this would also align with that | |
| 18:48:32 | mriedem | i realize the downside of this is nova and placement are out of sync on the status of the thing - we could potentially heal that in our compute update_available_resource periodic | |
| 18:48:38 | efried | Maybe eventually placement can be the sole source of truth | |
| 18:48:39 | jaypipes | erm, not really if it doesn't usually filter anything at all... | |
| 18:49:13 | mriedem | i imagine in a large cloud the ComputeFilter gets used a lot | |
| 18:49:16 | mriedem | during maintenance and such | |
| 18:50:29 | jaypipes | mriedem: we could run the servicegroup API ahead of time and pass a parameter like ?uuids=!in:<DISABLED_SET> | |
| 18:50:41 | jaypipes | which actually *would* be an efficient filter. | |
| 18:50:51 | jaypipes | but I digress. | |
| 18:51:11 | sean-k-mooney | that could be a long list of uuids | |
| 18:51:22 | jaypipes | sean-k-mooney: rarely. | |
| 18:51:22 | sean-k-mooney | which is why an aggregate would make more sense | |
| 18:51:45 | jaypipes | sean-k-mooney: you're going to keep an aggregate membership sync'd with yet another periodic task? :( | |
| 18:52:22 | sean-k-mooney | no on servcei startup have the compute node remove itself form the aggrrate if its currently listed | |
| 18:52:50 | mriedem | sean-k-mooney: i don't think you'd want an aggregate for all non-disabled computes if you have 14K computes | |
| 18:52:51 | sean-k-mooney | then we can add it if we miss a heartbeat or someone use force down | |
| 18:53:29 | jaypipes | mriedem: I think he's saying the opposite. an agg for disabled providers. | |
| 18:53:30 | sean-k-mooney | mriedem: sure but you can do it the other way around too as you said in the review | |
| 18:53:50 | mriedem | sean-k-mooney: if we did aggregates for this yes that's what i'd do, an aggregate for only disabled computes | |
| 18:53:57 | jaypipes | oh look, a case for forbidden aggregates :) | |
| 18:54:05 | mriedem | and yeah that's what we'd use ^ | |
| 18:54:07 | sean-k-mooney | jaypipes: orgiginly in the code is said an aggrate for enabled because i fogot negitive member of was merged | |
| 18:54:23 | jaypipes | sean-k-mooney: it's the forbidden fruit. | |
| 18:55:00 | sean-k-mooney | from a db perspecitve this really would not be that expensice either way | |
| 18:55:27 | sean-k-mooney | the negitive case would be better but it will be similar | |
| 18:55:49 | jaypipes | so, I suppose, in order of preference, I would personally go with: a) finish up https://review.opendev.org/#/c/623558/ and the other cell-specific-service-listing thing mentioned in there, b) use an aggregate for disabled computes and member_of=! |
|
| 18:55:57 | sean-k-mooney | the aggrates table is just too uuid fields both of which are indexed | |
| 18:56:21 | jaypipes | actually, it's not really an either/or. can/should do https://review.opendev.org/#/c/623558/ as well as a placement request filter for !member_of a disabled aggregate. | |
| 18:57:26 | sean-k-mooney | yes they are not mutally exclucive and i think both are useful | |
| 18:58:40 | jaypipes | mriedem, sean-k-mooney: maybe adding a CONF.disabled_aggregate_uuid option that would be used to construct a placement request filter that used a member_of=!<uuid> query parameter to allocation candidates? | |
| 18:58:54 | jaypipes | or something like that.. | |
| 18:59:06 | sean-k-mooney | jaypipes: well it would be a prefilter so it would be enabled via config | |
| 18:59:26 | sean-k-mooney | or are you also suggesting a config option for the compute node too | |
| 18:59:41 | mriedem | if we use an aggregate we have to use the same uuid everywhere, | |
| 18:59:51 | mriedem | either via config or hard-coded in api/compute/scheduler | |
| 18:59:53 | sean-k-mooney | ya that is simple however | |
| 19:00:14 | sean-k-mooney | just use a uuid5 that we caluate for a constat or hardcode it | |
| 19:14:59 | efried | oo, a special aggregate UUID. It's almost like... metadata. | |
| 19:15:22 | sean-k-mooney | that is sotred in nova | |
| 19:15:31 | sean-k-mooney | not placement | |
| 19:15:57 | sean-k-mooney | clinets of placement are free to use aggreate to group rps whatever way the like | |
| 19:27:25 | sean-k-mooney | by the way before i go i noticed some strange errors in the nfv-ci job | |
| 19:27:29 | sean-k-mooney | libvirtError: Cannot access storage file '/opt/stack/data/nova/instances/b4afcea9-12d0-4f79-953c-d09dad3e5515/disk' (as uid:107, gid:107): Permission denied | |
| 19:27:50 | sean-k-mooney | i also got DiskNotFound: No disk at /opt/stack/data/nova/instances/f5f5ce5f-a616-4e5e-aa35-d3cc469f869a/disk | |
| 19:28:46 | sean-k-mooney | i have set the concurance to 1 and its reruning currently as i think that is part of the issue but i have no iday way those disk errors are showing up | |
| 19:30:37 | sean-k-mooney | if people want to take a look the logs are here http://logs.openstack.org/97/652197/11/check/tempest-nfv-multinode/e1fa793/compute/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_20_22_14_03_503486 | |
| 19:32:13 | mriedem | efried: except metadata that is unparseable by the human eye :) | |
| 19:32:46 | sean-k-mooney | other vms worked fine so im wondering if this is related to load/memory pressure. | |
| 19:41:39 | mriedem | d15ab1ed-0000-0000-0000-00000000000 :) | |
| 19:43:32 | efried | sean-k-mooney: speaking of CI... | |
| 19:44:08 | sean-k-mooney | what did i break now :) | |
| 19:44:15 | efried | I got an extremely vague question from within the blue walls today about what kinds of tests the PCI job ought to be running. | |