| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-22 | |||
| 18:20:57 | mriedem | so, this is a poc hacked together for discussion at the ptg | |
| 18:21:05 | mriedem | i would write a spec | |
| 18:21:11 | mriedem | but wanted to post this first after working on it one morning | |
| 18:21:26 | efried | dig | |
| 18:21:53 | sean-k-mooney | an active service aggrate + member of is likely less of a sticking point in that an aggrate is just a bunch of resouce providers | |
| 18:32:51 | mriedem | efried: i thought i had something in the ptg etherpad for this bug but can't find it now - what would you prefer? nova general etherpad or the xp one with placement? | |
| 18:33:20 | efried | mriedem: "this bug" - disabled CPU prefiltering? | |
| 18:33:32 | mriedem | yes | |
| 18:33:34 | efried | Doesn't require any placement changes (I don't count os-traits really) so nova etherpad. | |
| 18:33:55 | efried | but please ask yourself: does it need PTG time? | |
| 18:33:56 | mriedem | alright | |
| 18:34:17 | efried | I haven't read your patch yet | |
| 18:34:19 | mriedem | given the questions i just got after posting this... | |
| 18:34:48 | efried | sure, questions that would be answered in a spec; I guess I'm asking whether discussion in the spec wolud suffice. | |
| 18:34:48 | efried | would | |
| 18:35:14 | sean-k-mooney | am it might help to do some of it in real time too | |
| 18:35:30 | mriedem | yeah idk, i guess i was thinking i'd rather crank it out in 15 minutes rather than waffle on a spec for 2 months | |
| 18:35:33 | sean-k-mooney | but i think the idea of adressing this in placment in some way is a good one | |
| 18:36:12 | efried | sean-k-mooney: Do you mean you think there's potential placement API changes needed? | |
| 18:36:13 | mriedem | sean-k-mooney: it has to be if cern (and huawei public cloud) want to grab more allocation candidates from placement during scheduling, cern is currently hacking around this | |
| 18:36:47 | sean-k-mooney | efried: no this should not need any api changes in placment | |
| 18:37:25 | sean-k-mooney | but we need to either use a trait or aggrage of some kind to track it in placement so placment can use its existing apis to filter out disabled hosts | |
| 18:37:37 | sean-k-mooney | mriedem: right if you use limit its going to be a huge pain | |
| 18:38:24 | mriedem | cern limits their allocation candidate results to like 20 and as a result were only getting back disabled computes in some cases | |
| 18:38:25 | sean-k-mooney | i left a quick comment https://review.opendev.org/#/c/654596/1/nova/scheduler/request_filter.py@104 by the way ill review it properly tomorow | |
| 18:38:42 | efried | Yeah, so the concept seems pretty simple to me, but if you think we need PTG time... | |
| 18:39:00 | mriedem | efried: i just assume that like all things placement related there will be a holy war | |
| 18:39:48 | efried | I really hope we're getting close to having an established process for these GET /a_c-based filters | |
| 18:40:53 | efried | to the point where soon we'll be able to have specless bps for them. "Add trait called X. Set trait X on compute RP when Y. Add filter for X a) in flavor, b) in image, c) when condition Z is present. Done." | |
| 18:41:56 | sean-k-mooney | that would make my "port num instance filter to placemtn" personal backlog item simpler :) | |
| 18:42:05 | efried | melwitt, lbragstad: I seem to recall this session https://www.openstack.org/summit/denver-2019/summit-schedule/events/23712/migrating-nova-apis-to-keystone-scope-types being merged with another - link please? | |
| 18:43:15 | efried | d'oh, never mind. | |
| 18:45:54 | jaypipes | mriedem: https://review.opendev.org/#/c/623558/ addresses the *reason* why CERN limits to 20. That's a more appropriate solution IMHO than making placement into another servicegroup API where we will need to sync active status between the nova cell DB tables and placement. | |
| 18:46:54 | jaypipes | mriedem: it's not a holy war. it's just fundamentally you'll get into a situation where there are two sources of truth for whether a provider is active or not. | |
| 18:47:05 | jaypipes | I'd prefer to avoid that if possilbe. | |
| 18:47:42 | mriedem | we already have 2 sources of truth for aggregates yeah? | |
| 18:47:58 | mriedem | i thought one of the goals of placement was to do as much filtering in sql as possible to avoid the slower post-filtering in python we have | |
| 18:48:03 | mriedem | so this would also align with that | |
| 18:48:32 | mriedem | i realize the downside of this is nova and placement are out of sync on the status of the thing - we could potentially heal that in our compute update_available_resource periodic | |
| 18:48:38 | efried | Maybe eventually placement can be the sole source of truth | |
| 18:48:39 | jaypipes | erm, not really if it doesn't usually filter anything at all... | |
| 18:49:13 | mriedem | i imagine in a large cloud the ComputeFilter gets used a lot | |
| 18:49:16 | mriedem | during maintenance and such | |
| 18:50:29 | jaypipes | mriedem: we could run the servicegroup API ahead of time and pass a parameter like ?uuids=!in:<DISABLED_SET> | |
| 18:50:41 | jaypipes | which actually *would* be an efficient filter. | |
| 18:50:51 | jaypipes | but I digress. | |
| 18:51:11 | sean-k-mooney | that could be a long list of uuids | |
| 18:51:22 | sean-k-mooney | which is why an aggregate would make more sense | |
| 18:51:22 | jaypipes | sean-k-mooney: rarely. | |
| 18:51:45 | jaypipes | sean-k-mooney: you're going to keep an aggregate membership sync'd with yet another periodic task? :( | |
| 18:52:22 | sean-k-mooney | no on servcei startup have the compute node remove itself form the aggrrate if its currently listed | |
| 18:52:50 | mriedem | sean-k-mooney: i don't think you'd want an aggregate for all non-disabled computes if you have 14K computes | |
| 18:52:51 | sean-k-mooney | then we can add it if we miss a heartbeat or someone use force down | |
| 18:53:29 | jaypipes | mriedem: I think he's saying the opposite. an agg for disabled providers. | |
| 18:53:30 | sean-k-mooney | mriedem: sure but you can do it the other way around too as you said in the review | |
| 18:53:50 | mriedem | sean-k-mooney: if we did aggregates for this yes that's what i'd do, an aggregate for only disabled computes | |
| 18:53:57 | jaypipes | oh look, a case for forbidden aggregates :) | |
| 18:54:05 | mriedem | and yeah that's what we'd use ^ | |
| 18:54:07 | sean-k-mooney | jaypipes: orgiginly in the code is said an aggrate for enabled because i fogot negitive member of was merged | |
| 18:54:23 | jaypipes | sean-k-mooney: it's the forbidden fruit. | |
| 18:55:00 | sean-k-mooney | from a db perspecitve this really would not be that expensice either way | |
| 18:55:27 | sean-k-mooney | the negitive case would be better but it will be similar | |
| 18:55:49 | jaypipes | so, I suppose, in order of preference, I would personally go with: a) finish up https://review.opendev.org/#/c/623558/ and the other cell-specific-service-listing thing mentioned in there, b) use an aggregate for disabled computes and member_of=! |
|
| 18:55:57 | sean-k-mooney | the aggrates table is just too uuid fields both of which are indexed | |
| 18:56:21 | jaypipes | actually, it's not really an either/or. can/should do https://review.opendev.org/#/c/623558/ as well as a placement request filter for !member_of a disabled aggregate. | |
| 18:57:26 | sean-k-mooney | yes they are not mutally exclucive and i think both are useful | |
| 18:58:40 | jaypipes | mriedem, sean-k-mooney: maybe adding a CONF.disabled_aggregate_uuid option that would be used to construct a placement request filter that used a member_of=!<uuid> query parameter to allocation candidates? | |
| 18:58:54 | jaypipes | or something like that.. | |
| 18:59:06 | sean-k-mooney | jaypipes: well it would be a prefilter so it would be enabled via config | |
| 18:59:26 | sean-k-mooney | or are you also suggesting a config option for the compute node too | |
| 18:59:41 | mriedem | if we use an aggregate we have to use the same uuid everywhere, | |
| 18:59:51 | mriedem | either via config or hard-coded in api/compute/scheduler | |
| 18:59:53 | sean-k-mooney | ya that is simple however | |
| 19:00:14 | sean-k-mooney | just use a uuid5 that we caluate for a constat or hardcode it | |
| 19:14:59 | efried | oo, a special aggregate UUID. It's almost like... metadata. | |
| 19:15:22 | sean-k-mooney | that is sotred in nova | |
| 19:15:31 | sean-k-mooney | not placement | |
| 19:15:57 | sean-k-mooney | clinets of placement are free to use aggreate to group rps whatever way the like | |
| 19:27:25 | sean-k-mooney | by the way before i go i noticed some strange errors in the nfv-ci job | |
| 19:27:29 | sean-k-mooney | libvirtError: Cannot access storage file '/opt/stack/data/nova/instances/b4afcea9-12d0-4f79-953c-d09dad3e5515/disk' (as uid:107, gid:107): Permission denied | |
| 19:27:50 | sean-k-mooney | i also got DiskNotFound: No disk at /opt/stack/data/nova/instances/f5f5ce5f-a616-4e5e-aa35-d3cc469f869a/disk | |
| 19:28:46 | sean-k-mooney | i have set the concurance to 1 and its reruning currently as i think that is part of the issue but i have no iday way those disk errors are showing up | |
| 19:30:37 | sean-k-mooney | if people want to take a look the logs are here http://logs.openstack.org/97/652197/11/check/tempest-nfv-multinode/e1fa793/compute/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_20_22_14_03_503486 | |
| 19:32:13 | mriedem | efried: except metadata that is unparseable by the human eye :) | |
| 19:32:46 | sean-k-mooney | other vms worked fine so im wondering if this is related to load/memory pressure. | |
| 19:41:39 | mriedem | d15ab1ed-0000-0000-0000-00000000000 :) | |
| 19:43:32 | efried | sean-k-mooney: speaking of CI... | |
| 19:44:08 | sean-k-mooney | what did i break now :) | |
| 19:44:15 | efried | I got an extremely vague question from within the blue walls today about what kinds of tests the PCI job ought to be running. | |
| 19:44:25 | efried | I hadn't a clue, but thought you might. | |
| 19:46:07 | sean-k-mooney | ideally pci passthough via alaise in the flavor + neutron sriov interface, then it would be nice to alos test cold migration/resize and possibly live migration in the neutron case | |
| 19:46:18 | efried | someone said something like the PCI job was just setting up devstack and not running any actual tests. I'm not sure how accurate that could be, or if they were looking at the wrong thing, or if at some point we disabled all the actual tests while we were working on the infra, or what. | |
| 19:46:32 | sean-k-mooney | i would just run the standared tempest test with tweeked flavors and different vnic_types | |
| 19:46:58 | sean-k-mooney | efried: it runs test but it boots like 1 vm or very few tests | |
| 19:48:02 | efried | was the PCI job just hitting SR-IOV? | |
| 19:48:17 | efried | Cause there's an SRIOV job that runs against neutron | |
| 19:48:22 | efried | not sure if those are related | |
| 19:49:17 | sean-k-mooney | the pci job was ment for testing QAT integration with flavor bassed passthough | |
| 19:49:29 | sean-k-mooney | there was a seperate one for neutron sriov testing | |
| 19:49:53 | sean-k-mooney | looking at http://52.27.155.124/pci/482200/7/ | |