| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-22 | |||
| 18:39:00 | mriedem | efried: i just assume that like all things placement related there will be a holy war | |
| 18:39:48 | efried | I really hope we're getting close to having an established process for these GET /a_c-based filters | |
| 18:40:53 | efried | to the point where soon we'll be able to have specless bps for them. "Add trait called X. Set trait X on compute RP when Y. Add filter for X a) in flavor, b) in image, c) when condition Z is present. Done." | |
| 18:41:56 | sean-k-mooney | that would make my "port num instance filter to placemtn" personal backlog item simpler :) | |
| 18:42:05 | efried | melwitt, lbragstad: I seem to recall this session https://www.openstack.org/summit/denver-2019/summit-schedule/events/23712/migrating-nova-apis-to-keystone-scope-types being merged with another - link please? | |
| 18:43:15 | efried | d'oh, never mind. | |
| 18:45:54 | jaypipes | mriedem: https://review.opendev.org/#/c/623558/ addresses the *reason* why CERN limits to 20. That's a more appropriate solution IMHO than making placement into another servicegroup API where we will need to sync active status between the nova cell DB tables and placement. | |
| 18:46:54 | jaypipes | mriedem: it's not a holy war. it's just fundamentally you'll get into a situation where there are two sources of truth for whether a provider is active or not. | |
| 18:47:05 | jaypipes | I'd prefer to avoid that if possilbe. | |
| 18:47:42 | mriedem | we already have 2 sources of truth for aggregates yeah? | |
| 18:47:58 | mriedem | i thought one of the goals of placement was to do as much filtering in sql as possible to avoid the slower post-filtering in python we have | |
| 18:48:03 | mriedem | so this would also align with that | |
| 18:48:32 | mriedem | i realize the downside of this is nova and placement are out of sync on the status of the thing - we could potentially heal that in our compute update_available_resource periodic | |
| 18:48:38 | efried | Maybe eventually placement can be the sole source of truth | |
| 18:48:39 | jaypipes | erm, not really if it doesn't usually filter anything at all... | |
| 18:49:13 | mriedem | i imagine in a large cloud the ComputeFilter gets used a lot | |
| 18:49:16 | mriedem | during maintenance and such | |
| 18:50:29 | jaypipes | mriedem: we could run the servicegroup API ahead of time and pass a parameter like ?uuids=!in:<DISABLED_SET> | |
| 18:50:41 | jaypipes | which actually *would* be an efficient filter. | |
| 18:50:51 | jaypipes | but I digress. | |
| 18:51:11 | sean-k-mooney | that could be a long list of uuids | |
| 18:51:22 | jaypipes | sean-k-mooney: rarely. | |
| 18:51:22 | sean-k-mooney | which is why an aggregate would make more sense | |
| 18:51:45 | jaypipes | sean-k-mooney: you're going to keep an aggregate membership sync'd with yet another periodic task? :( | |
| 18:52:22 | sean-k-mooney | no on servcei startup have the compute node remove itself form the aggrrate if its currently listed | |
| 18:52:50 | mriedem | sean-k-mooney: i don't think you'd want an aggregate for all non-disabled computes if you have 14K computes | |
| 18:52:51 | sean-k-mooney | then we can add it if we miss a heartbeat or someone use force down | |
| 18:53:29 | jaypipes | mriedem: I think he's saying the opposite. an agg for disabled providers. | |
| 18:53:30 | sean-k-mooney | mriedem: sure but you can do it the other way around too as you said in the review | |
| 18:53:50 | mriedem | sean-k-mooney: if we did aggregates for this yes that's what i'd do, an aggregate for only disabled computes | |
| 18:53:57 | jaypipes | oh look, a case for forbidden aggregates :) | |
| 18:54:05 | mriedem | and yeah that's what we'd use ^ | |
| 18:54:07 | sean-k-mooney | jaypipes: orgiginly in the code is said an aggrate for enabled because i fogot negitive member of was merged | |
| 18:54:23 | jaypipes | sean-k-mooney: it's the forbidden fruit. | |
| 18:55:00 | sean-k-mooney | from a db perspecitve this really would not be that expensice either way | |
| 18:55:27 | sean-k-mooney | the negitive case would be better but it will be similar | |
| 18:55:49 | jaypipes | so, I suppose, in order of preference, I would personally go with: a) finish up https://review.opendev.org/#/c/623558/ and the other cell-specific-service-listing thing mentioned in there, b) use an aggregate for disabled computes and member_of=! |
|
| 18:55:57 | sean-k-mooney | the aggrates table is just too uuid fields both of which are indexed | |
| 18:56:21 | jaypipes | actually, it's not really an either/or. can/should do https://review.opendev.org/#/c/623558/ as well as a placement request filter for !member_of a disabled aggregate. | |
| 18:57:26 | sean-k-mooney | yes they are not mutally exclucive and i think both are useful | |
| 18:58:40 | jaypipes | mriedem, sean-k-mooney: maybe adding a CONF.disabled_aggregate_uuid option that would be used to construct a placement request filter that used a member_of=!<uuid> query parameter to allocation candidates? | |
| 18:58:54 | jaypipes | or something like that.. | |
| 18:59:06 | sean-k-mooney | jaypipes: well it would be a prefilter so it would be enabled via config | |
| 18:59:26 | sean-k-mooney | or are you also suggesting a config option for the compute node too | |
| 18:59:41 | mriedem | if we use an aggregate we have to use the same uuid everywhere, | |
| 18:59:51 | mriedem | either via config or hard-coded in api/compute/scheduler | |
| 18:59:53 | sean-k-mooney | ya that is simple however | |
| 19:00:14 | sean-k-mooney | just use a uuid5 that we caluate for a constat or hardcode it | |
| 19:14:59 | efried | oo, a special aggregate UUID. It's almost like... metadata. | |
| 19:15:22 | sean-k-mooney | that is sotred in nova | |
| 19:15:31 | sean-k-mooney | not placement | |
| 19:15:57 | sean-k-mooney | clinets of placement are free to use aggreate to group rps whatever way the like | |
| 19:27:25 | sean-k-mooney | by the way before i go i noticed some strange errors in the nfv-ci job | |
| 19:27:29 | sean-k-mooney | libvirtError: Cannot access storage file '/opt/stack/data/nova/instances/b4afcea9-12d0-4f79-953c-d09dad3e5515/disk' (as uid:107, gid:107): Permission denied | |
| 19:27:50 | sean-k-mooney | i also got DiskNotFound: No disk at /opt/stack/data/nova/instances/f5f5ce5f-a616-4e5e-aa35-d3cc469f869a/disk | |
| 19:28:46 | sean-k-mooney | i have set the concurance to 1 and its reruning currently as i think that is part of the issue but i have no iday way those disk errors are showing up | |
| 19:30:37 | sean-k-mooney | if people want to take a look the logs are here http://logs.openstack.org/97/652197/11/check/tempest-nfv-multinode/e1fa793/compute/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_20_22_14_03_503486 | |
| 19:32:13 | mriedem | efried: except metadata that is unparseable by the human eye :) | |
| 19:32:46 | sean-k-mooney | other vms worked fine so im wondering if this is related to load/memory pressure. | |
| 19:41:39 | mriedem | d15ab1ed-0000-0000-0000-00000000000 :) | |
| 19:43:32 | efried | sean-k-mooney: speaking of CI... | |
| 19:44:08 | sean-k-mooney | what did i break now :) | |
| 19:44:15 | efried | I got an extremely vague question from within the blue walls today about what kinds of tests the PCI job ought to be running. | |
| 19:44:25 | efried | I hadn't a clue, but thought you might. | |
| 19:46:07 | sean-k-mooney | ideally pci passthough via alaise in the flavor + neutron sriov interface, then it would be nice to alos test cold migration/resize and possibly live migration in the neutron case | |
| 19:46:18 | efried | someone said something like the PCI job was just setting up devstack and not running any actual tests. I'm not sure how accurate that could be, or if they were looking at the wrong thing, or if at some point we disabled all the actual tests while we were working on the infra, or what. | |
| 19:46:32 | sean-k-mooney | i would just run the standared tempest test with tweeked flavors and different vnic_types | |
| 19:46:58 | sean-k-mooney | efried: it runs test but it boots like 1 vm or very few tests | |
| 19:48:02 | efried | was the PCI job just hitting SR-IOV? | |
| 19:48:17 | efried | Cause there's an SRIOV job that runs against neutron | |
| 19:48:22 | efried | not sure if those are related | |
| 19:49:17 | sean-k-mooney | the pci job was ment for testing QAT integration with flavor bassed passthough | |
| 19:49:29 | sean-k-mooney | there was a seperate one for neutron sriov testing | |
| 19:49:53 | sean-k-mooney | looking at http://52.27.155.124/pci/482200/7/ | |
| 19:50:07 | sean-k-mooney | there are not test in the test repofrt | |
| 19:50:35 | sean-k-mooney | but im not sure that means threare are not test | |
| 19:51:18 | sean-k-mooney | it looks like some test are run http://52.27.155.124/pci/482200/7/console_status.log.gz | |
| 19:52:48 | efried | I can't tell if they pass | |
| 19:54:21 | sean-k-mooney | nor can i | |
| 19:54:46 | sean-k-mooney | looking at the devstack output and the tempest config i cant see tehm creating any custom flavor either | |
| 19:55:55 | efried | how did you know where to find these logs? | |
| 19:56:11 | efried | the CI is running, just not reporting? | |
| 19:56:41 | sean-k-mooney | i clicked on an old log got a 404 and went up one directory | |
| 19:57:13 | efried | how did you know this patch had any results? Did you search for reviewedby:Intel PCI? | |
| 19:57:52 | sean-k-mooney | i looked at one of my patches that i knew it had commented on in the past | |
| 19:58:08 | sean-k-mooney | http://52.27.155.124/ you can see the rest of the logs are here for the other intel cis | |
| 19:58:28 | efried | IC | |
| 19:59:00 | sean-k-mooney | this is a ovh log server that i used ot own/payfor via intel for the ci logs | |
| 19:59:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion https://review.opendev.org/654468 | |
| 20:00:11 | sean-k-mooney | or maybe it moved | |
| 20:00:16 | sean-k-mooney | it used to be http://intel-openstack-ci-logs.ovh | |
| 20:00:37 | efried | I think the 52. is aws | |
| 20:01:18 | sean-k-mooney | ya the pci ci used an aws log server. they moved to using our ovh one for a while i guess it moved back to aws at some point | |
| 20:01:20 | efried | yeah, that whole class A is amazon's. Not sure if that means aws, but probably | |
| 20:01:54 | sean-k-mooney | basicaly it just had to be a server that was not on the intel network | |
| 20:02:06 | efried | right, so the outside world could see it. | |
| 20:02:45 | sean-k-mooney | actully no os that you did not have to go through a 3 level security review to host an external facing service on the coperate network | |
| 20:03:10 | efried | yeah, that too | |
| 20:03:33 | sean-k-mooney | if its not on the copreate network it was way eaiser aslo the pci ci was originally in china which casued issues | |
| 20:04:07 | efried | okay, I'm going to dive over the firewall and throw some of this intel back at intel. Thanks for the help. | |