Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-07
14:47:08 mriedem but if we're unsure of the design at this point then we should defer
14:48:40 jaypipes efried, edleafe, sean-k-mooney: I will remind yet again that there is zero generation protection for multiple threads changing a provider's traits. that protection only exists on the quantitative side of the house -- i.e. inventory and resource classes.
14:49:12 mriedem i'll dig through git history to see if i can find any words around why server group member quota is enforced at the user rather than project leve
14:49:15 mriedem *level
14:49:19 sean-k-mooney jaypipes: yep i know which is part of why i was hoping to avoid that.
14:49:21 mriedem i'm sure this is probably something melwitt would have memorized by now
14:49:34 sean-k-mooney jaypipes: anyway i dont want to get into it now
14:49:43 efried whoah, that I didn't realize.
14:49:52 dansmith mriedem: yeah I think we've been over that before.. definitely doesn't seem like something we can change on a whim regardless of the reason
14:50:24 jaypipes efried: I said it on the hangout the other day with Sundar and I've said it at each of the past PTGs where the whole custom RCs vs traits was discussed.
14:50:32 efried We should fix that regardless.
14:50:41 efried we've got multiple threads potentially dorking with traits already.
14:50:58 mriedem jaypipes: the good news is phil day was good about writing useful commit messages https://review.openstack.org/#/c/104957/
14:52:05 dansmith mriedem: am I missing where in there is justifies being user-only?
14:52:30 mriedem "They can be defined per project or per user within a project"
14:52:32 jaypipes mriedem: "They can be defined per project or per user within a project" <-- shows me that user-specific quota *always* will be "within a project" and therefore should have had the project_id in the query.
14:52:33 mriedem i'm also looking
14:52:40 jaypipes lol. jiunx.
14:52:43 edleafe efried: capabilities don't change dynamically. State does.
14:53:09 jaypipes edleafe: zactly.
14:53:21 dansmith mriedem: yeah that just doesn't tell me why
14:53:59 jaypipes dansmith: fwiw, I wasn't wondering why it's a user-specific quota, just why the query for it wasn't also appending a project_id
14:54:09 cdent efried: we've got generations on adding traits to providers.
14:54:13 dansmith jaypipes: okay
14:54:35 jaypipes cdent: that's not what I'm referring to actually. :)
14:54:36 mriedem yeah idk either on the server group member user-only quota check,
14:54:40 mriedem if it was a mistake,
14:54:51 mriedem it sure farted it's way through several very large changes and different cores
14:55:05 cdent jaypipes: well I kind of assumed not, but got a bit lost trying to track all of this round about
14:55:05 mriedem too bad we don't have people around that maintain this code...
14:55:53 mriedem maybe sdague would remember....
14:56:18 cdent so jaypipes what does "there is zero generation protection for multiple threads changing a provider's traits."
14:56:21 mriedem cyeoh and ken'ichi moved it through
14:56:21 jaypipes cdent: specifically, this is the protection that doesn't exist when traits are used as faux inventory: https://github.com/openstack/placement/blob/master/placement/objects/allocation.py#L427-L447
14:56:33 cdent jinxish
14:56:35 efried cdent, jaypipes: Updating what traits are set on a resource provider <== has generation controls. Phew.
14:56:56 jaypipes efried: not for claiming resources.
14:57:02 efried oh, sure, traits aren't part of an allocation
14:57:04 jaypipes efried: which is what I've been saying.
14:57:05 efried that's just fine.
14:57:27 efried had me scared there for a minute
14:58:09 cdent jaypipes: so your concerned about the case where we've chosen a target on which we want to allocate and since we selected it, the traits of that thing have changed?
14:58:41 efried We also don't do ^ for aggregate memberships we originally used to gather the candidate.
14:58:42 luksky hi, I have question about scheduler - how to debug case where I have 26 compute nodes, but in first iteration I see that RetryFilter is getting only 22 at start ?
14:58:48 cdent (that either as "you're concerned about" or "your concern is". your pick)
14:58:49 jaypipes cdent: precisely, which is exactly what their whole "dynamic case" does.
14:59:28 efried We're not talking about changing traits continuously. Just when the characteristics (um, traits) of a resource provider change.
14:59:43 cdent luksky: depends on what release you're on
14:59:55 luksky queens
15:00:27 luksky 9.1.1
15:00:27 luksky [root@nova-01 ~]# nova --version
15:01:03 jaypipes luksky: check the placement service's logs.
15:01:04 cdent luksky: combination of debug logs from nova-scheduler and nova-placement-api
15:01:14 luksky In logs I see:
15:01:16 luksky 'RamFilter: (start: 4, end: 4)', 'DiskFilter: (start: 4, end: 4)', 'ComputeFilter: (start: 4, end: 4)', 'ComputeCapabilitiesFilter: (start: 4, end: 4)', 'ImagePropertiesFilter: (start: 4, end: 4)', 'ServerGroupAntiAffinityFilter: (start: 4, end: 4)', 'ServerGroupAffinityFilter: (start: 4, end: 4)', 'IsolatedHostsFilter: (start: 4, end: 0)']
15:01:16 luksky INFO nova.filters [req-6553ad52-40f0-41d3-a11e-3682bdccf589 f0b6fd8f6a1a421094fd6259fa0a8d6c a17c1ce3cc114f05b5da8c9553e5e4d1 - default default] Filtering removed all hosts for the request with instance ID 'dbaecc98-c097-4b7b-b4f8-459f1ed1fecc'. Filter results: ['RetryFilter: (start: 22, end: 22)', 'IsolatedTenantFilter: (start: 22, end: 4)', 'AggregateMultiTenancyIsolation: (start: 4, end: 4)', 'AvailabilityZoneFilter: (start: 4, end: 4)',
15:01:34 efried edleafe: "capabilities don't change dynamically. State does." What an FPGA is "capable" of doing *does* change dynamically, when the thing is programmed.
15:02:02 cfriesen efried: thanks for the update. :) You caught me just after I left.
15:02:14 efried cfriesen: yw
15:02:24 sean-k-mooney luksky: the retry filter will only get the set of host that passed the first iteration - the one that was tried and failed
15:02:51 luksky (os-client) (PROD)[root@test1 ~]# openstack compute service list --service nova-compute | grep nova-compute | wc -l
15:02:52 luksky 26
15:02:52 luksky WARNING: Failed to import plugin clustering.
15:02:53 edleafe efried: an FPGA region is capable of all available bitstreams for that model. When it is programmed, it is still capable of running any of the other bitstreams
15:03:26 edleafe Calling the current program a capability instead of the current state of the region is kind of a twist in language
15:04:06 luksky hm... don
15:04:12 luksky hm... don't get it
15:04:40 edleafe cdent: try #openstack-etymology instead :)
15:04:49 luksky shouldn't see "'RetryFilter: (start: 26, end: 26)" instead: "'RetryFilter: (start: 22, end: 22)"
15:04:50 luksky ?
15:04:54 cdent naw, I think we need to go deeper
15:05:26 efried again, realizing it's not perfectly pure, this is the best way to get where we want to be. Which is why we've ended up here several times after having discussed it the same way every time.
15:06:03 sean-k-mooney luksky: no
15:06:38 sean-k-mooney luksky: the retry filter on subsequet iterations does not retry host it knows cant pass
15:07:16 edleafe I think calling something "perfectly pure" is a bit condescending. I'm just pointing out that this approach is not the way placement is designed to work, and there may be downsides. If you all have agreed to go ahead with that, your eyes are wide open.
15:07:47 sean-k-mooney luksky: if its the first iteration and no filter have ran then the delta between 26 and 22 is because placemnt determined the other 4 instace did not have space for the instance and elminated them
15:08:56 mriedem jaypipes: i've gone fishing in the ML for the mystery of the server_group_members counting logic
15:09:02 mriedem maybe someone like tim bell will remmeber
15:10:28 jaypipes mriedem: to understand the mystery of the server group member counting, one must first understand the mystery that is the server groups implementation.
15:11:25 mriedem you said you're cool with server groups
15:12:07 luksky sean-k-mooney: thank You, does it can have some relation with not running discovered_host on some (i.e 4 compute) computes ?
15:16:17 luksky sean-k-mooney: I mean "nova-manage cell_v2 discover_hosts"
15:22:38 jaypipes mriedem: the concept? yes. the implementation... erm, not so much.
15:23:46 mriedem efried_mtg: bauzas: commented on the semver version for the novaclient release https://review.openstack.org/#/c/641678/
15:23:48 mriedem i'm a bit torn
15:23:56 mriedem because of the first bullet here https://docs.openstack.org/releasenotes/python-novaclient/unreleased.html#upgrade-notes
15:24:41 openstackgerrit Boxiang Zhu proposed openstack/nova master: Raise error when force_raw_images and images_type are incompatible https://review.openstack.org/640271
15:25:21 bauzas mriedem: great comment
15:25:25 bauzas I missed that one
15:25:50 kashyap Dead-simple +2/+W, can anyone put this out of its misery: https://review.openstack.org/#/c/640866/
15:26:54 mriedem kashyap: uh...
15:27:08 mriedem kashyap: if that change isn't going to land in stein, the spec has to be re-proposed for train anyway,
15:27:20 mriedem so why not just make that change in the re-proposed spec with a note in the history abou the change?
15:28:03 kashyap Whatever works; alex_xu was also onboard with the above, FWIW.
15:28:09 kashyap mriedem: Will it / can it land for Stein?
15:28:37 kashyap mriedem: So, to be clear: you want me to repropose the whole spec fo Train?
15:28:44 mriedem kashyap: it would have to merge today
15:28:57 kashyap mriedem: Ah ... right, the freeze
15:29:14 mriedem i see alex_xu and stephenfin were reviewing it, but stephenfin was only reviewing docs
15:29:17 mriedem it looks like

Earlier   Later