| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-07 | |||
| 14:36:14 | cdent | sean-k-mooney, efried, edleafe: didn't we make a decision many months ago that we should just stick with, even if it isn't perfect? | |
| 14:36:33 | efried | this ^ | |
| 14:36:37 | efried | because there is no perfect | |
| 14:37:08 | sean-k-mooney | for the record we also commited to having the poc working by m1 at the latest | |
| 14:37:53 | edleafe | cdent: I guess I don't remember a decision, just lots of back-and-forth discussion. Maybe it was in Berlin, and that's why I don't recall it? | |
| 14:38:33 | cdent | edleafe: not certain. I _think_ I'm remembering Denver. | |
| 14:38:58 | efried | sean-k-mooney: m1 of what? At this point we're proposing it for train, since stein has sailed. The PoC is ready now. | |
| 14:39:08 | sean-k-mooney | cdent: this discusstion predates teh creation of placement. at least my involvement in it does | |
| 14:39:38 | cdent | sean-k-mooney: then I think you've proving my point | |
| 14:39:40 | sean-k-mooney | efried: the poc was expected to be completed by m1 of stein | |
| 14:39:55 | efried | sean-k-mooney: if we were going to get it into stein, sure. | |
| 14:40:23 | jaypipes | mriedem: is it a 0% chance of getting the quota-in-placement series reviewed and merged today then? even if I take over the bottom patch? there's actually a lot of good in that series and it opens up much-needed refactoring and cleanup in Train if we can get it done (not to mention a simpler way of reasoning about quotas) | |
| 14:40:36 | efried | cdent, edleafe: Denver is where consensus was reached. | |
| 14:40:52 | dansmith | jaypipes: yes, zero chance of landing the whole thing | |
| 14:40:53 | jaypipes | mriedem_afk: and yes, I realize I'm coming at this way late. | |
| 14:41:30 | dansmith | jaypipes: at best we could land the bottom few to get the data model in place, but the top several functional ones haven't seen review and rushing it today is not okay with me | |
| 14:46:47 | mriedem | jaypipes: dansmith: right if we take over the changes we'd get up through the online data migration, | |
| 14:46:59 | mriedem | which is arguably useful to get done earlier than later so there is less data to migrate in train, | |
| 14:47:06 | jaypipes | ++ | |
| 14:47:08 | mriedem | but if we're unsure of the design at this point then we should defer | |
| 14:48:40 | jaypipes | efried, edleafe, sean-k-mooney: I will remind yet again that there is zero generation protection for multiple threads changing a provider's traits. that protection only exists on the quantitative side of the house -- i.e. inventory and resource classes. | |
| 14:49:12 | mriedem | i'll dig through git history to see if i can find any words around why server group member quota is enforced at the user rather than project leve | |
| 14:49:15 | mriedem | *level | |
| 14:49:19 | sean-k-mooney | jaypipes: yep i know which is part of why i was hoping to avoid that. | |
| 14:49:21 | mriedem | i'm sure this is probably something melwitt would have memorized by now | |
| 14:49:34 | sean-k-mooney | jaypipes: anyway i dont want to get into it now | |
| 14:49:43 | efried | whoah, that I didn't realize. | |
| 14:49:52 | dansmith | mriedem: yeah I think we've been over that before.. definitely doesn't seem like something we can change on a whim regardless of the reason | |
| 14:50:24 | jaypipes | efried: I said it on the hangout the other day with Sundar and I've said it at each of the past PTGs where the whole custom RCs vs traits was discussed. | |
| 14:50:32 | efried | We should fix that regardless. | |
| 14:50:41 | efried | we've got multiple threads potentially dorking with traits already. | |
| 14:50:58 | mriedem | jaypipes: the good news is phil day was good about writing useful commit messages https://review.openstack.org/#/c/104957/ | |
| 14:52:05 | dansmith | mriedem: am I missing where in there is justifies being user-only? | |
| 14:52:30 | mriedem | "They can be defined per project or per user within a project" | |
| 14:52:32 | jaypipes | mriedem: "They can be defined per project or per user within a project" <-- shows me that user-specific quota *always* will be "within a project" and therefore should have had the project_id in the query. | |
| 14:52:33 | mriedem | i'm also looking | |
| 14:52:40 | jaypipes | lol. jiunx. | |
| 14:52:43 | edleafe | efried: capabilities don't change dynamically. State does. | |
| 14:53:09 | jaypipes | edleafe: zactly. | |
| 14:53:21 | dansmith | mriedem: yeah that just doesn't tell me why | |
| 14:53:59 | jaypipes | dansmith: fwiw, I wasn't wondering why it's a user-specific quota, just why the query for it wasn't also appending a project_id | |
| 14:54:09 | cdent | efried: we've got generations on adding traits to providers. | |
| 14:54:13 | dansmith | jaypipes: okay | |
| 14:54:35 | jaypipes | cdent: that's not what I'm referring to actually. :) | |
| 14:54:36 | mriedem | yeah idk either on the server group member user-only quota check, | |
| 14:54:40 | mriedem | if it was a mistake, | |
| 14:54:51 | mriedem | it sure farted it's way through several very large changes and different cores | |
| 14:55:05 | mriedem | too bad we don't have people around that maintain this code... | |
| 14:55:05 | cdent | jaypipes: well I kind of assumed not, but got a bit lost trying to track all of this round about | |
| 14:55:53 | mriedem | maybe sdague would remember.... | |
| 14:56:18 | cdent | so jaypipes what does "there is zero generation protection for multiple threads changing a provider's traits." | |
| 14:56:21 | jaypipes | cdent: specifically, this is the protection that doesn't exist when traits are used as faux inventory: https://github.com/openstack/placement/blob/master/placement/objects/allocation.py#L427-L447 | |
| 14:56:21 | mriedem | cyeoh and ken'ichi moved it through | |
| 14:56:33 | cdent | jinxish | |
| 14:56:35 | efried | cdent, jaypipes: Updating what traits are set on a resource provider <== has generation controls. Phew. | |
| 14:56:56 | jaypipes | efried: not for claiming resources. | |
| 14:57:02 | efried | oh, sure, traits aren't part of an allocation | |
| 14:57:04 | jaypipes | efried: which is what I've been saying. | |
| 14:57:05 | efried | that's just fine. | |
| 14:57:27 | efried | had me scared there for a minute | |
| 14:58:09 | cdent | jaypipes: so your concerned about the case where we've chosen a target on which we want to allocate and since we selected it, the traits of that thing have changed? | |
| 14:58:41 | efried | We also don't do ^ for aggregate memberships we originally used to gather the candidate. | |
| 14:58:42 | luksky | hi, I have question about scheduler - how to debug case where I have 26 compute nodes, but in first iteration I see that RetryFilter is getting only 22 at start ? | |
| 14:58:48 | cdent | (that either as "you're concerned about" or "your concern is". your pick) | |
| 14:58:49 | jaypipes | cdent: precisely, which is exactly what their whole "dynamic case" does. | |
| 14:59:28 | efried | We're not talking about changing traits continuously. Just when the characteristics (um, traits) of a resource provider change. | |
| 14:59:43 | cdent | luksky: depends on what release you're on | |
| 14:59:55 | luksky | queens | |
| 15:00:27 | luksky | [root@nova-01 ~]# nova --version | |
| 15:00:27 | luksky | 9.1.1 | |
| 15:01:03 | jaypipes | luksky: check the placement service's logs. | |
| 15:01:04 | cdent | luksky: combination of debug logs from nova-scheduler and nova-placement-api | |
| 15:01:14 | luksky | In logs I see: | |
| 15:01:16 | luksky | INFO nova.filters [req-6553ad52-40f0-41d3-a11e-3682bdccf589 f0b6fd8f6a1a421094fd6259fa0a8d6c a17c1ce3cc114f05b5da8c9553e5e4d1 - default default] Filtering removed all hosts for the request with instance ID 'dbaecc98-c097-4b7b-b4f8-459f1ed1fecc'. Filter results: ['RetryFilter: (start: 22, end: 22)', 'IsolatedTenantFilter: (start: 22, end: 4)', 'AggregateMultiTenancyIsolation: (start: 4, end: 4)', 'AvailabilityZoneFilter: (start: 4, end: 4)', | |
| 15:01:16 | luksky | 'RamFilter: (start: 4, end: 4)', 'DiskFilter: (start: 4, end: 4)', 'ComputeFilter: (start: 4, end: 4)', 'ComputeCapabilitiesFilter: (start: 4, end: 4)', 'ImagePropertiesFilter: (start: 4, end: 4)', 'ServerGroupAntiAffinityFilter: (start: 4, end: 4)', 'ServerGroupAffinityFilter: (start: 4, end: 4)', 'IsolatedHostsFilter: (start: 4, end: 0)'] | |
| 15:01:34 | efried | edleafe: "capabilities don't change dynamically. State does." What an FPGA is "capable" of doing *does* change dynamically, when the thing is programmed. | |
| 15:02:02 | cfriesen | efried: thanks for the update. :) You caught me just after I left. | |
| 15:02:14 | efried | cfriesen: yw | |
| 15:02:24 | sean-k-mooney | luksky: the retry filter will only get the set of host that passed the first iteration - the one that was tried and failed | |
| 15:02:51 | luksky | (os-client) (PROD)[root@test1 ~]# openstack compute service list --service nova-compute | grep nova-compute | wc -l | |
| 15:02:52 | luksky | WARNING: Failed to import plugin clustering. | |
| 15:02:52 | luksky | 26 | |
| 15:02:53 | edleafe | efried: an FPGA region is capable of all available bitstreams for that model. When it is programmed, it is still capable of running any of the other bitstreams | |
| 15:03:26 | edleafe | Calling the current program a capability instead of the current state of the region is kind of a twist in language | |
| 15:04:06 | luksky | hm... don | |
| 15:04:12 | luksky | hm... don't get it | |
| 15:04:40 | edleafe | cdent: try #openstack-etymology instead :) | |
| 15:04:49 | luksky | shouldn't see "'RetryFilter: (start: 26, end: 26)" instead: "'RetryFilter: (start: 22, end: 22)" | |
| 15:04:50 | luksky | ? | |
| 15:04:54 | cdent | naw, I think we need to go deeper | |
| 15:05:26 | efried | again, realizing it's not perfectly pure, this is the best way to get where we want to be. Which is why we've ended up here several times after having discussed it the same way every time. | |
| 15:06:03 | sean-k-mooney | luksky: no | |
| 15:06:38 | sean-k-mooney | luksky: the retry filter on subsequet iterations does not retry host it knows cant pass | |
| 15:07:16 | edleafe | I think calling something "perfectly pure" is a bit condescending. I'm just pointing out that this approach is not the way placement is designed to work, and there may be downsides. If you all have agreed to go ahead with that, your eyes are wide open. | |
| 15:07:47 | sean-k-mooney | luksky: if its the first iteration and no filter have ran then the delta between 26 and 22 is because placemnt determined the other 4 instace did not have space for the instance and elminated them | |
| 15:08:56 | mriedem | jaypipes: i've gone fishing in the ML for the mystery of the server_group_members counting logic | |
| 15:09:02 | mriedem | maybe someone like tim bell will remmeber | |
| 15:10:28 | jaypipes | mriedem: to understand the mystery of the server group member counting, one must first understand the mystery that is the server groups implementation. | |
| 15:11:25 | mriedem | you said you're cool with server groups | |
| 15:12:07 | luksky | sean-k-mooney: thank You, does it can have some relation with not running discovered_host on some (i.e 4 compute) computes ? | |