| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-28 | |||
| 09:08:14 | gibi | https://etherpad.opendev.org/p/blazar-preemptible-and-gpus L64 | |
| 09:08:23 | gibi | or more like L57 | |
| 09:11:28 | sean-k-mooney | jkulik: we only supprot this at the az level | |
| 09:11:33 | sean-k-mooney | well no | |
| 09:11:36 | sean-k-mooney | at the host level | |
| 09:11:42 | sean-k-mooney | you can use AZs per rack | |
| 09:11:57 | sean-k-mooney | but we have no aggreate or az anti affintiy concept in nova | |
| 09:12:06 | sean-k-mooney | and its not simple to add | |
| 09:12:28 | gibi | sean-k-mooney: I had a suggestion in the above etherpad how to add it to nova | |
| 09:12:31 | sean-k-mooney | jkulik: if you want to isolate custoemr we have tenat isolation filters | |
| 09:13:00 | sean-k-mooney | gibi: we coudl do it via placment aggreate if we wanted too | |
| 09:13:13 | gibi | sean-k-mooney: nope | |
| 09:13:18 | sean-k-mooney | but i was just sayign we dont currently support it | |
| 09:13:25 | gibi | yes, we not currently support it | |
| 09:13:40 | gibi | the placement way would be harder than a nova way as placmenet aggregates has no metadata | |
| 09:13:49 | sean-k-mooney | we have also rejected it in the past as we did not want to add more orchestration to nova | |
| 09:13:50 | gibi | while nova aggregates has metadata | |
| 09:14:18 | sean-k-mooney | well nova aggreate are mapped to palcemnt aggreates | |
| 09:14:27 | gibi | without the metadata piece | |
| 09:14:27 | sean-k-mooney | but sure for the rack affinty that is doable | |
| 09:14:46 | gibi | we need the metadata to mark an aggregate as target for affinity/anti-affinity | |
| 09:15:14 | sean-k-mooney | az anti affintiy which was the otehr request cant be done that way but we can do rack/row/room antiaffinity with aggreate metadta and a filter | |
| 09:15:29 | sean-k-mooney | gibi: ya so if wee were to do this i would make the filter generic | |
| 09:15:36 | gibi | sure | |
| 09:15:53 | gibi | that was an afterthought for me as well | |
| 09:15:54 | sean-k-mooney | so that you can define a set of lables and then express the anti affintiy requirement in the flavor | |
| 09:16:06 | sean-k-mooney | like you do with ceph | |
| 09:16:14 | sean-k-mooney | and the palcment maps | |
| 09:16:31 | gibi | yep generic label based affinity/anti-affinity either via the flavor or via the server group API | |
| 09:16:41 | sean-k-mooney | yep | |
| 09:17:08 | gibi | so we only need some devs to propose a spec and then the implementation :D | |
| 09:17:11 | gibi | easy peasy :D | |
| 09:17:12 | sean-k-mooney | that i would be ok with but you could map the info to placment too perhaps as a step 2 | |
| 09:17:32 | gibi | sean-k-mooney: you mean extend the aggregate concept in placement with metadta? | |
| 09:17:58 | sean-k-mooney | no we could likely modle this with aggreates and custom_traits | |
| 09:18:14 | sean-k-mooney | i have not fully tought that out | |
| 09:18:17 | gibi | I don't like it, as then the trait needs to be on the all the PRs in the aggregate | |
| 09:18:20 | sean-k-mooney | but it feels like we shoudl be able to do that | |
| 09:18:42 | sean-k-mooney | well i was thinking more like how misc_share_via_aggrate works | |
| 09:18:42 | gibi | technically doable but I would extend the aggregate concept instead in placement | |
| 09:19:13 | sean-k-mooney | i would be fine with extendign aggreates too if we had a clean way to extend it | |
| 09:19:36 | sean-k-mooney | perhaps traits on aggretes or some other metadta but doing it in nova first is alot simpler as you said | |
| 09:19:54 | sean-k-mooney | so get it working (nova) then make it fast (placment) | |
| 09:20:32 | gibi | we are in agreement :) | |
| 09:20:33 | sean-k-mooney | jkulik: interested in working on ^ | |
| 09:20:57 | sean-k-mooney | jkulik: its not an uncommen request so we can also try an pitch it to our pm dowstream | |
| 09:21:25 | sean-k-mooney | but no promices they will go for it and we wont look at it until at least the A cycle in any case | |
| 09:21:56 | sean-k-mooney | jkulik: but if you had time to work on it we have 2 week to spec freeze and we could proably reivew it if you had somethign ready before code freeze | |
| 09:22:10 | sean-k-mooney | jkulik:realisticly though its proably an A or lather cycle thing | |
| 09:22:20 | sean-k-mooney | but it woudl be doable in an out of tree schduler filter today | |
| 09:22:57 | sean-k-mooney | at least if you took the flavor approch initaly to expres the affinity requiremnt | |
| 09:23:32 | sean-k-mooney | extending teh server group api woudl require an api change which need a spec and is not backportable due to how api versioning works | |
| 09:24:14 | sean-k-mooney | gibi: what was the blazar usecase/imporance of this | |
| 09:24:45 | sean-k-mooney | i assume some sort of ha/fault tollernace usecase | |
| 09:25:12 | sean-k-mooney | oh no performance | |
| 09:25:13 | sean-k-mooney | i.e. reserve three GPU VMs in the same rack, where possible (hard/soft affinity). | |
| 09:25:15 | sean-k-mooney | Specifically this is to reduce network latency (MPI/horovod), and unnessacery TOR switch network contention. | |
| 09:25:21 | gibi | yepp | |
| 09:25:27 | sean-k-mooney | i mean i gues it could be used for either | |
| 09:25:38 | gibi | but there was limited time so we did not dig deep | |
| 09:25:44 | sean-k-mooney | ack | |
| 09:26:04 | sean-k-mooney | so there soft vs hard "requirement" would be filter vs weigher | |
| 09:26:20 | sean-k-mooney | but most of the logic would be the same so i woudl proably just do both | |
| 09:26:23 | jkulik | sean-k-mooney: I don't think I can make it in 2 weeks, but would be able to do it in the longer run | |
| 09:26:26 | sean-k-mooney | and try and shre the code | |
| 09:26:51 | sean-k-mooney | jkulik: ack | |
| 09:28:40 | jkulik | custom scheduler filter would be my idea, too. but we'd need it in the server-group API anyways as having one flavor per rack really doesn't scale | |
| 09:29:29 | sean-k-mooney | not per rack | |
| 09:29:36 | sean-k-mooney | so in the flavor you would have a policy | |
| 09:29:54 | sean-k-mooney | aggreate_anti_affinity:rack | |
| 09:30:12 | sean-k-mooney | aggreate_anti_affinity:room | |
| 09:30:26 | sean-k-mooney | aggreate_anti_affinity:row | |
| 09:30:30 | jkulik | ah, but still. customers are already overwhelmed by the number of flavors as is. I don't think we can add more for this :) | |
| 09:30:57 | sean-k-mooney | ya that is why the server-group api woudl be beeter | |
| 09:31:18 | sean-k-mooney | ther is a hack that you could use if you promise not to mention my name as the source :P | |
| 09:31:26 | jkulik | :D | |
| 09:31:37 | sean-k-mooney | so you could use server tags or instance metadta for now | |
| 09:31:42 | bauzas | I wonder how feasible a weigher could be | |
| 09:32:15 | bauzas | we could weigh per aggregatesd | |
| 09:32:17 | sean-k-mooney | so you could add aggreate_anti_affinity_* set of server tags | |
| 09:32:32 | sean-k-mooney | and then look at those in the filter/weigher | |
| 09:32:34 | bauzas | each host getting the same weigh | |
| 09:32:49 | jkulik | sean-k-mooney: that sounds wrong :D but thanks for the idea | |
| 09:32:51 | sean-k-mooney | bauzas: that is doable yes | |
| 09:33:32 | sean-k-mooney | jkulik: tags are somethimes used for this in other out of tree implemenation like tripleos instance ha feature | |
| 09:33:35 | jkulik | speaking of the server-group API: we've extended it to allow adding servers to server-groups after they're spawned. this can be helpful if you need to spawn a new instance anti-affin to a previously spawned one, where you didn't know that requirement, yet. | |
| 09:33:42 | bauzas | I'm still confused by the weights | |
| 09:33:48 | bauzas | but | |
| 09:33:48 | sean-k-mooney | its not how the api is inteded to be used but still | |
| 09:33:52 | jkulik | is that something that has a chance upstream, if we write a spec for it? | |
| 09:33:56 | bauzas | you could pass a hint | |
| 09:34:09 | bauzas | and then have a weigher looking up at the hint | |
| 09:34:14 | sean-k-mooney | schduler hint ya but that also versioned | |
| 09:34:26 | sean-k-mooney | bauzas: oh for the pack vs spread policy | |
| 09:34:33 | sean-k-mooney | for weighers | |
| 09:34:34 | bauzas | yup | |
| 09:35:06 | sean-k-mooney | ya so i was thining the weigher woudl jsut look at the metadta in the server-group policy | |
| 09:35:18 | sean-k-mooney | and ignore the multiplpere sign | |
| 09:35:31 | sean-k-mooney | or we woudl just set min =0 for the config option | |
| 09:35:47 | bauzas | I'm not in a favor of adding more in the existing server group API but... | |