| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-03 | |||
| 15:53:14 | dansmith | efried: alternately if you want to trade me the -2 responsibility on the base I can hold that torch | |
| 15:53:30 | efried | dansmith: let's do that, yah | |
| 15:53:56 | stephenfin | artom: I'm reluctant to do that because I'm supposed to know this stuff yet it takes me ages to re-parse this patch each time I review it :( | |
| 15:54:08 | stephenfin | so that doesn't bode well for long-term maintainability of same | |
| 15:54:55 | artom | stephenfin, I think any way you cut it it'll be complex :( | |
| 15:57:14 | sean-k-mooney | at somepoint having the ability to pass different config to different services/computes in the functional test would be useful. we often want to do it but never do because its complex to do | |
| 15:57:58 | artom | sean-k-mooney, CONF is global - I didn't find a way to make it work | |
| 15:58:33 | artom | stephenfin, ah, though of a thing: | |
| 15:58:52 | sean-k-mooney | it is but i think we could mock it differently per service i just have not trided doing that | |
| 15:59:04 | artom | 2 hosts, 2 NUMA nodes each, N1: CPUs 0, 1, N2: CPUs 3, 4 | |
| 15:59:15 | artom | Start with vcpu_dedicated_set = 0, 1 on both hosts, boot 2 instances | |
| 15:59:25 | artom | They're now both on CPUs 0, 1 | |
| 15:59:31 | artom | On different hosts | |
| 15:59:50 | artom | Change vcpu_dedicated_set to 0-4 on both hosts | |
| 15:59:55 | artom | Live migrate an instance | |
| 15:59:56 | efried | cdent did some work to make conf not global in placement (incl tests). Could follow that lead. | |
| 16:00:42 | sean-k-mooney | efried: ya i think he started passing the config to where its needed right? | |
| 16:01:13 | sean-k-mooney | i was thinking we might be able to have different code paths have different mocks and avoid that | |
| 16:01:28 | stephenfin | that could work. My only concern is that those config options are used for reporting the HostState object to the scheduler, so you'd have to restart all services I think | |
| 16:06:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Handle unset 'connection_info' https://review.opendev.org/686997 | |
| 16:06:45 | stephenfin | efried: Added the requested test for that oldish patch ^ | |
| 16:08:40 | stephenfin | artom: wait, we don't need to figure out what NUMA node is which. Force two instance to one host, the remainder to another, then live migrate/revert both of those first two instances? | |
| 16:08:58 | stephenfin | they should both end up on the same NUMA node when live migrated | |
| 16:09:14 | sean-k-mooney | stephenfin: that is what the intel nfv ci used to do | |
| 16:09:28 | sean-k-mooney | we booted 3 vms | |
| 16:09:43 | sean-k-mooney | 2 using anti affinity filter and then one more that we then live migrated | |
| 16:09:52 | sean-k-mooney | at least i think that is what we did | |
| 16:09:58 | sean-k-mooney | so that would work | |
| 16:10:25 | artom | stephenfin, ohhh, by live migrating 2 instances, we force that "overlap" | |
| 16:10:37 | sean-k-mooney | you dont need to migrate 2 | |
| 16:10:39 | artom | Well... | |
| 16:10:44 | artom | Not quite | |
| 16:10:51 | stephenfin | 'zactly. That seems super obvious | |
| 16:11:07 | sean-k-mooney | by booting 2 vms one on each if we rely on the ordering we know they will use teh cam cpus | |
| 16:11:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | |
| 16:11:41 | artom | stephenfin, heh, so to be honest, it's *not* obvious to me, but OK | |
| 16:11:53 | artom | Like, I need to think about that more than my 3/2 split :) | |
| 16:11:57 | sean-k-mooney | actully ya we just need 2 vms | |
| 16:12:13 | stephenfin | sean-k-mooney and I need to think about it less | |
| 16:12:16 | stephenfin | 2/1 split | |
| 16:12:17 | stephenfin | :P | |
| 16:12:54 | sean-k-mooney | if we are ok with relying on the ordering and call it out in the test with a doc comment | |
| 16:12:57 | gibi | stephenfin: you were +2 and and needed dansmith's oppinion on https://review.opendev.org/#/c/702261 now dansmith is +2 on that patch. Could you re-review at some point? | |
| 16:13:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475 | |
| 16:13:18 | stephenfin | sure, I'll grab it after this meeting | |
| 16:13:20 | sean-k-mooney | then just boot two vms with the host option to ensure they are booted on different hosts | |
| 16:13:25 | sean-k-mooney | and then live migate | |
| 16:13:38 | stephenfin | sean-k-mooney: so rely on the fact they'll land on NUMA node #0? | |
| 16:13:41 | artom | sean-k-mooney, and hope they both land on the "same" NUMA node on each host? | |
| 16:13:55 | sean-k-mooney | we use functools.permuations to iterate over the compute nodes in the same order | |
| 16:14:08 | sean-k-mooney | artom: the current libvirt code will | |
| 16:14:17 | artom | I suppose we could fail the test if we detect one of the instances went to NUMA node 1 | |
| 16:14:17 | sean-k-mooney | but its an implemenation detial | |
| 16:14:30 | sean-k-mooney | if we ever break that the fucntest will fail | |
| 16:14:35 | sean-k-mooney | but its valid for the current code | |
| 16:14:42 | stephenfin | I though we made a point of not making that contract, but that would work for me to keep this moving | |
| 16:14:53 | stephenfin | just stick the note in that we're making assumptions | |
| 16:14:56 | sean-k-mooney | oh actuly that wil break potentally with placment in the future | |
| 16:15:04 | stephenfin | we can rework the test then | |
| 16:15:10 | sean-k-mooney | yep | |
| 16:15:12 | artom | And re-visit this conversation? | |
| 16:15:14 | artom | Ugh | |
| 16:15:44 | stephenfin | artom: But we'll all be rich and living in Barbados or somewhere by then | |
| 16:15:47 | sean-k-mooney | artom: without the ablity to pass differnt configs i think that is the simplest thing we can do at the momemt | |
| 16:15:48 | stephenfin | so what does it matter | |
| 16:16:03 | artom | stephenfin, *glare* | |
| 16:16:04 | sean-k-mooney | :) | |
| 16:40:53 | openstackgerrit | Mykola Yakovliev proposed openstack/nova master: Fix boot_roles in InstanceSystemMetadata https://review.opendev.org/698040 | |
| 17:09:02 | artom | sean-k-mooney, stephenfin, I *really* don't like relying on both VMs landing on NUMA node #0, because you made the very good point that it could change with placement | |
| 17:09:11 | artom | And I *don't* think pushing this conversation down the line is fair | |
| 17:09:32 | artom | So I'll got with my "change vcpu_dedicated_set" for both computes idea | |
| 17:10:33 | bauzas | sean-k-mooney: efried: trying to decode() https://review.opendev.org/#/c/552924/ | |
| 17:10:48 | bauzas | in particular what's written on https://etherpad.openstack.org/p/mem_page_size_and_placement | |
| 17:11:17 | efried | bauzas: Sure, ask away. | |
| 17:11:57 | sean-k-mooney | ill be back in two seconds just getting water | |
| 17:11:58 | bauzas | efried: so, the current proposal was nacked because of two things : | |
| 17:11:59 | efried | TL;DR: Memory in "grandchild" providers according to page size. | |
| 17:12:05 | bauzas | 1/ the memory split case | |
| 17:12:27 | bauzas | 2/ the group_policy=isolate issue | |
| 17:12:56 | bauzas | for 1/, IIUC the alternative that seems to have consensus is to have a grand-child RP | |
| 17:12:58 | efried | If you see it that crisply, cool. For me it was more a nebulous mess of "lots of things are hard to do" | |
| 17:13:14 | efried | Yes | |
| 17:13:15 | bauzas | I need to process this information but that seems reasonable to me | |
| 17:13:29 | bauzas | but for 2/, I don't see any counter-proposal | |
| 17:13:34 | efried | For 2/, the grandchild concept makes group_policy *moot* | |
| 17:13:56 | efried | meaning we can ignore it and let the other parts of the code (e.g. bandwidth stuff) set it as needed. | |
| 17:13:58 | bauzas | ah cool, I missed the whole point | |
| 17:14:12 | sean-k-mooney | bauzas: becasue of how it modeled in the tree with same_tree we dont care what group_policy is set to | |
| 17:14:14 | bauzas | ok, I'll sharp my pen and write a new rev | |
| 17:14:14 | efried | btw, this was one of the things we noticed when we were working on the 'nested magic' design for placement. | |
| 17:14:20 | sean-k-mooney | in the 3 level proposal | |
| 17:14:42 | efried | ...that group_policy no longer really makes sense. | |
| 17:14:44 | bauzas | that's a very good alternative then, thanks both of you for the better proposal | |
| 17:24:10 | sean-k-mooney | efried: one minor point. so we have same_subtree=!NUMA_1,NUMA_2 | |
| 17:24:18 | sean-k-mooney | right | |
| 17:24:49 | sean-k-mooney | we technically should be doing that for each of the multi numa ones if we want to skip that in the numa filter | |
| 17:24:50 | efried | I don't believe so, no. | |
| 17:25:42 | sean-k-mooney | ok then without group_policy=isolate we are not guarenteed that we get the correct behavior | |
| 17:26:30 | sean-k-mooney | that said the numa toplogy filter could do the check | |
| 17:26:41 | efried | sean-k-mooney: See https://docs.openstack.org/placement/train/specs/train/implemented/2005575-nested-magic-1.html#default-group-policy-to-none | |