| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-03 | |||
| 15:42:56 | artom | So 1. is that a fair thing to want? 2. can we achieve it with a less artificial NUMA topology | |
| 15:43:03 | sean-k-mooney | so we shoudl froce it with the pinsets | |
| 15:43:23 | sean-k-mooney | artom: no we do not want to exposee the ability to select a host numa node | |
| 15:43:41 | stephenfin | sean-k-mooney: we can't do different pinsets on different hosts | |
| 15:43:43 | artom | sean-k-mooney, except we can't have different pinsets per "host" in func tests | |
| 15:43:43 | sean-k-mooney | telcos want that but that is too invaisive to be a thing in the flavor | |
| 15:43:47 | stephenfin | not easily | |
| 15:44:03 | sean-k-mooney | stephenfin: in tempest we can are you talking about func tests? | |
| 15:44:08 | sean-k-mooney | i was assuming whitebox | |
| 15:44:22 | stephenfin | nah, artom's giant live migration functional test | |
| 15:44:30 | dansmith | Sundar: this can be abandoned, right? it still looks as if it's in the stack, despite being very outdated: https://review.opendev.org/#/c/673939/1 | |
| 15:45:10 | Sundar | sean-k-mooney: The local.conf seems fine to me. You are pulling in one more patch than what you need. | |
| 15:45:14 | Sundar | dansmith: Yes | |
| 15:45:30 | sean-k-mooney | Sundar: the uuid one | |
| 15:45:37 | sean-k-mooney | ya i know that log was annoying me | |
| 15:45:40 | Sundar | dansmith: I'll ping the developer | |
| 15:45:45 | dansmith | Sundar: I can do it | |
| 15:46:17 | Sundar | sean-k-mooney: The more egregious one about the bad UUID is already fixed: https://review.opendev.org/#/c/702785/ | |
| 15:46:24 | Sundar | dansmith: OK, thanks | |
| 15:46:39 | dansmith | Sundar: what about the two WIP ones underneath? | |
| 15:46:47 | dansmith | I think those are all effectively orphaned | |
| 15:47:30 | stephenfin | artom: create three instances | |
| 15:47:35 | Sundar | dansmith: Yes. The to-be-abandoned patch is based on very old versions of my patches. | |
| 15:47:49 | dansmith | Sundar: ack, will get those too | |
| 15:48:04 | dansmith | oh nm, I see, | |
| 15:48:13 | dansmith | those are when the current patches were marked as WIP, nevermind | |
| 15:48:26 | artom | stephenfin, go on... | |
| 15:48:28 | Sundar | But only 73939 nees to be abandoned | |
| 15:48:29 | stephenfin | artom: if you've two nodes on N1 and one on N2, move one of the former | |
| 15:48:47 | Sundar | dansmith: Others are part of the Nova patch series | |
| 15:48:53 | stephenfin | *two instances on N1 | |
| 15:49:04 | Sundar | dansmith: I meant 673939 | |
| 15:49:15 | stephenfin | and if you've one instance on N1 and two on N2, move one of the latter | |
| 15:49:20 | dansmith | Sundar: I abandoned two and it looks cleaner nwo | |
| 15:49:24 | stephenfin | guaranteed XML regeneration | |
| 15:49:24 | artom | stephenfin, that... seems like it's equally complex :/ | |
| 15:49:51 | artom | As in, we need to figure out the instance on the host with 2 of them that's pinned to the "equivalent" NUMA node as the other instance on the host with 1 instance | |
| 15:50:14 | Sundar | dansmith: https://review.opendev.org/#/c/631244/26 is part of the current patch series | |
| 15:50:35 | dansmith | Sundar: yes I see now, which is why I said this: [07:48:13] <dansmith>those are when the current patches were marked as WIP, nevermind | |
| 15:51:18 | artom | stephenfin, my "counter-offer" would be keep things as is, but with a nice ASCII diagram explaining the thing | |
| 15:52:49 | dansmith | efried: Sundar sean-k-mooney: I have a couple comments high up on the stack, but nothing critical.. I want to get another ack from sean-k-mooney's local testing, efried to look at the resources patch and then I think I'm good removing the -2 on the base patch | |
| 15:53:14 | dansmith | efried: alternately if you want to trade me the -2 responsibility on the base I can hold that torch | |
| 15:53:30 | efried | dansmith: let's do that, yah | |
| 15:53:56 | stephenfin | artom: I'm reluctant to do that because I'm supposed to know this stuff yet it takes me ages to re-parse this patch each time I review it :( | |
| 15:54:08 | stephenfin | so that doesn't bode well for long-term maintainability of same | |
| 15:54:55 | artom | stephenfin, I think any way you cut it it'll be complex :( | |
| 15:57:14 | sean-k-mooney | at somepoint having the ability to pass different config to different services/computes in the functional test would be useful. we often want to do it but never do because its complex to do | |
| 15:57:58 | artom | sean-k-mooney, CONF is global - I didn't find a way to make it work | |
| 15:58:33 | artom | stephenfin, ah, though of a thing: | |
| 15:58:52 | sean-k-mooney | it is but i think we could mock it differently per service i just have not trided doing that | |
| 15:59:04 | artom | 2 hosts, 2 NUMA nodes each, N1: CPUs 0, 1, N2: CPUs 3, 4 | |
| 15:59:15 | artom | Start with vcpu_dedicated_set = 0, 1 on both hosts, boot 2 instances | |
| 15:59:25 | artom | They're now both on CPUs 0, 1 | |
| 15:59:31 | artom | On different hosts | |
| 15:59:50 | artom | Change vcpu_dedicated_set to 0-4 on both hosts | |
| 15:59:55 | artom | Live migrate an instance | |
| 15:59:56 | efried | cdent did some work to make conf not global in placement (incl tests). Could follow that lead. | |
| 16:00:42 | sean-k-mooney | efried: ya i think he started passing the config to where its needed right? | |
| 16:01:13 | sean-k-mooney | i was thinking we might be able to have different code paths have different mocks and avoid that | |
| 16:01:28 | stephenfin | that could work. My only concern is that those config options are used for reporting the HostState object to the scheduler, so you'd have to restart all services I think | |
| 16:06:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Handle unset 'connection_info' https://review.opendev.org/686997 | |
| 16:06:45 | stephenfin | efried: Added the requested test for that oldish patch ^ | |
| 16:08:40 | stephenfin | artom: wait, we don't need to figure out what NUMA node is which. Force two instance to one host, the remainder to another, then live migrate/revert both of those first two instances? | |
| 16:08:58 | stephenfin | they should both end up on the same NUMA node when live migrated | |
| 16:09:14 | sean-k-mooney | stephenfin: that is what the intel nfv ci used to do | |
| 16:09:28 | sean-k-mooney | we booted 3 vms | |
| 16:09:43 | sean-k-mooney | 2 using anti affinity filter and then one more that we then live migrated | |
| 16:09:52 | sean-k-mooney | at least i think that is what we did | |
| 16:09:58 | sean-k-mooney | so that would work | |
| 16:10:25 | artom | stephenfin, ohhh, by live migrating 2 instances, we force that "overlap" | |
| 16:10:37 | sean-k-mooney | you dont need to migrate 2 | |
| 16:10:39 | artom | Well... | |
| 16:10:44 | artom | Not quite | |
| 16:10:51 | stephenfin | 'zactly. That seems super obvious | |
| 16:11:07 | sean-k-mooney | by booting 2 vms one on each if we rely on the ordering we know they will use teh cam cpus | |
| 16:11:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | |
| 16:11:41 | artom | stephenfin, heh, so to be honest, it's *not* obvious to me, but OK | |
| 16:11:53 | artom | Like, I need to think about that more than my 3/2 split :) | |
| 16:11:57 | sean-k-mooney | actully ya we just need 2 vms | |
| 16:12:13 | stephenfin | sean-k-mooney and I need to think about it less | |
| 16:12:16 | stephenfin | 2/1 split | |
| 16:12:17 | stephenfin | :P | |
| 16:12:54 | sean-k-mooney | if we are ok with relying on the ordering and call it out in the test with a doc comment | |
| 16:12:57 | gibi | stephenfin: you were +2 and and needed dansmith's oppinion on https://review.opendev.org/#/c/702261 now dansmith is +2 on that patch. Could you re-review at some point? | |
| 16:13:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475 | |
| 16:13:18 | stephenfin | sure, I'll grab it after this meeting | |
| 16:13:20 | sean-k-mooney | then just boot two vms with the host option to ensure they are booted on different hosts | |
| 16:13:25 | sean-k-mooney | and then live migate | |
| 16:13:38 | stephenfin | sean-k-mooney: so rely on the fact they'll land on NUMA node #0? | |
| 16:13:41 | artom | sean-k-mooney, and hope they both land on the "same" NUMA node on each host? | |
| 16:13:55 | sean-k-mooney | we use functools.permuations to iterate over the compute nodes in the same order | |
| 16:14:08 | sean-k-mooney | artom: the current libvirt code will | |
| 16:14:17 | sean-k-mooney | but its an implemenation detial | |
| 16:14:17 | artom | I suppose we could fail the test if we detect one of the instances went to NUMA node 1 | |
| 16:14:30 | sean-k-mooney | if we ever break that the fucntest will fail | |
| 16:14:35 | sean-k-mooney | but its valid for the current code | |
| 16:14:42 | stephenfin | I though we made a point of not making that contract, but that would work for me to keep this moving | |
| 16:14:53 | stephenfin | just stick the note in that we're making assumptions | |
| 16:14:56 | sean-k-mooney | oh actuly that wil break potentally with placment in the future | |
| 16:15:04 | stephenfin | we can rework the test then | |