| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-03 | |||
| 15:35:36 | dansmith | sean-k-mooney: why is that? shouldn't it come in via reqs? | |
| 15:36:10 | sean-k-mooney | the cyborg plugin does not install it | |
| 15:36:21 | sean-k-mooney | devstack plugin that is | |
| 15:36:31 | sean-k-mooney | its not needed for nova its just for osc | |
| 15:36:41 | sean-k-mooney | so i dont have to use curl | |
| 15:37:01 | dansmith | oh you mean you have to install the cyborg osc plugin? | |
| 15:37:08 | sean-k-mooney | ya | |
| 15:37:16 | dansmith | okay gotcha | |
| 15:37:31 | stephenfin | artom: Could just do a negative test? | |
| 15:37:41 | stephenfin | actually, nvm | |
| 15:37:49 | sean-k-mooney | thats why im also checking out the openstack sdk from a review branch | |
| 15:37:57 | sean-k-mooney | that isnt neededed by nova | |
| 15:38:09 | sean-k-mooney | its just for osc | |
| 15:38:18 | stephenfin | artom: I'm dumb and don't see the problem you're pointing out with that comment. Can you restate it for me? :) | |
| 15:38:45 | artom | stephenfin, image the ascii diagram on the first "line", but with VM1 on N2 | |
| 15:38:58 | mmidolesov | Does anayone know why functional-py36 returns TIMED_OUT in nova when rechecking patch? This happens for quite some time now. | |
| 15:39:30 | artom | If we then live migrate it, and we end up with the ascii diagram on the second "line", we have no way of knowing whether that's because NUMA live migration worked and updated the XML, or just because that's the old behaviour anyways | |
| 15:39:42 | sean-k-mooney | mmidolesov: in the gate? | |
| 15:39:59 | sean-k-mooney | i have not seen that on any patches i have reviewd lately | |
| 15:40:13 | stephenfin | Oh, so you're saying if it ended up landing on node 2, we wouldn't really be testing anything? Fair point. Could you just force the instances onto separate hosts though? | |
| 15:40:22 | stephenfin | Using the 'host' parameter from...2.53? | |
| 15:40:32 | sean-k-mooney | mmidolesov: if you are seing it on patche you pushed it might mean the mocking is not done correctly | |
| 15:41:07 | sean-k-mooney | stephenfin: the anti affintiy filer might be better | |
| 15:41:18 | stephenfin | sean-k-mooney: any reason? | |
| 15:41:28 | sean-k-mooney | im not sure tempest suppofrt the host parmater currently | |
| 15:41:32 | artom | stephenfin, we can force on separate hosts, but... can we force on specific NUMA nodes? Or, can we count that they'll be assigned to instances deterministically? | |
| 15:41:40 | sean-k-mooney | and we dont know what the host would be called generically | |
| 15:41:58 | sean-k-mooney | artom: no we cant force to a specif numa node | |
| 15:42:21 | artom | stephenfin, IOW, I want a situation where we *know for sure* that without the NUMA live migration feature, we'd have ended up with overlapping instances on the destination | |
| 15:42:41 | sean-k-mooney | we should not rely on the ordering | |
| 15:42:53 | sean-k-mooney | it tecnically is determisitic but only if we run serially | |
| 15:42:56 | artom | So 1. is that a fair thing to want? 2. can we achieve it with a less artificial NUMA topology | |
| 15:43:03 | sean-k-mooney | so we shoudl froce it with the pinsets | |
| 15:43:23 | sean-k-mooney | artom: no we do not want to exposee the ability to select a host numa node | |
| 15:43:41 | stephenfin | sean-k-mooney: we can't do different pinsets on different hosts | |
| 15:43:43 | artom | sean-k-mooney, except we can't have different pinsets per "host" in func tests | |
| 15:43:43 | sean-k-mooney | telcos want that but that is too invaisive to be a thing in the flavor | |
| 15:43:47 | stephenfin | not easily | |
| 15:44:03 | sean-k-mooney | stephenfin: in tempest we can are you talking about func tests? | |
| 15:44:08 | sean-k-mooney | i was assuming whitebox | |
| 15:44:22 | stephenfin | nah, artom's giant live migration functional test | |
| 15:44:30 | dansmith | Sundar: this can be abandoned, right? it still looks as if it's in the stack, despite being very outdated: https://review.opendev.org/#/c/673939/1 | |
| 15:45:10 | Sundar | sean-k-mooney: The local.conf seems fine to me. You are pulling in one more patch than what you need. | |
| 15:45:14 | Sundar | dansmith: Yes | |
| 15:45:30 | sean-k-mooney | Sundar: the uuid one | |
| 15:45:37 | sean-k-mooney | ya i know that log was annoying me | |
| 15:45:40 | Sundar | dansmith: I'll ping the developer | |
| 15:45:45 | dansmith | Sundar: I can do it | |
| 15:46:17 | Sundar | sean-k-mooney: The more egregious one about the bad UUID is already fixed: https://review.opendev.org/#/c/702785/ | |
| 15:46:24 | Sundar | dansmith: OK, thanks | |
| 15:46:39 | dansmith | Sundar: what about the two WIP ones underneath? | |
| 15:46:47 | dansmith | I think those are all effectively orphaned | |
| 15:47:30 | stephenfin | artom: create three instances | |
| 15:47:35 | Sundar | dansmith: Yes. The to-be-abandoned patch is based on very old versions of my patches. | |
| 15:47:49 | dansmith | Sundar: ack, will get those too | |
| 15:48:04 | dansmith | oh nm, I see, | |
| 15:48:13 | dansmith | those are when the current patches were marked as WIP, nevermind | |
| 15:48:26 | artom | stephenfin, go on... | |
| 15:48:28 | Sundar | But only 73939 nees to be abandoned | |
| 15:48:29 | stephenfin | artom: if you've two nodes on N1 and one on N2, move one of the former | |
| 15:48:47 | Sundar | dansmith: Others are part of the Nova patch series | |
| 15:48:53 | stephenfin | *two instances on N1 | |
| 15:49:04 | Sundar | dansmith: I meant 673939 | |
| 15:49:15 | stephenfin | and if you've one instance on N1 and two on N2, move one of the latter | |
| 15:49:20 | dansmith | Sundar: I abandoned two and it looks cleaner nwo | |
| 15:49:24 | stephenfin | guaranteed XML regeneration | |
| 15:49:24 | artom | stephenfin, that... seems like it's equally complex :/ | |
| 15:49:51 | artom | As in, we need to figure out the instance on the host with 2 of them that's pinned to the "equivalent" NUMA node as the other instance on the host with 1 instance | |
| 15:50:14 | Sundar | dansmith: https://review.opendev.org/#/c/631244/26 is part of the current patch series | |
| 15:50:35 | dansmith | Sundar: yes I see now, which is why I said this: [07:48:13] <dansmith>those are when the current patches were marked as WIP, nevermind | |
| 15:51:18 | artom | stephenfin, my "counter-offer" would be keep things as is, but with a nice ASCII diagram explaining the thing | |
| 15:52:49 | dansmith | efried: Sundar sean-k-mooney: I have a couple comments high up on the stack, but nothing critical.. I want to get another ack from sean-k-mooney's local testing, efried to look at the resources patch and then I think I'm good removing the -2 on the base patch | |
| 15:53:14 | dansmith | efried: alternately if you want to trade me the -2 responsibility on the base I can hold that torch | |
| 15:53:30 | efried | dansmith: let's do that, yah | |
| 15:53:56 | stephenfin | artom: I'm reluctant to do that because I'm supposed to know this stuff yet it takes me ages to re-parse this patch each time I review it :( | |
| 15:54:08 | stephenfin | so that doesn't bode well for long-term maintainability of same | |
| 15:54:55 | artom | stephenfin, I think any way you cut it it'll be complex :( | |
| 15:57:14 | sean-k-mooney | at somepoint having the ability to pass different config to different services/computes in the functional test would be useful. we often want to do it but never do because its complex to do | |
| 15:57:58 | artom | sean-k-mooney, CONF is global - I didn't find a way to make it work | |
| 15:58:33 | artom | stephenfin, ah, though of a thing: | |
| 15:58:52 | sean-k-mooney | it is but i think we could mock it differently per service i just have not trided doing that | |
| 15:59:04 | artom | 2 hosts, 2 NUMA nodes each, N1: CPUs 0, 1, N2: CPUs 3, 4 | |
| 15:59:15 | artom | Start with vcpu_dedicated_set = 0, 1 on both hosts, boot 2 instances | |
| 15:59:25 | artom | They're now both on CPUs 0, 1 | |
| 15:59:31 | artom | On different hosts | |
| 15:59:50 | artom | Change vcpu_dedicated_set to 0-4 on both hosts | |
| 15:59:55 | artom | Live migrate an instance | |
| 15:59:56 | efried | cdent did some work to make conf not global in placement (incl tests). Could follow that lead. | |
| 16:00:42 | sean-k-mooney | efried: ya i think he started passing the config to where its needed right? | |
| 16:01:13 | sean-k-mooney | i was thinking we might be able to have different code paths have different mocks and avoid that | |
| 16:01:28 | stephenfin | that could work. My only concern is that those config options are used for reporting the HostState object to the scheduler, so you'd have to restart all services I think | |
| 16:06:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Handle unset 'connection_info' https://review.opendev.org/686997 | |
| 16:06:45 | stephenfin | efried: Added the requested test for that oldish patch ^ | |
| 16:08:40 | stephenfin | artom: wait, we don't need to figure out what NUMA node is which. Force two instance to one host, the remainder to another, then live migrate/revert both of those first two instances? | |
| 16:08:58 | stephenfin | they should both end up on the same NUMA node when live migrated | |
| 16:09:14 | sean-k-mooney | stephenfin: that is what the intel nfv ci used to do | |
| 16:09:28 | sean-k-mooney | we booted 3 vms | |
| 16:09:43 | sean-k-mooney | 2 using anti affinity filter and then one more that we then live migrated | |
| 16:09:52 | sean-k-mooney | at least i think that is what we did | |
| 16:09:58 | sean-k-mooney | so that would work | |