| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 14:44:07 | mriedem | yeah probably | |
| 14:44:19 | dansmith | mriedem: you gonna do that? | |
| 14:44:24 | mriedem | dansmith: you do it | |
| 14:44:29 | bauzas | efried: wait | |
| 14:44:41 | efried | bauzas: So now we can land on host1 or host2 (but not host3). If we land on host1, we have to use that one allocation request for that guy. If we land on host2, we have a choice of three allocation requests. | |
| 14:44:44 | efried | bauzas: waiting... | |
| 14:45:18 | bauzas | efried: the fact that we have multiple allocation requests related to one host shouldn't impact filters | |
| 14:45:41 | dansmith | mriedem: actually melwitt should be around in 15 mins, so let's just wait | |
| 14:45:46 | bauzas | efried: because filters interface is against *host*, not allocation request | |
| 14:45:50 | dansmith | she clearly logged that she removed it yesterday | |
| 14:45:57 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: resource requests for nested providers https://review.openstack.org/554529 | |
| 14:46:48 | efried | bauzas: That's going to need to be rethought, then. | |
| 14:47:08 | bauzas | efried: what needs to rethought ? | |
| 14:47:10 | efried | bauzas: If we can't have NUMATopologyFilter winnow down the list of *allocation requests* then we're back to the drawing board. | |
| 14:47:32 | jaypipes | mriedem, dansmith: your review needed on https://review.openstack.org/#/c/556873/ pls (nested allocation candidates spec) | |
| 14:47:53 | bauzas | efried: I considered Placement as the way to winnow down the list of hosts we were checking | |
| 14:48:04 | bauzas | efried: so that's still a net win | |
| 14:48:07 | gameon | Hello all - I am trying to configure live migration between hosts, I have a Broadwell 56 core host and a 32 core Haswell server. Both single sockets. I have set cpu_mode=custom and cpu_model to various things, kvm64, haswell, SandyBridge - but still I get an error about CPU incompatibility when attempting to live migration from 32 to 56 core hosts. Is there any way of fixing this? I thought the configuration of custom mode woul | |
| 14:48:29 | mriedem | gameon: please see channel topic | |
| 14:48:32 | efried | bauzas: Because in our scenario above, we've filtered down to host1 and host2, great, let's say we pick host1 -we can't just pick *any* allocation request that relates to host1. Because some of those still have allocation requests where <#numa node RPs> != hw:numa_nodes from extra specs. | |
| 14:48:46 | gameon | mriedem: Sorry I missed that. | |
| 14:48:58 | bauzas | efried: where is the spec describing the query side of nested RPs ? | |
| 14:49:47 | bauzas | efried: probably worth hangouting you know | |
| 14:49:48 | efried | bauzas: https://review.openstack.org/#/c/556873/ | |
| 14:50:00 | efried | bauzas: But I'm not sure that's going to help you much. | |
| 14:50:20 | bauzas | efried: well, I'm not in need of anything | |
| 14:50:44 | bauzas | efried: here, I'm just saying "let's use Placement to winnow the list of hosts" | |
| 14:50:50 | bauzas | ah snap | |
| 14:50:56 | bauzas | it will work for NUMA | |
| 14:51:01 | bauzas | but not for VGPU | |
| 14:51:09 | bauzas | oh wait | |
| 14:51:11 | bauzas | it will | |
| 14:51:11 | efried | separate use case, let's focus on one at a time. | |
| 14:51:16 | bauzas | no no | |
| 14:51:17 | bauzas | sec | |
| 14:51:20 | efried | no, it won't. But separate use case. | |
| 14:51:25 | bauzas | trying to wrap around my heard | |
| 14:51:27 | bauzas | head | |
| 14:51:34 | bauzas | so | |
| 14:51:46 | bauzas | we said we're going to get a list of allocation requests | |
| 14:52:14 | bauzas | then, each filter will go thru the list of corresponding hosts and do the checks they want - which are unrelated to what Placement checked | |
| 14:52:40 | bauzas | at the end, we'll figure out that, say host345, host346 are valid | |
| 14:52:51 | bauzas | then, we'll look back at the allocation candidates | |
| 14:53:14 | bauzas | and claim against one of the allocation requests corresponding to those hosts | |
| 14:53:54 | bauzas | what filtering will do is just reducing that list of allocation requests to the ones that are related to hosts that match filtering | |
| 14:54:03 | efried | Right, but since the filter only filtered *hosts*, there can still be allocation requests we can't use. | |
| 14:54:13 | efried | Placement will winnow the list of hosts so far. The filter can winnow the list of hosts further. But - assuming the filter *only* returns a list of hosts - what's going to be left after that is still going to include allocation requests we can't use. And now there's nobody left to filter those out. | |
| 14:54:38 | efried | bauzas: So what I was saying we could reimagine was the role of filters in this flow. | |
| 14:54:41 | bauzas | efried: why should we filter more ? | |
| 14:54:49 | sahid | gameon: you should try with cpu_mode=none | |
| 14:55:03 | efried | bauzas: Instead of being just focused on hosts, a filter is allowed to filter allocation requests too. | |
| 14:55:08 | bauzas | efried: filters give you a subset of allocation requests that are supported | |
| 14:55:15 | bauzas | efried: please no. | |
| 14:55:23 | efried | bauzas: That's not what I understood from what you said above. | |
| 14:55:32 | sahid | so no check are done by in libvirt layer but it's possible that qemu failed to start the -incoming process on destination | |
| 14:55:37 | efried | bauzas: What I understood you to say was that filters give you a subset of *hosts*. | |
| 14:56:04 | bauzas | they give you a subset of hosts, which turns nova into knowing which allocation requests are valid accordingly | |
| 14:56:04 | sahid | at least you should see the log o QEMU which will indicated which cpu feature is not supported by destination | |
| 14:56:11 | efried | bauzas: Exactly | |
| 14:56:18 | bauzas | so, we're cool | |
| 14:56:29 | bauzas | in the example of NUMA | |
| 14:56:31 | efried | bauzas: No. The problem is that there are still invalid allocation requests for the remaining hosts | |
| 14:56:54 | bauzas | oh right, because the NUMA filter did crazy things | |
| 14:57:14 | bauzas | so it said "that NUMA node, I'll take it" | |
| 14:57:26 | bauzas | but it doesn't really do that you know | |
| 14:57:34 | bauzas | it just consider the host valid | |
| 14:57:50 | gameon | sahid: setting it to none makes no difference, I get the same error. I can't see which feature set isn't supported. I am beginning to think it's to do with the number of cores on the hosts. Even kvm64 doesn't work for live migration from 56 to 32 cores (it's a 1 core VM..) | |
| 14:57:52 | bauzas | efried: the NUMA attachment is a late bind on compute | |
| 14:58:07 | bauzas | efried: IIUC the construct | |
| 14:58:36 | sahid | gameon: what qemu is saying? /var/log/libvirt/qemu/instance-xxx.log | |
| 14:58:52 | efried | bauzas: Example: flavor says hw:numa_nodes=2. So we ask for resources1=VCPU:1&resources2=VCPU:1. We get back the following from GET /allocation_candidates: | |
| 14:58:53 | efried | ] | |
| 14:58:53 | efried | [ host2_NUMA1: { VCPU: 2 } ] | |
| 14:58:53 | efried | [ host1_NUMA0: { VCPU: 2 } ], | |
| 14:58:53 | efried | [ [ host1_NUMA0: { VCPU: 1 }, host1_NUMA1: { VCPU: 1 } ], | |
| 14:59:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | |
| 14:59:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560930 | |
| 14:59:25 | sahid | gameon: an other point, after updating nova.conf you should restart the service and also create a new guest | |
| 14:59:40 | efried | bauzas: NUMATopologyFilter looks through that, sees that host1 has a valid allocation request - the first one - because it represents two NUMA nodes and we want to adhere to hw:numa_nodes=2. | |
| 14:59:59 | efried | bauzas: So it chucks out host2, and returns "here is your valid list of hosts: [host1]" | |
| 15:00:22 | efried | bauzas: Next step in the flow says, "Oh, cool, host1 is fine; let's go see what allocation requests are available for host1" | |
| 15:00:32 | efried | bauzas: ...and it picks the second one, which is wrong. | |
| 15:00:52 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560931 | |
| 15:01:05 | bauzas | efried: so that's a good reason to port that logic to Placement then | |
| 15:01:32 | bauzas | so we can get rid of that crazy filter which makes assumptions that I disagree | |
| 15:01:33 | efried | bauzas: Or make a filter more fine-grained so that it actually returns the allocation requests instead of just the host name. | |
| 15:01:50 | bauzas | efried: the filters return a boolean | |
| 15:02:07 | gameon | sahid: Thanks for this, I am restarting the service and also the guest which starts it with the updated CPU feature. I have found ' host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]' so I guess that needs to be investigated | |
| 15:02:09 | bauzas | efried: what you ask is a post-placement-filtering | |
| 15:02:21 | bauzas | efried: not a scheduler filter adaptation | |
| 15:02:30 | bauzas | efried: wrt what dansmith implemented | |
| 15:02:48 | bauzas | whatever the name was, I don't recall exactly | |
| 15:02:56 | efried | bauzas: I admit I don't understand the sequence of events. But yes, I was originally assuming filters are (or can be) applied *after* GET /allocation_candidates. Is that wrong? | |
| 15:03:08 | bauzas | efried: you're still right | |
| 15:03:19 | efried | bauzas: I was also assuming there was nothing stopping us from allowing filters to cull allocation requests. | |
| 15:03:36 | bauzas | efried: the filtering logic is based on iterating over hosts | |
| 15:03:47 | efried | bauzas: But what's the input to the filter? | |
| 15:03:53 | bauzas | efried: sec, giving you code | |
| 15:03:55 | efried | bauzas: Does the filter get to see the result of GET /allocation_candidates? | |
| 15:04:12 | bauzas | as of now, not | |