| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 14:47:32 | jaypipes | mriedem, dansmith: your review needed on https://review.openstack.org/#/c/556873/ pls (nested allocation candidates spec) | |
| 14:47:53 | bauzas | efried: I considered Placement as the way to winnow down the list of hosts we were checking | |
| 14:48:04 | bauzas | efried: so that's still a net win | |
| 14:48:07 | gameon | Hello all - I am trying to configure live migration between hosts, I have a Broadwell 56 core host and a 32 core Haswell server. Both single sockets. I have set cpu_mode=custom and cpu_model to various things, kvm64, haswell, SandyBridge - but still I get an error about CPU incompatibility when attempting to live migration from 32 to 56 core hosts. Is there any way of fixing this? I thought the configuration of custom mode woul | |
| 14:48:29 | mriedem | gameon: please see channel topic | |
| 14:48:32 | efried | bauzas: Because in our scenario above, we've filtered down to host1 and host2, great, let's say we pick host1 -we can't just pick *any* allocation request that relates to host1. Because some of those still have allocation requests where <#numa node RPs> != hw:numa_nodes from extra specs. | |
| 14:48:46 | gameon | mriedem: Sorry I missed that. | |
| 14:48:58 | bauzas | efried: where is the spec describing the query side of nested RPs ? | |
| 14:49:47 | bauzas | efried: probably worth hangouting you know | |
| 14:49:48 | efried | bauzas: https://review.openstack.org/#/c/556873/ | |
| 14:50:00 | efried | bauzas: But I'm not sure that's going to help you much. | |
| 14:50:20 | bauzas | efried: well, I'm not in need of anything | |
| 14:50:44 | bauzas | efried: here, I'm just saying "let's use Placement to winnow the list of hosts" | |
| 14:50:50 | bauzas | ah snap | |
| 14:50:56 | bauzas | it will work for NUMA | |
| 14:51:01 | bauzas | but not for VGPU | |
| 14:51:09 | bauzas | oh wait | |
| 14:51:11 | efried | separate use case, let's focus on one at a time. | |
| 14:51:11 | bauzas | it will | |
| 14:51:16 | bauzas | no no | |
| 14:51:17 | bauzas | sec | |
| 14:51:20 | efried | no, it won't. But separate use case. | |
| 14:51:25 | bauzas | trying to wrap around my heard | |
| 14:51:27 | bauzas | head | |
| 14:51:34 | bauzas | so | |
| 14:51:46 | bauzas | we said we're going to get a list of allocation requests | |
| 14:52:14 | bauzas | then, each filter will go thru the list of corresponding hosts and do the checks they want - which are unrelated to what Placement checked | |
| 14:52:40 | bauzas | at the end, we'll figure out that, say host345, host346 are valid | |
| 14:52:51 | bauzas | then, we'll look back at the allocation candidates | |
| 14:53:14 | bauzas | and claim against one of the allocation requests corresponding to those hosts | |
| 14:53:54 | bauzas | what filtering will do is just reducing that list of allocation requests to the ones that are related to hosts that match filtering | |
| 14:54:03 | efried | Right, but since the filter only filtered *hosts*, there can still be allocation requests we can't use. | |
| 14:54:13 | efried | Placement will winnow the list of hosts so far. The filter can winnow the list of hosts further. But - assuming the filter *only* returns a list of hosts - what's going to be left after that is still going to include allocation requests we can't use. And now there's nobody left to filter those out. | |
| 14:54:38 | efried | bauzas: So what I was saying we could reimagine was the role of filters in this flow. | |
| 14:54:41 | bauzas | efried: why should we filter more ? | |
| 14:54:49 | sahid | gameon: you should try with cpu_mode=none | |
| 14:55:03 | efried | bauzas: Instead of being just focused on hosts, a filter is allowed to filter allocation requests too. | |
| 14:55:08 | bauzas | efried: filters give you a subset of allocation requests that are supported | |
| 14:55:15 | bauzas | efried: please no. | |
| 14:55:23 | efried | bauzas: That's not what I understood from what you said above. | |
| 14:55:32 | sahid | so no check are done by in libvirt layer but it's possible that qemu failed to start the -incoming process on destination | |
| 14:55:37 | efried | bauzas: What I understood you to say was that filters give you a subset of *hosts*. | |
| 14:56:04 | sahid | at least you should see the log o QEMU which will indicated which cpu feature is not supported by destination | |
| 14:56:04 | bauzas | they give you a subset of hosts, which turns nova into knowing which allocation requests are valid accordingly | |
| 14:56:11 | efried | bauzas: Exactly | |
| 14:56:18 | bauzas | so, we're cool | |
| 14:56:29 | bauzas | in the example of NUMA | |
| 14:56:31 | efried | bauzas: No. The problem is that there are still invalid allocation requests for the remaining hosts | |
| 14:56:54 | bauzas | oh right, because the NUMA filter did crazy things | |
| 14:57:14 | bauzas | so it said "that NUMA node, I'll take it" | |
| 14:57:26 | bauzas | but it doesn't really do that you know | |
| 14:57:34 | bauzas | it just consider the host valid | |
| 14:57:50 | gameon | sahid: setting it to none makes no difference, I get the same error. I can't see which feature set isn't supported. I am beginning to think it's to do with the number of cores on the hosts. Even kvm64 doesn't work for live migration from 56 to 32 cores (it's a 1 core VM..) | |
| 14:57:52 | bauzas | efried: the NUMA attachment is a late bind on compute | |
| 14:58:07 | bauzas | efried: IIUC the construct | |
| 14:58:36 | sahid | gameon: what qemu is saying? /var/log/libvirt/qemu/instance-xxx.log | |
| 14:58:52 | efried | bauzas: Example: flavor says hw:numa_nodes=2. So we ask for resources1=VCPU:1&resources2=VCPU:1. We get back the following from GET /allocation_candidates: | |
| 14:58:53 | efried | [ [ host1_NUMA0: { VCPU: 1 }, host1_NUMA1: { VCPU: 1 } ], | |
| 14:58:53 | efried | [ host1_NUMA0: { VCPU: 2 } ], | |
| 14:58:53 | efried | [ host2_NUMA1: { VCPU: 2 } ] | |
| 14:58:53 | efried | ] | |
| 14:59:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | |
| 14:59:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560930 | |
| 14:59:25 | sahid | gameon: an other point, after updating nova.conf you should restart the service and also create a new guest | |
| 14:59:40 | efried | bauzas: NUMATopologyFilter looks through that, sees that host1 has a valid allocation request - the first one - because it represents two NUMA nodes and we want to adhere to hw:numa_nodes=2. | |
| 14:59:59 | efried | bauzas: So it chucks out host2, and returns "here is your valid list of hosts: [host1]" | |
| 15:00:22 | efried | bauzas: Next step in the flow says, "Oh, cool, host1 is fine; let's go see what allocation requests are available for host1" | |
| 15:00:32 | efried | bauzas: ...and it picks the second one, which is wrong. | |
| 15:00:52 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560931 | |
| 15:01:05 | bauzas | efried: so that's a good reason to port that logic to Placement then | |
| 15:01:32 | bauzas | so we can get rid of that crazy filter which makes assumptions that I disagree | |
| 15:01:33 | efried | bauzas: Or make a filter more fine-grained so that it actually returns the allocation requests instead of just the host name. | |
| 15:01:50 | bauzas | efried: the filters return a boolean | |
| 15:02:07 | gameon | sahid: Thanks for this, I am restarting the service and also the guest which starts it with the updated CPU feature. I have found ' host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]' so I guess that needs to be investigated | |
| 15:02:09 | bauzas | efried: what you ask is a post-placement-filtering | |
| 15:02:21 | bauzas | efried: not a scheduler filter adaptation | |
| 15:02:30 | bauzas | efried: wrt what dansmith implemented | |
| 15:02:48 | bauzas | whatever the name was, I don't recall exactly | |
| 15:02:56 | efried | bauzas: I admit I don't understand the sequence of events. But yes, I was originally assuming filters are (or can be) applied *after* GET /allocation_candidates. Is that wrong? | |
| 15:03:08 | bauzas | efried: you're still right | |
| 15:03:19 | efried | bauzas: I was also assuming there was nothing stopping us from allowing filters to cull allocation requests. | |
| 15:03:36 | bauzas | efried: the filtering logic is based on iterating over hosts | |
| 15:03:47 | efried | bauzas: But what's the input to the filter? | |
| 15:03:53 | bauzas | efried: sec, giving you code | |
| 15:03:55 | efried | bauzas: Does the filter get to see the result of GET /allocation_candidates? | |
| 15:04:12 | bauzas | as of now, not | |
| 15:04:46 | bauzas | efried: that's the filter interface https://github.com/openstack/nova/blob/master/nova/scheduler/filters/__init__.py#L46 | |
| 15:04:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: RT: replace _instance_in_resize_state with _is_trackable_migration https://review.openstack.org/560467 | |
| 15:05:26 | bauzas | efried: and that's were we iterate over the list of hosts https://github.com/openstack/nova/blob/master/nova/filters.py#L43-L44 | |
| 15:05:43 | efried | bauzas: What's in filter_properties? | |
| 15:06:07 | bauzas | efried: it's the RequestSpec object | |
| 15:06:23 | bauzas | eg. https://github.com/openstack/nova/blob/master/nova/scheduler/filters/core_filter.py#L34 | |
| 15:06:36 | efried | bauzas: And does RequestSpec contain the GET /allocation_candidates results? | |
| 15:06:47 | bauzas | we could do that | |
| 15:06:55 | bauzas | but that's not yet the case | |
| 15:07:11 | bauzas | so we could pass the list of requests per host to the filter | |
| 15:07:17 | bauzas | up to the filter to read that info | |
| 15:07:56 | bauzas | or we could do a post-filtering-logic, exactly like we have now a pre-filtering-logic | |
| 15:07:59 | bauzas | well | |
| 15:08:02 | bauzas | pre-placement rather | |