Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-12
14:48:07 gameon Hello all - I am trying to configure live migration between hosts, I have a Broadwell 56 core host and a 32 core Haswell server. Both single sockets. I have set cpu_mode=custom and cpu_model to various things, kvm64, haswell, SandyBridge - but still I get an error about CPU incompatibility when attempting to live migration from 32 to 56 core hosts. Is there any way of fixing this? I thought the configuration of custom mode woul
14:48:29 mriedem gameon: please see channel topic
14:48:32 efried bauzas: Because in our scenario above, we've filtered down to host1 and host2, great, let's say we pick host1 -we can't just pick *any* allocation request that relates to host1. Because some of those still have allocation requests where <#numa node RPs> != hw:numa_nodes from extra specs.
14:48:46 gameon mriedem: Sorry I missed that.
14:48:58 bauzas efried: where is the spec describing the query side of nested RPs ?
14:49:47 bauzas efried: probably worth hangouting you know
14:49:48 efried bauzas: https://review.openstack.org/#/c/556873/
14:50:00 efried bauzas: But I'm not sure that's going to help you much.
14:50:20 bauzas efried: well, I'm not in need of anything
14:50:44 bauzas efried: here, I'm just saying "let's use Placement to winnow the list of hosts"
14:50:50 bauzas ah snap
14:50:56 bauzas it will work for NUMA
14:51:01 bauzas but not for VGPU
14:51:09 bauzas oh wait
14:51:11 bauzas it will
14:51:11 efried separate use case, let's focus on one at a time.
14:51:16 bauzas no no
14:51:17 bauzas sec
14:51:20 efried no, it won't. But separate use case.
14:51:25 bauzas trying to wrap around my heard
14:51:27 bauzas head
14:51:34 bauzas so
14:51:46 bauzas we said we're going to get a list of allocation requests
14:52:14 bauzas then, each filter will go thru the list of corresponding hosts and do the checks they want - which are unrelated to what Placement checked
14:52:40 bauzas at the end, we'll figure out that, say host345, host346 are valid
14:52:51 bauzas then, we'll look back at the allocation candidates
14:53:14 bauzas and claim against one of the allocation requests corresponding to those hosts
14:53:54 bauzas what filtering will do is just reducing that list of allocation requests to the ones that are related to hosts that match filtering
14:54:03 efried Right, but since the filter only filtered *hosts*, there can still be allocation requests we can't use.
14:54:13 efried Placement will winnow the list of hosts so far. The filter can winnow the list of hosts further. But - assuming the filter *only* returns a list of hosts - what's going to be left after that is still going to include allocation requests we can't use. And now there's nobody left to filter those out.
14:54:38 efried bauzas: So what I was saying we could reimagine was the role of filters in this flow.
14:54:41 bauzas efried: why should we filter more ?
14:54:49 sahid gameon: you should try with cpu_mode=none
14:55:03 efried bauzas: Instead of being just focused on hosts, a filter is allowed to filter allocation requests too.
14:55:08 bauzas efried: filters give you a subset of allocation requests that are supported
14:55:15 bauzas efried: please no.
14:55:23 efried bauzas: That's not what I understood from what you said above.
14:55:32 sahid so no check are done by in libvirt layer but it's possible that qemu failed to start the -incoming process on destination
14:55:37 efried bauzas: What I understood you to say was that filters give you a subset of *hosts*.
14:56:04 bauzas they give you a subset of hosts, which turns nova into knowing which allocation requests are valid accordingly
14:56:04 sahid at least you should see the log o QEMU which will indicated which cpu feature is not supported by destination
14:56:11 efried bauzas: Exactly
14:56:18 bauzas so, we're cool
14:56:29 bauzas in the example of NUMA
14:56:31 efried bauzas: No. The problem is that there are still invalid allocation requests for the remaining hosts
14:56:54 bauzas oh right, because the NUMA filter did crazy things
14:57:14 bauzas so it said "that NUMA node, I'll take it"
14:57:26 bauzas but it doesn't really do that you know
14:57:34 bauzas it just consider the host valid
14:57:50 gameon sahid: setting it to none makes no difference, I get the same error. I can't see which feature set isn't supported. I am beginning to think it's to do with the number of cores on the hosts. Even kvm64 doesn't work for live migration from 56 to 32 cores (it's a 1 core VM..)
14:57:52 bauzas efried: the NUMA attachment is a late bind on compute
14:58:07 bauzas efried: IIUC the construct
14:58:36 sahid gameon: what qemu is saying? /var/log/libvirt/qemu/instance-xxx.log
14:58:52 efried bauzas: Example: flavor says hw:numa_nodes=2. So we ask for resources1=VCPU:1&resources2=VCPU:1. We get back the following from GET /allocation_candidates:
14:58:53 efried ]
14:58:53 efried [ host2_NUMA1: { VCPU: 2 } ]
14:58:53 efried [ host1_NUMA0: { VCPU: 2 } ],
14:58:53 efried [ [ host1_NUMA0: { VCPU: 1 }, host1_NUMA1: { VCPU: 1 } ],
14:59:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317
14:59:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560930
14:59:25 sahid gameon: an other point, after updating nova.conf you should restart the service and also create a new guest
14:59:40 efried bauzas: NUMATopologyFilter looks through that, sees that host1 has a valid allocation request - the first one - because it represents two NUMA nodes and we want to adhere to hw:numa_nodes=2.
14:59:59 efried bauzas: So it chucks out host2, and returns "here is your valid list of hosts: [host1]"
15:00:22 efried bauzas: Next step in the flow says, "Oh, cool, host1 is fine; let's go see what allocation requests are available for host1"
15:00:32 efried bauzas: ...and it picks the second one, which is wrong.
15:00:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560931
15:01:05 bauzas efried: so that's a good reason to port that logic to Placement then
15:01:32 bauzas so we can get rid of that crazy filter which makes assumptions that I disagree
15:01:33 efried bauzas: Or make a filter more fine-grained so that it actually returns the allocation requests instead of just the host name.
15:01:50 bauzas efried: the filters return a boolean
15:02:07 gameon sahid: Thanks for this, I am restarting the service and also the guest which starts it with the updated CPU feature. I have found ' host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]' so I guess that needs to be investigated
15:02:09 bauzas efried: what you ask is a post-placement-filtering
15:02:21 bauzas efried: not a scheduler filter adaptation
15:02:30 bauzas efried: wrt what dansmith implemented
15:02:48 bauzas whatever the name was, I don't recall exactly
15:02:56 efried bauzas: I admit I don't understand the sequence of events. But yes, I was originally assuming filters are (or can be) applied *after* GET /allocation_candidates. Is that wrong?
15:03:08 bauzas efried: you're still right
15:03:19 efried bauzas: I was also assuming there was nothing stopping us from allowing filters to cull allocation requests.
15:03:36 bauzas efried: the filtering logic is based on iterating over hosts
15:03:47 efried bauzas: But what's the input to the filter?
15:03:53 bauzas efried: sec, giving you code
15:03:55 efried bauzas: Does the filter get to see the result of GET /allocation_candidates?
15:04:12 bauzas as of now, not
15:04:46 bauzas efried: that's the filter interface https://github.com/openstack/nova/blob/master/nova/scheduler/filters/__init__.py#L46
15:04:52 openstackgerrit Matt Riedemann proposed openstack/nova master: RT: replace _instance_in_resize_state with _is_trackable_migration https://review.openstack.org/560467
15:05:26 bauzas efried: and that's were we iterate over the list of hosts https://github.com/openstack/nova/blob/master/nova/filters.py#L43-L44
15:05:43 efried bauzas: What's in filter_properties?
15:06:07 bauzas efried: it's the RequestSpec object
15:06:23 bauzas eg. https://github.com/openstack/nova/blob/master/nova/scheduler/filters/core_filter.py#L34
15:06:36 efried bauzas: And does RequestSpec contain the GET /allocation_candidates results?
15:06:47 bauzas we could do that
15:06:55 bauzas but that's not yet the case
15:07:11 bauzas so we could pass the list of requests per host to the filter
15:07:17 bauzas up to the filter to read that info
15:07:56 bauzas or we could do a post-filtering-logic, exactly like we have now a pre-filtering-logic
15:07:59 bauzas well
15:08:02 bauzas pre-placement rather
15:08:04 sahid gameon: a possible solution for you now is to set a model that is supported by both of your hosts
15:08:29 gameon sahid: actually that was a red herring, there is no warning in the qemu log when I try to live migrat - svm is for AMD anyway. I have tried setting the most basic model, kvm64 but still it doesnt work..
15:09:11 efried bauzas: post-filtering logic, at the level of allocation requests, is what is needed here, unless we're going to implement it in placement.

Earlier   Later