Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-12
14:54:49 sahid gameon: you should try with cpu_mode=none
14:55:03 efried bauzas: Instead of being just focused on hosts, a filter is allowed to filter allocation requests too.
14:55:08 bauzas efried: filters give you a subset of allocation requests that are supported
14:55:15 bauzas efried: please no.
14:55:23 efried bauzas: That's not what I understood from what you said above.
14:55:32 sahid so no check are done by in libvirt layer but it's possible that qemu failed to start the -incoming process on destination
14:55:37 efried bauzas: What I understood you to say was that filters give you a subset of *hosts*.
14:56:04 bauzas they give you a subset of hosts, which turns nova into knowing which allocation requests are valid accordingly
14:56:04 sahid at least you should see the log o QEMU which will indicated which cpu feature is not supported by destination
14:56:11 efried bauzas: Exactly
14:56:18 bauzas so, we're cool
14:56:29 bauzas in the example of NUMA
14:56:31 efried bauzas: No. The problem is that there are still invalid allocation requests for the remaining hosts
14:56:54 bauzas oh right, because the NUMA filter did crazy things
14:57:14 bauzas so it said "that NUMA node, I'll take it"
14:57:26 bauzas but it doesn't really do that you know
14:57:34 bauzas it just consider the host valid
14:57:50 gameon sahid: setting it to none makes no difference, I get the same error. I can't see which feature set isn't supported. I am beginning to think it's to do with the number of cores on the hosts. Even kvm64 doesn't work for live migration from 56 to 32 cores (it's a 1 core VM..)
14:57:52 bauzas efried: the NUMA attachment is a late bind on compute
14:58:07 bauzas efried: IIUC the construct
14:58:36 sahid gameon: what qemu is saying? /var/log/libvirt/qemu/instance-xxx.log
14:58:52 efried bauzas: Example: flavor says hw:numa_nodes=2. So we ask for resources1=VCPU:1&resources2=VCPU:1. We get back the following from GET /allocation_candidates:
14:58:53 efried ]
14:58:53 efried [ host2_NUMA1: { VCPU: 2 } ]
14:58:53 efried [ host1_NUMA0: { VCPU: 2 } ],
14:58:53 efried [ [ host1_NUMA0: { VCPU: 1 }, host1_NUMA1: { VCPU: 1 } ],
14:59:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317
14:59:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560930
14:59:25 sahid gameon: an other point, after updating nova.conf you should restart the service and also create a new guest
14:59:40 efried bauzas: NUMATopologyFilter looks through that, sees that host1 has a valid allocation request - the first one - because it represents two NUMA nodes and we want to adhere to hw:numa_nodes=2.
14:59:59 efried bauzas: So it chucks out host2, and returns "here is your valid list of hosts: [host1]"
15:00:22 efried bauzas: Next step in the flow says, "Oh, cool, host1 is fine; let's go see what allocation requests are available for host1"
15:00:32 efried bauzas: ...and it picks the second one, which is wrong.
15:00:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560931
15:01:05 bauzas efried: so that's a good reason to port that logic to Placement then
15:01:32 bauzas so we can get rid of that crazy filter which makes assumptions that I disagree
15:01:33 efried bauzas: Or make a filter more fine-grained so that it actually returns the allocation requests instead of just the host name.
15:01:50 bauzas efried: the filters return a boolean
15:02:07 gameon sahid: Thanks for this, I am restarting the service and also the guest which starts it with the updated CPU feature. I have found ' host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]' so I guess that needs to be investigated
15:02:09 bauzas efried: what you ask is a post-placement-filtering
15:02:21 bauzas efried: not a scheduler filter adaptation
15:02:30 bauzas efried: wrt what dansmith implemented
15:02:48 bauzas whatever the name was, I don't recall exactly
15:02:56 efried bauzas: I admit I don't understand the sequence of events. But yes, I was originally assuming filters are (or can be) applied *after* GET /allocation_candidates. Is that wrong?
15:03:08 bauzas efried: you're still right
15:03:19 efried bauzas: I was also assuming there was nothing stopping us from allowing filters to cull allocation requests.
15:03:36 bauzas efried: the filtering logic is based on iterating over hosts
15:03:47 efried bauzas: But what's the input to the filter?
15:03:53 bauzas efried: sec, giving you code
15:03:55 efried bauzas: Does the filter get to see the result of GET /allocation_candidates?
15:04:12 bauzas as of now, not
15:04:46 bauzas efried: that's the filter interface https://github.com/openstack/nova/blob/master/nova/scheduler/filters/__init__.py#L46
15:04:52 openstackgerrit Matt Riedemann proposed openstack/nova master: RT: replace _instance_in_resize_state with _is_trackable_migration https://review.openstack.org/560467
15:05:26 bauzas efried: and that's were we iterate over the list of hosts https://github.com/openstack/nova/blob/master/nova/filters.py#L43-L44
15:05:43 efried bauzas: What's in filter_properties?
15:06:07 bauzas efried: it's the RequestSpec object
15:06:23 bauzas eg. https://github.com/openstack/nova/blob/master/nova/scheduler/filters/core_filter.py#L34
15:06:36 efried bauzas: And does RequestSpec contain the GET /allocation_candidates results?
15:06:47 bauzas we could do that
15:06:55 bauzas but that's not yet the case
15:07:11 bauzas so we could pass the list of requests per host to the filter
15:07:17 bauzas up to the filter to read that info
15:07:56 bauzas or we could do a post-filtering-logic, exactly like we have now a pre-filtering-logic
15:07:59 bauzas well
15:08:02 bauzas pre-placement rather
15:08:04 sahid gameon: a possible solution for you now is to set a model that is supported by both of your hosts
15:08:29 gameon sahid: actually that was a red herring, there is no warning in the qemu log when I try to live migrat - svm is for AMD anyway. I have tried setting the most basic model, kvm64 but still it doesnt work..
15:09:11 efried bauzas: post-filtering logic, at the level of allocation requests, is what is needed here, unless we're going to implement it in placement.
15:09:37 efried bauzas: Sorry, not post-filtering necessarily
15:10:04 bauzas efried: okay, I think I have enough in mind to write in a spec
15:10:05 efried bauzas: Filtering of the result of GET /allocation_candidates. Wherever/however that happens to be.
15:10:29 bauzas efried: so we could chime in that about the possible implementation options we have
15:11:40 efried bauzas: Yes. I see us at some point needing to do filtering of GET /allocation_candidates results. This would be a good use case for setting that up.
15:11:59 efried bauzas: By the way, I assume a weigher has the same issue. It just weighs hosts, not allocation requests.
15:12:12 efried bauzas: I think that needs to be re-imagined in the same way.
15:13:19 sahid gameon: if you set a specific model you have to configure cpu_mode=custom, that is what you did?
15:14:01 gameon sahid: Yeah I did do that, no dice :(
15:14:28 sahid gameon: that with master?
15:14:34 openstackgerrit Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597
15:14:45 gameon sahid: only on hypervisors, not the scheduler or api node
15:15:06 sahid gameon: i mean you are using master branch of nova?
15:15:27 gameon Pike
15:16:38 bauzas efried: I don't like the wording "reimagined" for filters and weighers
15:17:04 bauzas efried: given it's one of the most custom pieces we have in Nova, with a ton of operators having their own filters, changing that isn't trivial
15:17:20 bauzas efried: but adding some extra field to the RequestSpec object seems fine with me
15:17:32 efried bauzas: yes, fair enough; it may have to be a new/different kind of filter.
15:17:35 bauzas so that filters can opt-in and check if necessary
15:17:46 efried bauzas: Because filtering at the host level is simply not going to be enough long-term.
15:18:15 efried bauzas: It doesn't have to be customizable or opt-in for the use case we're talking about.
15:18:36 efried bauzas: Other than in the sense they're customizing/opting-in based on what they put in their flavor
15:18:45 jaypipes bauzas: Besides "check my switch to see if it's on before scheduling to this node", what are the custom filters/weighers you have seen from operators?
15:18:48 bauzas efried: the problem is really because of backwards compatibility you know
15:19:13 bauzas efried: for the VGPU usecase, I don't care about that because scheduler will pick one allocation for me
15:19:31 efried bauzas: I contend that this is the same.
15:19:32 bauzas I don't have to make sure to mimic any pre-existing logic
15:19:41 melwitt dansmith, mriedem: ah, dammit. I messed up thinking the cert one was supposed to end on 2018-04-11. sorry
15:19:46 efried bauzas: The scheduler has to pick one allocation request based on the numa topology in the flavor.
15:21:28 gameon sahid: OK - setting them to kvm64 has seemed to have a different result. Maybe I didn't restart the service. Now I have 'Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor'
15:22:33 bauzas efried: for VGPUs ?
15:22:42 efried bauzas: For NUMA

Earlier   Later