Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-12
14:56:54 bauzas oh right, because the NUMA filter did crazy things
14:57:14 bauzas so it said "that NUMA node, I'll take it"
14:57:26 bauzas but it doesn't really do that you know
14:57:34 bauzas it just consider the host valid
14:57:50 gameon sahid: setting it to none makes no difference, I get the same error. I can't see which feature set isn't supported. I am beginning to think it's to do with the number of cores on the hosts. Even kvm64 doesn't work for live migration from 56 to 32 cores (it's a 1 core VM..)
14:57:52 bauzas efried: the NUMA attachment is a late bind on compute
14:58:07 bauzas efried: IIUC the construct
14:58:36 sahid gameon: what qemu is saying? /var/log/libvirt/qemu/instance-xxx.log
14:58:52 efried bauzas: Example: flavor says hw:numa_nodes=2. So we ask for resources1=VCPU:1&resources2=VCPU:1. We get back the following from GET /allocation_candidates:
14:58:53 efried [ [ host1_NUMA0: { VCPU: 1 }, host1_NUMA1: { VCPU: 1 } ],
14:58:53 efried [ host1_NUMA0: { VCPU: 2 } ],
14:58:53 efried [ host2_NUMA1: { VCPU: 2 } ]
14:58:53 efried ]
14:59:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317
14:59:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560930
14:59:25 sahid gameon: an other point, after updating nova.conf you should restart the service and also create a new guest
14:59:40 efried bauzas: NUMATopologyFilter looks through that, sees that host1 has a valid allocation request - the first one - because it represents two NUMA nodes and we want to adhere to hw:numa_nodes=2.
14:59:59 efried bauzas: So it chucks out host2, and returns "here is your valid list of hosts: [host1]"
15:00:22 efried bauzas: Next step in the flow says, "Oh, cool, host1 is fine; let's go see what allocation requests are available for host1"
15:00:32 efried bauzas: ...and it picks the second one, which is wrong.
15:00:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Remove the branch specifier from the nova-multiattach job https://review.openstack.org/560931
15:01:05 bauzas efried: so that's a good reason to port that logic to Placement then
15:01:32 bauzas so we can get rid of that crazy filter which makes assumptions that I disagree
15:01:33 efried bauzas: Or make a filter more fine-grained so that it actually returns the allocation requests instead of just the host name.
15:01:50 bauzas efried: the filters return a boolean
15:02:07 gameon sahid: Thanks for this, I am restarting the service and also the guest which starts it with the updated CPU feature. I have found ' host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]' so I guess that needs to be investigated
15:02:09 bauzas efried: what you ask is a post-placement-filtering
15:02:21 bauzas efried: not a scheduler filter adaptation
15:02:30 bauzas efried: wrt what dansmith implemented
15:02:48 bauzas whatever the name was, I don't recall exactly
15:02:56 efried bauzas: I admit I don't understand the sequence of events. But yes, I was originally assuming filters are (or can be) applied *after* GET /allocation_candidates. Is that wrong?
15:03:08 bauzas efried: you're still right
15:03:19 efried bauzas: I was also assuming there was nothing stopping us from allowing filters to cull allocation requests.
15:03:36 bauzas efried: the filtering logic is based on iterating over hosts
15:03:47 efried bauzas: But what's the input to the filter?
15:03:53 bauzas efried: sec, giving you code
15:03:55 efried bauzas: Does the filter get to see the result of GET /allocation_candidates?
15:04:12 bauzas as of now, not
15:04:46 bauzas efried: that's the filter interface https://github.com/openstack/nova/blob/master/nova/scheduler/filters/__init__.py#L46
15:04:52 openstackgerrit Matt Riedemann proposed openstack/nova master: RT: replace _instance_in_resize_state with _is_trackable_migration https://review.openstack.org/560467
15:05:26 bauzas efried: and that's were we iterate over the list of hosts https://github.com/openstack/nova/blob/master/nova/filters.py#L43-L44
15:05:43 efried bauzas: What's in filter_properties?
15:06:07 bauzas efried: it's the RequestSpec object
15:06:23 bauzas eg. https://github.com/openstack/nova/blob/master/nova/scheduler/filters/core_filter.py#L34
15:06:36 efried bauzas: And does RequestSpec contain the GET /allocation_candidates results?
15:06:47 bauzas we could do that
15:06:55 bauzas but that's not yet the case
15:07:11 bauzas so we could pass the list of requests per host to the filter
15:07:17 bauzas up to the filter to read that info
15:07:56 bauzas or we could do a post-filtering-logic, exactly like we have now a pre-filtering-logic
15:07:59 bauzas well
15:08:02 bauzas pre-placement rather
15:08:04 sahid gameon: a possible solution for you now is to set a model that is supported by both of your hosts
15:08:29 gameon sahid: actually that was a red herring, there is no warning in the qemu log when I try to live migrat - svm is for AMD anyway. I have tried setting the most basic model, kvm64 but still it doesnt work..
15:09:11 efried bauzas: post-filtering logic, at the level of allocation requests, is what is needed here, unless we're going to implement it in placement.
15:09:37 efried bauzas: Sorry, not post-filtering necessarily
15:10:04 bauzas efried: okay, I think I have enough in mind to write in a spec
15:10:05 efried bauzas: Filtering of the result of GET /allocation_candidates. Wherever/however that happens to be.
15:10:29 bauzas efried: so we could chime in that about the possible implementation options we have
15:11:40 efried bauzas: Yes. I see us at some point needing to do filtering of GET /allocation_candidates results. This would be a good use case for setting that up.
15:11:59 efried bauzas: By the way, I assume a weigher has the same issue. It just weighs hosts, not allocation requests.
15:12:12 efried bauzas: I think that needs to be re-imagined in the same way.
15:13:19 sahid gameon: if you set a specific model you have to configure cpu_mode=custom, that is what you did?
15:14:01 gameon sahid: Yeah I did do that, no dice :(
15:14:28 sahid gameon: that with master?
15:14:34 openstackgerrit Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597
15:14:45 gameon sahid: only on hypervisors, not the scheduler or api node
15:15:06 sahid gameon: i mean you are using master branch of nova?
15:15:27 gameon Pike
15:16:38 bauzas efried: I don't like the wording "reimagined" for filters and weighers
15:17:04 bauzas efried: given it's one of the most custom pieces we have in Nova, with a ton of operators having their own filters, changing that isn't trivial
15:17:20 bauzas efried: but adding some extra field to the RequestSpec object seems fine with me
15:17:32 efried bauzas: yes, fair enough; it may have to be a new/different kind of filter.
15:17:35 bauzas so that filters can opt-in and check if necessary
15:17:46 efried bauzas: Because filtering at the host level is simply not going to be enough long-term.
15:18:15 efried bauzas: It doesn't have to be customizable or opt-in for the use case we're talking about.
15:18:36 efried bauzas: Other than in the sense they're customizing/opting-in based on what they put in their flavor
15:18:45 jaypipes bauzas: Besides "check my switch to see if it's on before scheduling to this node", what are the custom filters/weighers you have seen from operators?
15:18:48 bauzas efried: the problem is really because of backwards compatibility you know
15:19:13 bauzas efried: for the VGPU usecase, I don't care about that because scheduler will pick one allocation for me
15:19:31 efried bauzas: I contend that this is the same.
15:19:32 bauzas I don't have to make sure to mimic any pre-existing logic
15:19:41 melwitt dansmith, mriedem: ah, dammit. I messed up thinking the cert one was supposed to end on 2018-04-11. sorry
15:19:46 efried bauzas: The scheduler has to pick one allocation request based on the numa topology in the flavor.
15:21:28 gameon sahid: OK - setting them to kvm64 has seemed to have a different result. Maybe I didn't restart the service. Now I have 'Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor'
15:22:33 bauzas efried: for VGPUs ?
15:22:42 efried bauzas: For NUMA
15:22:47 sahid gameon: ok in same time i was trying to find a patch which address an issue with compare CPU but it seems that it's already on Pike https://review.openstack.org/#/c/53746/
15:23:20 bauzas efried: I'm confused
15:23:27 efried bauzas: So maybe "filter" is the wrong word for it. The scheduler is going to need to pick from among the allocation requests returned by GET /a_c. It's gotta employ *some* kind of logic to do that.
15:23:39 sahid gameon: I can't help you for the issue with apparmar perhaps you could try #virt in OFTC
15:23:45 bauzas efried: I think we agreed something was necessary for keeping the existing behaviour
15:23:55 gameon sahid: Thank you for pointing me in the right direction, much appriciated
15:24:01 bauzas efried: because of some assumption from the filter
15:24:08 bauzas efried: that's usecase #1 in my spec
15:24:18 efried bauzas: In the case we're talking about, the NUMATopologyFilter may (or may not) have already filtered down to a certain subset of hosts. Now the scheduler knows it can ignore allocation requests related to those hosts. But it still has to employ additional logic to ignore allocation requests that don't have the right number of numa nodes in them.
15:24:26 bauzas efried: for usecase #2 (which also matches the SR-IOV usecase), we don't care
15:24:41 bauzas efried: right, I don't disagree with that
15:25:08 efried bauzas: That being the case, I'm not sure NUMATopologyFilter is actually doing us any good here.
15:25:29 bauzas that's the stephenfin vs. sahid point

Earlier   Later