Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-11
10:25:42 openstackgerrit Stephen Finucane proposed openstack/nova master: DNM: Try to fallback to PCPU request when VCPU failed https://review.opendev.org/681383
10:25:56 stephenfin efried, bauzas: Can you sanity check before I waste more time fixing up tests ^
10:26:06 stephenfin the approach in general, that is
10:26:42 bauzas stephenfin: I left a comment on https://review.opendev.org/#/c/671800/
10:28:03 bauzas stephenfin: cool, I'll do it a bit later (needs to refuel my stomach)
10:32:46 efried stephenfin: sorry, looking at this patch for the first time, and... is this really the right approach? If we bounce with PCPUs, try VCPUs?? If they *have* cut over, and we're just *actually* out of PCPUs in the cloud, they're going to get *not* desired behavior, nah?
10:33:21 openstackgerrit Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResourceList objs https://review.opendev.org/678448
10:33:21 openstackgerrit Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447
10:33:22 openstackgerrit Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450
10:33:22 openstackgerrit Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449
11:04:37 sean-k-mooney efried: the numa toplogy filter will handel the out of PCPU case when they have swapped over to using cpu_dedicted_set
11:05:10 efried eh?
11:05:38 sean-k-mooney the numa toplogy filter will still run
11:06:03 sean-k-mooney it can tell the difference between a pre train host and a train host with non pCPUs free
11:06:04 efried If I'm out of PCPU resource per placement, this code will try the alloc request again with VCPUs. All of that happens before we get to the filter, yah?
11:06:18 sean-k-mooney yep
11:06:24 efried so... that's bad
11:06:47 sean-k-mooney and the filter will reject all host that are train but have not pcpus left when we are given vcpu allocations
11:07:05 efried that's not the scenario I'm concerned with.
11:07:15 sean-k-mooney what is?
11:07:51 efried Let's say I'm fully upgraded, with my confs properly cut over to cpu_{shared|dedicated}_set.
11:07:58 sean-k-mooney yep
11:08:01 efried this means my cloud has both PCPU and VCPU resources scattered around it.
11:08:05 sean-k-mooney yep
11:08:09 efried sometimes on the same host, maybe sometimes not, whatever.
11:08:16 sean-k-mooney sure
11:08:21 efried I request a dedicated VM.
11:08:35 efried But I'm genuinely out of PCPU resources in my cloud.
11:08:41 openstackgerrit Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681445
11:08:42 sean-k-mooney yep then we fall back
11:08:45 efried This code will bounce in GET /a_c and then *retry* with VCPUs instead.
11:08:53 efried So I'll wind up with my instance on VCPUs
11:08:58 sean-k-mooney no
11:09:00 stephenfin efried: nope
11:09:16 stephenfin scroll up and look at the conversation between alex_xu and I about two hours ago
11:09:20 sean-k-mooney you get allocation from placmenet that are for VCPUs
11:09:30 sean-k-mooney but then the numa toplogy filter will reject all the hosts
11:09:47 sean-k-mooney stephenfin: do you want to explain why
11:09:48 efried ...because the host knows the VCPUs aren't dedicated.
11:09:52 efried ?
11:10:02 stephenfin efried: Yeah
11:10:05 efried which it doesn't know pre-Train, but in Train it does
11:10:21 stephenfin Placement will try for PCPU, not get it, and then try for VCPU
11:10:23 efried now, this is the version of the filter that runs on the compute host, not the scheduler, right?
11:10:36 stephenfin For Stein hosts, that's fine
11:10:37 sean-k-mooney efried: not its in the schduler
11:10:50 efried oh
11:10:51 stephenfin For Train hosts, the NUMATopologyFilter will then reject the host
11:10:55 sean-k-mooney stepehn modifled the host numa toplogy object
11:11:07 stephenfin and if that's disabled, it'll late fail on the compute node
11:11:08 efried ...so in a partially-upgraded scenario, how does the sched know the diff between stein & train hosts?
11:11:16 stephenfin NUMATopology.pcpuset
11:11:20 efried ack
11:11:22 efried it all comes together
11:11:26 stephenfin For Stein nodes, that isn't set
11:11:31 openstackgerrit Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452
11:11:43 donnyd sean-k-mooney: I made some adjustments yesterday and I am hoping that fixed the issue we are hainvg getting the numa job to run
11:11:56 alex_xu actually scheduler needn't know the diff of stein and trait
11:12:02 stephenfin For Train nodes with vcpu_pin_set, it's set to the same value as NUMATopology.cpuset
11:12:07 sean-k-mooney i only kicked off one run last night and it passed
11:12:18 stephenfin For Train nodes with cpu_dedicated_set, it's set to those cores
11:12:28 stephenfin For Train nodes with _only_ cpu_shared_set, it's set to None
11:12:28 sean-k-mooney donnyd: there is a new version of the numa migration scripts so i can kick it off again to test tehm
11:12:48 donnyd ok
11:13:10 sean-k-mooney alex_xu: we want the filter to know the difference so it can do the right thing in a mixed case
11:13:12 stephenfin We check to see if it's unset (meaning this is Stein compute node) and set it to the same value as NUMATopology.cpuset manually in the scheduler
11:13:21 donnyd lmk if there are more issues, I would like to get them ironed out so the label is actually usable
11:13:24 stephenfin However, if it's set to None, we do nothing, which means we'll fail to pin
11:13:35 openstackgerrit Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021
11:13:36 openstackgerrit Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595
11:13:49 stephenfin My functional tests prove that out. See here
11:14:23 alex_xu sean-k-mooney: the numa filter doesn't know. if stephenfin is going to update the patch, we will ensure we have value in host.pcpuset and host.cpuset. Then you asking pcpu, we get it from host.pcpuset, if you asking vcpu, then get it from host.cpuset
11:14:26 sean-k-mooney donnyd: sure. i vexhost and limestone might also be willing to provide the lables so we could maybe make this a voting job in the future
11:14:53 stephenfin https://review.opendev.org/#/c/681445/1/nova/tests/functional/libvirt/test_numa_servers.py@249
11:15:06 alex_xu ensure we have value in host.pcpuset and host.cpuset whatever it is stein node or train node.
11:15:14 stephenfin It's now the NUMATopologyFilter failing the request due to lack of PCPUs, not placement
11:15:55 stephenfin TIL 'git commit --amend --fixup=???' resets the author commit message (losing the change-ID). The more you know
11:17:39 efried stephenfin, sean-k-mooney: I re-commented. Good to move forward. Thanks for patiently explaining :)
11:18:06 sean-k-mooney we still need the numa toplogy filter for a release or two anyway since plament know nothing about numa yet so it will be enabled if you are using this feature anyway
11:18:27 efried yah
11:18:35 efried I was going to complain about that aspect, but yah.
11:19:29 sean-k-mooney once we fix placment of tis john snow syntrom re numa that is also proably the releas we want to drop this fallback and the numa toplogy filter
11:19:38 sean-k-mooney im guessing that will be V
11:19:44 openstackgerrit Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681383
11:20:13 sean-k-mooney assuming we make good progress in Ussuri on using the new placment features
11:20:56 sean-k-mooney that might be a bit aggressive but i dont think a short time line will be possible
11:21:23 sean-k-mooney we will likely want at least 1 release where the numa toplogy filter is optional
11:21:40 sean-k-mooney which hopefully could be ussuri?
11:22:35 sean-k-mooney we woudl need to get mempages,cpus,vgpus and pci devices reported in placment under numa nodes first
11:26:46 stephenfin efried: If that reworded comment looks good to you, I'll go ahead and squash that commit back
11:26:47 openstackgerrit Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681383
11:26:55 stephenfin in there ^, that is
11:28:22 bauzas dang
11:28:30 bauzas stephenfin: got questions for you https://review.opendev.org/#/c/675571/21
11:34:14 openstackgerrit sean mooney proposed openstack/nova master: [DNM] testing with old lable https://review.opendev.org/680739
11:34:14 openstackgerrit sean mooney proposed openstack/nova master: [DNM] testing with new lable. https://review.opendev.org/680738
11:37:07 sean-k-mooney artom: you patches just went into merge conflict
11:37:50 sean-k-mooney artom: looks like its only the functional tests
11:38:14 sean-k-mooney is it ok to rebase them
11:38:19 openstackgerrit Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453

Earlier   Later