Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-11
11:09:00 stephenfin efried: nope
11:09:16 stephenfin scroll up and look at the conversation between alex_xu and I about two hours ago
11:09:20 sean-k-mooney you get allocation from placmenet that are for VCPUs
11:09:30 sean-k-mooney but then the numa toplogy filter will reject all the hosts
11:09:47 sean-k-mooney stephenfin: do you want to explain why
11:09:48 efried ...because the host knows the VCPUs aren't dedicated.
11:09:52 efried ?
11:10:02 stephenfin efried: Yeah
11:10:05 efried which it doesn't know pre-Train, but in Train it does
11:10:21 stephenfin Placement will try for PCPU, not get it, and then try for VCPU
11:10:23 efried now, this is the version of the filter that runs on the compute host, not the scheduler, right?
11:10:36 stephenfin For Stein hosts, that's fine
11:10:37 sean-k-mooney efried: not its in the schduler
11:10:50 efried oh
11:10:51 stephenfin For Train hosts, the NUMATopologyFilter will then reject the host
11:10:55 sean-k-mooney stepehn modifled the host numa toplogy object
11:11:07 stephenfin and if that's disabled, it'll late fail on the compute node
11:11:08 efried ...so in a partially-upgraded scenario, how does the sched know the diff between stein & train hosts?
11:11:16 stephenfin NUMATopology.pcpuset
11:11:20 efried ack
11:11:22 efried it all comes together
11:11:26 stephenfin For Stein nodes, that isn't set
11:11:31 openstackgerrit Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452
11:11:43 donnyd sean-k-mooney: I made some adjustments yesterday and I am hoping that fixed the issue we are hainvg getting the numa job to run
11:11:56 alex_xu actually scheduler needn't know the diff of stein and trait
11:12:02 stephenfin For Train nodes with vcpu_pin_set, it's set to the same value as NUMATopology.cpuset
11:12:07 sean-k-mooney i only kicked off one run last night and it passed
11:12:18 stephenfin For Train nodes with cpu_dedicated_set, it's set to those cores
11:12:28 sean-k-mooney donnyd: there is a new version of the numa migration scripts so i can kick it off again to test tehm
11:12:28 stephenfin For Train nodes with _only_ cpu_shared_set, it's set to None
11:12:48 donnyd ok
11:13:10 sean-k-mooney alex_xu: we want the filter to know the difference so it can do the right thing in a mixed case
11:13:12 stephenfin We check to see if it's unset (meaning this is Stein compute node) and set it to the same value as NUMATopology.cpuset manually in the scheduler
11:13:21 donnyd lmk if there are more issues, I would like to get them ironed out so the label is actually usable
11:13:24 stephenfin However, if it's set to None, we do nothing, which means we'll fail to pin
11:13:35 openstackgerrit Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021
11:13:36 openstackgerrit Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595
11:13:49 stephenfin My functional tests prove that out. See here
11:14:23 alex_xu sean-k-mooney: the numa filter doesn't know. if stephenfin is going to update the patch, we will ensure we have value in host.pcpuset and host.cpuset. Then you asking pcpu, we get it from host.pcpuset, if you asking vcpu, then get it from host.cpuset
11:14:26 sean-k-mooney donnyd: sure. i vexhost and limestone might also be willing to provide the lables so we could maybe make this a voting job in the future
11:14:53 stephenfin https://review.opendev.org/#/c/681445/1/nova/tests/functional/libvirt/test_numa_servers.py@249
11:15:06 alex_xu ensure we have value in host.pcpuset and host.cpuset whatever it is stein node or train node.
11:15:14 stephenfin It's now the NUMATopologyFilter failing the request due to lack of PCPUs, not placement
11:15:55 stephenfin TIL 'git commit --amend --fixup=???' resets the author commit message (losing the change-ID). The more you know
11:17:39 efried stephenfin, sean-k-mooney: I re-commented. Good to move forward. Thanks for patiently explaining :)
11:18:06 sean-k-mooney we still need the numa toplogy filter for a release or two anyway since plament know nothing about numa yet so it will be enabled if you are using this feature anyway
11:18:27 efried yah
11:18:35 efried I was going to complain about that aspect, but yah.
11:19:29 sean-k-mooney once we fix placment of tis john snow syntrom re numa that is also proably the releas we want to drop this fallback and the numa toplogy filter
11:19:38 sean-k-mooney im guessing that will be V
11:19:44 openstackgerrit Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681383
11:20:13 sean-k-mooney assuming we make good progress in Ussuri on using the new placment features
11:20:56 sean-k-mooney that might be a bit aggressive but i dont think a short time line will be possible
11:21:23 sean-k-mooney we will likely want at least 1 release where the numa toplogy filter is optional
11:21:40 sean-k-mooney which hopefully could be ussuri?
11:22:35 sean-k-mooney we woudl need to get mempages,cpus,vgpus and pci devices reported in placment under numa nodes first
11:26:46 stephenfin efried: If that reworded comment looks good to you, I'll go ahead and squash that commit back
11:26:47 openstackgerrit Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681383
11:26:55 stephenfin in there ^, that is
11:28:22 bauzas dang
11:28:30 bauzas stephenfin: got questions for you https://review.opendev.org/#/c/675571/21
11:34:14 openstackgerrit sean mooney proposed openstack/nova master: [DNM] testing with new lable. https://review.opendev.org/680738
11:34:14 openstackgerrit sean mooney proposed openstack/nova master: [DNM] testing with old lable https://review.opendev.org/680739
11:37:07 sean-k-mooney artom: you patches just went into merge conflict
11:37:50 sean-k-mooney artom: looks like its only the functional tests
11:38:14 sean-k-mooney is it ok to rebase them
11:38:19 openstackgerrit Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453
11:38:36 sean-k-mooney i want to test the different nodepool lables
11:39:56 openstackgerrit Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454
11:41:17 artom sean-k-mooney, I know - I don't want to do it just yet
11:41:19 sean-k-mooney its such a dumb confilct
11:41:32 artom sean-k-mooney, make it Depends-on on the patch below it
11:41:41 sean-k-mooney <<<<<< HEAD
11:41:43 sean-k-mooney import fixtures
11:41:45 sean-k-mooney =======
11:41:46 artom I'd rather make mriedem's and dansmith's job easier at this point
11:41:47 sean-k-mooney import itertools
11:41:49 sean-k-mooney >>>>>>> Functional tests for NUMA live migration
11:41:59 sean-k-mooney ya ok i can do that
11:42:13 openstackgerrit Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455
11:42:41 sean-k-mooney both fixure and itertools are need by the way and its in alphabetical order so we jsut need to remove the conflict markers.
11:43:52 openstackgerrit Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456
11:46:13 openstackgerrit Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640
11:47:32 openstackgerrit Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470
11:49:34 efried stephenfin: commented. Sorry for being dense here.
11:51:34 openstackgerrit Luyao Zhong proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397
11:58:51 luyao sean-k-mooney: I gave some wrong infomation about vpmem yestoday, so sorry to misleading you. I clarified at here https://review.opendev.org/#/c/678455/24/nova/virt/libvirt/config.py@3180
12:00:59 sean-k-mooney ok so we dont need to subtract and it should be fine to just have 2KB lables since they are just markers
12:01:15 sean-k-mooney to denote the start and end of the namespace right?
12:02:48 luyao sean-k-mooney: yes, I think so
12:05:54 sean-k-mooney cool
12:07:56 efried luyao: Any update on the vpmem CI?
12:13:39 yaawang Hi, could anyone review this patch, it's part of vCPU models selection. https://review.opendev.org/#/c/670298
12:16:21 efried yaawang: I've been staying clear of that one because Alex is representing. bauzas or gibi_fly, might you have room for that ^
12:17:14 efried kashyap: Could you please clarify your opinion on https://review.opendev.org/#/c/678453/27/nova/virt/libvirt/driver.py@447 -- should we use InternalError to be consistent, or go rogue and use InvalidConfig?
12:19:51 efried stephenfin: also ^ when you get back
12:19:53 luyao efried: I know Dolpher and Rui is trying to apply the comunity patches then trigger the tests. Besides, I'm not very clear about that.
12:20:17 efried luyao: okay, thanks.
12:22:14 efried http://52.27.155.124/55/678455/28/check/pmem-tempest-plugin-filtered/7336479/job-output.txt.gz @ 2019-09-11 12:14:29.162146 looks like we're still getting PS13
12:22:22 efried that completed less than 10 minutes ago.

Earlier   Later