Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-11
10:09:44 stephenfin but __contains__ was implemented many moons ago
10:09:59 stephenfin anywhere is particular?
10:17:39 bauzas nah it's okay then
10:18:02 bauzas my o.vo skills are a bit rusty
10:18:07 bauzas but I could have tested it
10:19:04 sean-k-mooney efried: after 6 year of working on nova the main lesson i have learns is unlesss you have 2 cores from different compaines lined up to review the think you want to do before you start it will take 2 cycles to complete
10:19:39 sean-k-mooney the first cycle you get it working and people beging to become familar with and the second you finall land the thing
10:20:22 efried sean-k-mooney: I would love to be able to commit cores to review blueprints before/as they are approved. I just a) doubt cores would be willing to commit, and b) don't want to do that much PM type work
10:21:43 sean-k-mooney its partly a social thing. you need to explain why this thing you care about is imporant so that people will be interested enogh to review and demonstrate it works and is sane.
10:22:21 sean-k-mooney the non craze telco things i have worked on are less hard to enabel but im normlally in craze telco land
10:22:26 efried It sucks that "be interested" is a criterion
10:23:31 sean-k-mooney well it only is in terms of making it higher on list of other things you have to review
10:24:03 sean-k-mooney there are only a limit number of sme fore each system and they are overloaded frequently
10:25:14 sean-k-mooney if we had more subsystem maintainer or more reviewres it would help but we work with what we got.
10:25:31 sean-k-mooney we still get a lot done each cycle
10:25:42 sean-k-mooney the latency can just be a bit high sometimes
10:25:42 openstackgerrit Stephen Finucane proposed openstack/nova master: DNM: Try to fallback to PCPU request when VCPU failed https://review.opendev.org/681383
10:25:56 stephenfin efried, bauzas: Can you sanity check before I waste more time fixing up tests ^
10:26:06 stephenfin the approach in general, that is
10:26:42 bauzas stephenfin: I left a comment on https://review.opendev.org/#/c/671800/
10:28:03 bauzas stephenfin: cool, I'll do it a bit later (needs to refuel my stomach)
10:32:46 efried stephenfin: sorry, looking at this patch for the first time, and... is this really the right approach? If we bounce with PCPUs, try VCPUs?? If they *have* cut over, and we're just *actually* out of PCPUs in the cloud, they're going to get *not* desired behavior, nah?
10:33:21 openstackgerrit Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResourceList objs https://review.opendev.org/678448
10:33:21 openstackgerrit Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447
10:33:22 openstackgerrit Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450
10:33:22 openstackgerrit Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449
11:04:37 sean-k-mooney efried: the numa toplogy filter will handel the out of PCPU case when they have swapped over to using cpu_dedicted_set
11:05:10 efried eh?
11:05:38 sean-k-mooney the numa toplogy filter will still run
11:06:03 sean-k-mooney it can tell the difference between a pre train host and a train host with non pCPUs free
11:06:04 efried If I'm out of PCPU resource per placement, this code will try the alloc request again with VCPUs. All of that happens before we get to the filter, yah?
11:06:18 sean-k-mooney yep
11:06:24 efried so... that's bad
11:06:47 sean-k-mooney and the filter will reject all host that are train but have not pcpus left when we are given vcpu allocations
11:07:05 efried that's not the scenario I'm concerned with.
11:07:15 sean-k-mooney what is?
11:07:51 efried Let's say I'm fully upgraded, with my confs properly cut over to cpu_{shared|dedicated}_set.
11:07:58 sean-k-mooney yep
11:08:01 efried this means my cloud has both PCPU and VCPU resources scattered around it.
11:08:05 sean-k-mooney yep
11:08:09 efried sometimes on the same host, maybe sometimes not, whatever.
11:08:16 sean-k-mooney sure
11:08:21 efried I request a dedicated VM.
11:08:35 efried But I'm genuinely out of PCPU resources in my cloud.
11:08:41 openstackgerrit Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681445
11:08:42 sean-k-mooney yep then we fall back
11:08:45 efried This code will bounce in GET /a_c and then *retry* with VCPUs instead.
11:08:53 efried So I'll wind up with my instance on VCPUs
11:08:58 sean-k-mooney no
11:09:00 stephenfin efried: nope
11:09:16 stephenfin scroll up and look at the conversation between alex_xu and I about two hours ago
11:09:20 sean-k-mooney you get allocation from placmenet that are for VCPUs
11:09:30 sean-k-mooney but then the numa toplogy filter will reject all the hosts
11:09:47 sean-k-mooney stephenfin: do you want to explain why
11:09:48 efried ...because the host knows the VCPUs aren't dedicated.
11:09:52 efried ?
11:10:02 stephenfin efried: Yeah
11:10:05 efried which it doesn't know pre-Train, but in Train it does
11:10:21 stephenfin Placement will try for PCPU, not get it, and then try for VCPU
11:10:23 efried now, this is the version of the filter that runs on the compute host, not the scheduler, right?
11:10:36 stephenfin For Stein hosts, that's fine
11:10:37 sean-k-mooney efried: not its in the schduler
11:10:50 efried oh
11:10:51 stephenfin For Train hosts, the NUMATopologyFilter will then reject the host
11:10:55 sean-k-mooney stepehn modifled the host numa toplogy object
11:11:07 stephenfin and if that's disabled, it'll late fail on the compute node
11:11:08 efried ...so in a partially-upgraded scenario, how does the sched know the diff between stein & train hosts?
11:11:16 stephenfin NUMATopology.pcpuset
11:11:20 efried ack
11:11:22 efried it all comes together
11:11:26 stephenfin For Stein nodes, that isn't set
11:11:31 openstackgerrit Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452
11:11:43 donnyd sean-k-mooney: I made some adjustments yesterday and I am hoping that fixed the issue we are hainvg getting the numa job to run
11:11:56 alex_xu actually scheduler needn't know the diff of stein and trait
11:12:02 stephenfin For Train nodes with vcpu_pin_set, it's set to the same value as NUMATopology.cpuset
11:12:07 sean-k-mooney i only kicked off one run last night and it passed
11:12:18 stephenfin For Train nodes with cpu_dedicated_set, it's set to those cores
11:12:28 stephenfin For Train nodes with _only_ cpu_shared_set, it's set to None
11:12:28 sean-k-mooney donnyd: there is a new version of the numa migration scripts so i can kick it off again to test tehm
11:12:48 donnyd ok
11:13:10 sean-k-mooney alex_xu: we want the filter to know the difference so it can do the right thing in a mixed case
11:13:12 stephenfin We check to see if it's unset (meaning this is Stein compute node) and set it to the same value as NUMATopology.cpuset manually in the scheduler
11:13:21 donnyd lmk if there are more issues, I would like to get them ironed out so the label is actually usable
11:13:24 stephenfin However, if it's set to None, we do nothing, which means we'll fail to pin
11:13:35 openstackgerrit Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021
11:13:36 openstackgerrit Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595
11:13:49 stephenfin My functional tests prove that out. See here
11:14:23 alex_xu sean-k-mooney: the numa filter doesn't know. if stephenfin is going to update the patch, we will ensure we have value in host.pcpuset and host.cpuset. Then you asking pcpu, we get it from host.pcpuset, if you asking vcpu, then get it from host.cpuset
11:14:26 sean-k-mooney donnyd: sure. i vexhost and limestone might also be willing to provide the lables so we could maybe make this a voting job in the future
11:14:53 stephenfin https://review.opendev.org/#/c/681445/1/nova/tests/functional/libvirt/test_numa_servers.py@249
11:15:06 alex_xu ensure we have value in host.pcpuset and host.cpuset whatever it is stein node or train node.
11:15:14 stephenfin It's now the NUMATopologyFilter failing the request due to lack of PCPUs, not placement
11:15:55 stephenfin TIL 'git commit --amend --fixup=???' resets the author commit message (losing the change-ID). The more you know
11:17:39 efried stephenfin, sean-k-mooney: I re-commented. Good to move forward. Thanks for patiently explaining :)
11:18:06 sean-k-mooney we still need the numa toplogy filter for a release or two anyway since plament know nothing about numa yet so it will be enabled if you are using this feature anyway
11:18:27 efried yah
11:18:35 efried I was going to complain about that aspect, but yah.
11:19:29 sean-k-mooney once we fix placment of tis john snow syntrom re numa that is also proably the releas we want to drop this fallback and the numa toplogy filter
11:19:38 sean-k-mooney im guessing that will be V
11:19:44 openstackgerrit Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681383

Earlier   Later