| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 10:09:07 | bauzas | IIRC, we need to use a specific o.vo method | |
| 10:09:13 | bauzas | unless the field is set to None | |
| 10:09:14 | stephenfin | bauzas: You can do that, yeah | |
| 10:09:17 | stephenfin | For some time now | |
| 10:09:33 | stephenfin | you used to need to use is_attr_set or something like that | |
| 10:09:37 | bauzas | ok, I had concerns about the virt.hardware module in https://review.opendev.org/#/c/671800/ | |
| 10:09:44 | stephenfin | but __contains__ was implemented many moons ago | |
| 10:09:59 | stephenfin | anywhere is particular? | |
| 10:17:39 | bauzas | nah it's okay then | |
| 10:18:02 | bauzas | my o.vo skills are a bit rusty | |
| 10:18:07 | bauzas | but I could have tested it | |
| 10:19:04 | sean-k-mooney | efried: after 6 year of working on nova the main lesson i have learns is unlesss you have 2 cores from different compaines lined up to review the think you want to do before you start it will take 2 cycles to complete | |
| 10:19:39 | sean-k-mooney | the first cycle you get it working and people beging to become familar with and the second you finall land the thing | |
| 10:20:22 | efried | sean-k-mooney: I would love to be able to commit cores to review blueprints before/as they are approved. I just a) doubt cores would be willing to commit, and b) don't want to do that much PM type work | |
| 10:21:43 | sean-k-mooney | its partly a social thing. you need to explain why this thing you care about is imporant so that people will be interested enogh to review and demonstrate it works and is sane. | |
| 10:22:21 | sean-k-mooney | the non craze telco things i have worked on are less hard to enabel but im normlally in craze telco land | |
| 10:22:26 | efried | It sucks that "be interested" is a criterion | |
| 10:23:31 | sean-k-mooney | well it only is in terms of making it higher on list of other things you have to review | |
| 10:24:03 | sean-k-mooney | there are only a limit number of sme fore each system and they are overloaded frequently | |
| 10:25:14 | sean-k-mooney | if we had more subsystem maintainer or more reviewres it would help but we work with what we got. | |
| 10:25:31 | sean-k-mooney | we still get a lot done each cycle | |
| 10:25:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: DNM: Try to fallback to PCPU request when VCPU failed https://review.opendev.org/681383 | |
| 10:25:42 | sean-k-mooney | the latency can just be a bit high sometimes | |
| 10:25:56 | stephenfin | efried, bauzas: Can you sanity check before I waste more time fixing up tests ^ | |
| 10:26:06 | stephenfin | the approach in general, that is | |
| 10:26:42 | bauzas | stephenfin: I left a comment on https://review.opendev.org/#/c/671800/ | |
| 10:28:03 | bauzas | stephenfin: cool, I'll do it a bit later (needs to refuel my stomach) | |
| 10:32:46 | efried | stephenfin: sorry, looking at this patch for the first time, and... is this really the right approach? If we bounce with PCPUs, try VCPUs?? If they *have* cut over, and we're just *actually* out of PCPUs in the cloud, they're going to get *not* desired behavior, nah? | |
| 10:33:21 | openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | |
| 10:33:21 | openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResourceList objs https://review.opendev.org/678448 | |
| 10:33:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | |
| 10:33:22 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | |
| 11:04:37 | sean-k-mooney | efried: the numa toplogy filter will handel the out of PCPU case when they have swapped over to using cpu_dedicted_set | |
| 11:05:10 | efried | eh? | |
| 11:05:38 | sean-k-mooney | the numa toplogy filter will still run | |
| 11:06:03 | sean-k-mooney | it can tell the difference between a pre train host and a train host with non pCPUs free | |
| 11:06:04 | efried | If I'm out of PCPU resource per placement, this code will try the alloc request again with VCPUs. All of that happens before we get to the filter, yah? | |
| 11:06:18 | sean-k-mooney | yep | |
| 11:06:24 | efried | so... that's bad | |
| 11:06:47 | sean-k-mooney | and the filter will reject all host that are train but have not pcpus left when we are given vcpu allocations | |
| 11:07:05 | efried | that's not the scenario I'm concerned with. | |
| 11:07:15 | sean-k-mooney | what is? | |
| 11:07:51 | efried | Let's say I'm fully upgraded, with my confs properly cut over to cpu_{shared|dedicated}_set. | |
| 11:07:58 | sean-k-mooney | yep | |
| 11:08:01 | efried | this means my cloud has both PCPU and VCPU resources scattered around it. | |
| 11:08:05 | sean-k-mooney | yep | |
| 11:08:09 | efried | sometimes on the same host, maybe sometimes not, whatever. | |
| 11:08:16 | sean-k-mooney | sure | |
| 11:08:21 | efried | I request a dedicated VM. | |
| 11:08:35 | efried | But I'm genuinely out of PCPU resources in my cloud. | |
| 11:08:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fixup! Add support for translating CPU policy extra specs, image meta https://review.opendev.org/681445 | |
| 11:08:42 | sean-k-mooney | yep then we fall back | |
| 11:08:45 | efried | This code will bounce in GET /a_c and then *retry* with VCPUs instead. | |
| 11:08:53 | efried | So I'll wind up with my instance on VCPUs | |
| 11:08:58 | sean-k-mooney | no | |
| 11:09:00 | stephenfin | efried: nope | |
| 11:09:16 | stephenfin | scroll up and look at the conversation between alex_xu and I about two hours ago | |
| 11:09:20 | sean-k-mooney | you get allocation from placmenet that are for VCPUs | |
| 11:09:30 | sean-k-mooney | but then the numa toplogy filter will reject all the hosts | |
| 11:09:47 | sean-k-mooney | stephenfin: do you want to explain why | |
| 11:09:48 | efried | ...because the host knows the VCPUs aren't dedicated. | |
| 11:09:52 | efried | ? | |
| 11:10:02 | stephenfin | efried: Yeah | |
| 11:10:05 | efried | which it doesn't know pre-Train, but in Train it does | |
| 11:10:21 | stephenfin | Placement will try for PCPU, not get it, and then try for VCPU | |
| 11:10:23 | efried | now, this is the version of the filter that runs on the compute host, not the scheduler, right? | |
| 11:10:36 | stephenfin | For Stein hosts, that's fine | |
| 11:10:37 | sean-k-mooney | efried: not its in the schduler | |
| 11:10:50 | efried | oh | |
| 11:10:51 | stephenfin | For Train hosts, the NUMATopologyFilter will then reject the host | |
| 11:10:55 | sean-k-mooney | stepehn modifled the host numa toplogy object | |
| 11:11:07 | stephenfin | and if that's disabled, it'll late fail on the compute node | |
| 11:11:08 | efried | ...so in a partially-upgraded scenario, how does the sched know the diff between stein & train hosts? | |
| 11:11:16 | stephenfin | NUMATopology.pcpuset | |
| 11:11:20 | efried | ack | |
| 11:11:22 | efried | it all comes together | |
| 11:11:26 | stephenfin | For Stein nodes, that isn't set | |
| 11:11:31 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | |
| 11:11:43 | donnyd | sean-k-mooney: I made some adjustments yesterday and I am hoping that fixed the issue we are hainvg getting the numa job to run | |
| 11:11:56 | alex_xu | actually scheduler needn't know the diff of stein and trait | |
| 11:12:02 | stephenfin | For Train nodes with vcpu_pin_set, it's set to the same value as NUMATopology.cpuset | |
| 11:12:07 | sean-k-mooney | i only kicked off one run last night and it passed | |
| 11:12:18 | stephenfin | For Train nodes with cpu_dedicated_set, it's set to those cores | |
| 11:12:28 | sean-k-mooney | donnyd: there is a new version of the numa migration scripts so i can kick it off again to test tehm | |
| 11:12:28 | stephenfin | For Train nodes with _only_ cpu_shared_set, it's set to None | |
| 11:12:48 | donnyd | ok | |
| 11:13:10 | sean-k-mooney | alex_xu: we want the filter to know the difference so it can do the right thing in a mixed case | |
| 11:13:12 | stephenfin | We check to see if it's unset (meaning this is Stein compute node) and set it to the same value as NUMATopology.cpuset manually in the scheduler | |
| 11:13:21 | donnyd | lmk if there are more issues, I would like to get them ironed out so the label is actually usable | |
| 11:13:24 | stephenfin | However, if it's set to None, we do nothing, which means we'll fail to pin | |
| 11:13:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 11:13:36 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 11:13:49 | stephenfin | My functional tests prove that out. See here | |
| 11:14:23 | alex_xu | sean-k-mooney: the numa filter doesn't know. if stephenfin is going to update the patch, we will ensure we have value in host.pcpuset and host.cpuset. Then you asking pcpu, we get it from host.pcpuset, if you asking vcpu, then get it from host.cpuset | |
| 11:14:26 | sean-k-mooney | donnyd: sure. i vexhost and limestone might also be willing to provide the lables so we could maybe make this a voting job in the future | |
| 11:14:53 | stephenfin | https://review.opendev.org/#/c/681445/1/nova/tests/functional/libvirt/test_numa_servers.py@249 | |
| 11:15:06 | alex_xu | ensure we have value in host.pcpuset and host.cpuset whatever it is stein node or train node. | |
| 11:15:14 | stephenfin | It's now the NUMATopologyFilter failing the request due to lack of PCPUs, not placement | |
| 11:15:55 | stephenfin | TIL 'git commit --amend --fixup=???' resets the author commit message (losing the change-ID). The more you know | |
| 11:17:39 | efried | stephenfin, sean-k-mooney: I re-commented. Good to move forward. Thanks for patiently explaining :) | |