| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-09 | |||
| 15:52:44 | cfriesen | mnaser: danpb: dansmith: this looks sort of related, but the fix should be in newton: https://bugs.launchpad.net/nova/+bug/1578155 | |
| 15:52:45 | openstack | Launchpad bug 1578155 in OpenStack Compute (nova) newton "'hw:cpu_thread_policy=prefer' misbehaviour" [Medium,Fix committed] - Assigned to Stephen Finucane (stephenfinucane) | |
| 15:52:57 | dansmith | I really don't know anything about this stuff | |
| 15:54:53 | jaypipes | mriedem, dansmith: I will be afk for 2 hours, just FYI. | |
| 15:54:55 | cfriesen | mnaser: so are 16/17 the siblings of 0/1 which are not included in vcpu_pin_set? | |
| 15:55:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [placement] Add api-ref for usages https://review.openstack.org/480563 | |
| 15:55:11 | artom | About the fix, at any rate. | |
| 15:55:13 | mnaser | cfriesen nope, 0/1 are the ones which are not included | |
| 15:55:26 | mnaser | so our vcpu_pin_set is 2-31 | |
| 15:55:30 | mnaser | (32 core machine) | |
| 15:55:33 | cfriesen | mnaser: yes, but which are the HT siblings of 0/1? | |
| 15:56:15 | danpb | hmm, yes, it seems numa node 0 has even numbered cpus, and node 1 has odd numbered cpus | |
| 15:56:19 | mnaser | cfriesen sorry, not sure i follow, but here's lscpu output if that helps? http://paste.openstack.org/show/617955/ | |
| 15:56:34 | danpb | so excluding 0-1, kills 1 cpu each host numa node, instead of 2 cpus | |
| 15:56:35 | cfriesen | mnaser: based on the pattern above, the HT siblings of 0/1 would be 16/17....I'm wondering if that's confusing the logic in hardware.py | |
| 15:56:42 | mnaser | but i think that they are | |
| 15:57:19 | cfriesen | mnaswer: running "virsh capabilities" will show sibling info | |
| 15:57:26 | danpb | so that in turn kills a entire sibling pair from each numa node | |
| 15:57:37 | danpb | effectively making 4 cpus unavailable | |
| 15:57:53 | cfriesen | danpb: theoretically it shouldn't kill the sibling pair though...we should have two pCPUs with no siblings | |
| 15:58:03 | mnaser | here is output of virsh caps. http://paste.openstack.org/show/617956/ | |
| 15:58:06 | cfriesen | but maybe the logic can't handle a mix of sibs and no sibs | |
| 15:58:15 | mriedem | someone want to approve this? https://review.openstack.org/#/c/491855/ | |
| 15:58:18 | danpb | yeah that wouldn't be surprising | |
| 15:58:46 | mnaser | so am i better off reserving 2 siblings and maybe that might get rid of the 'confusion' ? | |
| 15:58:53 | cfriesen | mnaswer: seems likely | |
| 15:58:55 | mnaser | so instead of 0,1 .. do 0,16 instead? | |
| 15:59:02 | danpb | yeah | |
| 15:59:53 | mnaser | let me test that theory out | |
| 16:00:03 | openstackgerrit | Ed Leafe proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954 | |
| 16:00:12 | edleafe | dansmith: dtantsur: ^^ updated to remove the node RC change handling | |
| 16:00:37 | dtantsur | awesome! I'm just out of a meeting, will do the ironic part now | |
| 16:00:50 | mnaser | so my vcpu_pin_set should technically be 1-15,17-31 in that case, correct? | |
| 16:00:53 | mriedem | gibi: wasn't that also a problem in ocata then? CoreFilter isn't enabled by default | |
| 16:00:53 | mnaser | leaving 0,16 for the OS | |
| 16:00:59 | cfriesen | mnaswer: either way, please raise a bug for this issue, it'd be nice to handle it properly | |
| 16:01:02 | danpb | mnaser: yeah | |
| 16:01:11 | mnaser | cfriesen agreed | |
| 16:01:15 | mnaser | ok, lets try this out | |
| 16:01:35 | mnaser | ill reboot with the new updates isolcpus= values so itll take me a few minutes and report | |
| 16:02:31 | dansmith | mriedem: melwitt: I'm going to disappear suddenly somewhere in the hour of the cells meeting.. assume we were on track to punt anyway, but.. that okay? | |
| 16:03:57 | dansmith | jaypipes: so what's the deal with that last patch? do you want me to update it and remove the extraneous continue? or log something there or what? | |
| 16:04:15 | jaypipes | please, yes, go for it. | |
| 16:04:32 | dansmith | ...which? | |
| 16:04:34 | gibi | mriedem: you are right CoreFilter is not enabled by default in Ocata. I will quickly backport the test locally to Ocata to confirm... | |
| 16:04:36 | jaypipes | dansmith: ^ sorry, I've been in meetings and dealing with russian visa mess all morning and now have a doctor's appt. | |
| 16:04:51 | jaypipes | dansmith: remove the extraneous continue thign | |
| 16:04:55 | dansmith | okay | |
| 16:06:03 | mriedem | dansmith: yeah planned on skipping cells v2 meeting today | |
| 16:06:11 | dansmith | mriedem: okay | |
| 16:06:32 | mriedem | gibi: the fix in ocata would have to be different probably since this code all got refactored in pike | |
| 16:06:42 | mriedem | gibi: but just wanted to make sure it's not an rc1 regression blocker thing | |
| 16:08:18 | dansmith | mriedem: I actually think we probably should enable the cache.. I thought it was already done on the computes, but it's not. The other use on compute is to calculate the rpc pin, which is cached until restart anyway | |
| 16:09:14 | mnaser | danpb cfriesen -- i'm now getting "Not enough available CPUs to schedule instance. Oversubscription is not possible with pinned instances. Required: 1, actual: 0" | |
| 16:09:28 | mnaser | HOST_CELL: NUMACell(cpu_usage=14,cpuset=set([2,4,6,8,10,12,14,18,20,22,24,26,28,30]),id=0,memory=196562,memory_usage=57344,mempages=[NUMAPagesTopology,NUMAPagesTopology],pinned_cpus=set([2,4,6,8,10,12,14,18,20,22,24,26,28,30]),siblings=[set([8,24]),set([2,18]),set([10,26]),set([12,28]),set([6,22]),set([14,30]),set([4,20])]) | |
| 16:11:34 | danpb | where's your other numa cell | |
| 16:11:44 | danpb | that just shows the first cell | |
| 16:12:16 | mnaser | i dont know why its not listed. i added LOG.debug("HOST_CELL: %s" % host_cell) to _numa_fit_instance_cell_with_pinning | |
| 16:12:40 | cfriesen | mnaser: I think I know what's going on...you now have fewer pCPUs in one node than the other, and you're asking for 2-node guests | |
| 16:12:42 | mnaser | let me get you | |
| 16:12:43 | mnaser | both host cell output | |
| 16:12:57 | mnaser | oh | |
| 16:13:01 | mnaser | i think you're right | |
| 16:13:11 | danpb | cfriesen: yep makes sense | |
| 16:13:47 | mnaser | let me switch things back to how they were and get the output of both host cells | |
| 16:14:02 | danpb | mnaser: why do you want the guests to have multiple virtual numa cells ? | |
| 16:14:46 | cfriesen | mriedem: for a reno for https://review.openstack.org/#/c/491854/ would we want to describe the removal of the two default filters in "features", "upgrade", or "other"? | |
| 16:14:52 | danpb | its generally not something you'd do unless guest memory exceeds the amount available in a single host, or need to consume say, PCI devices from separate nods at the same time | |
| 16:15:08 | mnaser | danpb: still experimenting but the idea was more efficent use of hardware. i have 120x 1gb large pages which means that i'll end up with 60gb on each numanode, if i put 8gb sized instances only, i'll end up with 4096mb in each numa node that's unused | |
| 16:15:13 | cfriesen | danpb: or you want increased memory bandwidth | |
| 16:15:54 | danpb | cfriesen: that's only increased if your guest workload avoids cross-node traffic | |
| 16:16:01 | cfriesen | danpb: agreed | |
| 16:16:01 | mnaser | by doing this, it splits 4096 into each numanode and i can fill the freepages .. but it's not something that's set in stone fully | |
| 16:16:11 | danpb | cfriesen: otherwise you'd actually decrease throughput | |
| 16:16:22 | cfriesen | mnaser: the downside is that your guests need to be able to avoid cross-numa traffic, which makes guest coding trickier | |
| 16:16:24 | danpb | by having the guest all contend on the cross-node memory bus | |
| 16:16:48 | cfriesen | danpb: yes, it'd be a special-case | |
| 16:17:00 | mnaser | yeah i was thinking the linux kernel already does an ok job handling multiple numa nodes | |
| 16:17:01 | danpb | IOW unless your guest app is intelligent you want to avoid multiple numa nods | |
| 16:17:10 | mnaser | i see | |
| 16:17:28 | cfriesen | mnaser: the kernel does, but not all userspace code does. | |
| 16:17:38 | mriedem | dansmith: ack - gonna be afk for about an hour | |
| 16:17:40 | mnaser | also the only other tradeoff that comes with this is that the threads are not shared which is not ideal | |
| 16:17:46 | mnaser | so that was something i didnt like about doing this | |
| 16:17:49 | mriedem | can't be worst dad of the summer 2 days in a row | |
| 16:17:57 | cfriesen | mnaser: so if you've got a single large app that wants most of that 8GB... | |
| 16:18:16 | dansmith | mriedem: okay I'll push this up with those changes and I'll be gone before jenkins gets to it | |
| 16:18:23 | mnaser | for completition sake btw, this is both host cells | |
| 16:18:24 | dansmith | mriedem: jaypipes so baton back to you at that point | |
| 16:18:45 | mnaser | http://paste.openstack.org/show/617957/ | |
| 16:19:26 | mnaser | there are 15 sets there | |
| 16:19:37 | mnaser | let me see what it was when we had 0-1 only | |
| 16:20:17 | cdent | dansmith: couple questions about expected discover_hosts behavior: is it supposed to be idempotent (run it again and again, it’s okay)? It is supposed to cope if two different processes run it at the same time? | |
| 16:20:24 | cfriesen | mnaser: but only 14 pairs of pCPUs from different numa nodes | |
| 16:20:43 | mnaser | yeah, thats why it failed now, but in the original case | |
| 16:20:44 | mnaser | strangely enough | |
| 16:20:51 | mnaser | i only see one output of numacell, not two | |
| 16:20:51 | dansmith | cdent: yeah, it's expected to just run it over and over again, from cron even | |
| 16:21:15 | dansmith | cdent: you might get a failure if you run multiples and one loses the race to insert the record, but other than that it should be fine to run multiple threads of it | |
| 16:21:41 | cdent | dansmith: thanks that’s what I was hoping/expecting but wanted to confirm | |