| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-20 | |||
| 16:08:45 | dansmith | we have that :) | |
| 16:08:49 | dansmith | I added it | |
| 16:09:01 | efried | from compute driver or from compute manager? | |
| 16:09:04 | dansmith | ...but it's just a ping() rpc method compute makes to conductor to see if it's alive before it goes :) | |
| 16:09:08 | dansmith | manager | |
| 16:09:22 | efried | yeah, that's in manager. I'm talking about libvirt compute driver's init_host, which seems... dirtier. | |
| 16:09:37 | dansmith | yes, I'm just pointing out that I called it ping :) | |
| 16:10:42 | efried | okay, so lacking a jroll for live discussion, I'll mod the spec as if this is the plan, and ping for reviews. | |
| 16:10:45 | efried | Thanks dansmith | |
| 16:11:14 | dansmith | sounds good | |
| 16:13:27 | mnaser | sean-k-mooney: so my new map is 3C per NUMA node (6 threads total), ~32GB per NUMA node, with quite interesting latency numbers | |
| 16:13:37 | mnaser | https://www.irccloud.com/pastebin/91D5sGuS/ | |
| 16:13:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: zuul: Remove unnecessary 'tox_install_siblings' https://review.opendev.org/695235 | |
| 16:16:19 | efried | mriedem: the barbican-gating-on-py3 thing from yesterday seems to be busted https://review.opendev.org/#/c/695052/ -- would you mind having a quick look at some point? | |
| 16:17:51 | mnaser | w | |
| 16:22:13 | stephenfin | efried: didn't AJaeger have a fix for that? | |
| 16:22:26 | sean-k-mooney | mnaser: that kindof look like what i would expect | |
| 16:22:30 | stephenfin | efried: https://review.opendev.org/#/c/689461/ | |
| 16:22:32 | efried | stephenfin: yes, gmann said something in -barbican | |
| 16:22:40 | sean-k-mooney | as the numa distance increase teh latnecy does too | |
| 16:22:50 | stephenfin | ah, maybe not the same thing | |
| 16:22:52 | stephenfin | nvm me | |
| 16:23:04 | mnaser | sean-k-mooney: i also bumped the "4-link xGMI max speed" setting to 18Gbps .. but the machine is now crashing (it was set to 10.667 earlier) | |
| 16:23:09 | efried | stephenfin: yeah, he pointed to https://review.opendev.org/#/c/689458/ (same thing but in train) | |
| 16:23:14 | mnaser | so ill reset it back to that, and hopefully the latency stays at these numbers | |
| 16:23:34 | gmann | stephenfin: yeah, stable/train backport will fix the baribican. but still asking the reason to infra on ML for learning. | |
| 16:23:51 | stephenfin | sweet | |
| 16:24:10 | sean-k-mooney | but it lends more evidence to the idea we should look at cache affinity and numa distance when selecting vm numa node mappings | |
| 16:24:39 | mnaser | sean-k-mooney: yep, giving an instance two NUMA nodes that are '32' away vs '10' .. | |
| 16:24:54 | sean-k-mooney | mnaser: can you dump the numa distance info | |
| 16:24:56 | mnaser | or rather 11/12 | |
| 16:25:15 | sean-k-mooney | i would be gould to confimr the latency corralate with the numa distance reported | |
| 16:25:24 | mnaser | https://www.irccloud.com/pastebin/GwxGM24b/ | |
| 16:25:47 | mnaser | they're not that consistent but yeah | |
| 16:26:14 | sean-k-mooney | ya it does if we look noes 8-15 have the same numa distacne of 32 | |
| 16:26:32 | mnaser | which are all socket #2 | |
| 16:26:36 | sean-k-mooney | yep | |
| 16:27:18 | sean-k-mooney | we see 2-7 are all reported at numa distance of 12 and have almost the same latency | |
| 16:27:24 | gmann | we need some review from mriedem and tonyb here to get these backport in (pinged dave-mccowan barbican stable core also) - we need to get these merged as soon as we can - https://review.opendev.org/#/q/I24a46d0d7476203feccb1250d4ce3ad94b2e0ecd | |
| 16:27:38 | mnaser | sean-k-mooney: struggling what the 11/12 are there for | |
| 16:29:00 | sean-k-mooney | ya its not fully clear but we could use this info to optimise slightly | |
| 16:29:25 | sean-k-mooney | i suspect that is related to how the dies are grouped in pairs | |
| 16:29:35 | sean-k-mooney | distance of 11 is to the other ccx in the pari | |
| 16:29:49 | sean-k-mooney | 12 is the other ccx on the socket no in the pair | |
| 16:29:56 | sean-k-mooney | and 32 is cross socket in this case | |
| 16:32:09 | mnaser | sean-k-mooney: ok that makes sense then, i will try to reset the infinity fabric speed down to see if the machine stops crashing :\ | |
| 16:34:28 | Sundar | sean-k-mooney: Thanks a lot for reviewing the Cyborg spec despite your busy schedule. All the best. | |
| 16:40:15 | mriedem | gmann: the only one passing is stable/train | |
| 16:42:30 | efried | mriedem: it sounds like if that one merges it might fix master (I still don't understand how though) | |
| 16:42:59 | gmann | mriedem: yeah some unrelated failure. i did recheck on stein. after train cherry pick merged we can check if problem is sovled. otherwise need to get all in | |
| 16:43:30 | gmann | it all depends on how zuul pick the job definition due to those branche variant. | |
| 16:44:16 | gmann | we can try one by one. last option to avoid barbican gate is to make grenade job as py2 explicitly. | |
| 16:44:25 | gmann | *gate break | |
| 16:47:01 | mriedem | efried: because the job definitions are branch specific and grenade deploys the old side using stable/train and the new side using master | |
| 16:47:17 | mriedem | so sometimes to fix a thing on master for grenade you need to do something in master-1 | |
| 16:47:48 | mnaser | sean-k-mooney: ok making good progress, according to numastat it ended up scheduling to numa cell0 and cell1 which means this is the most "optimized" setup | |
| 16:48:22 | mnaser | topology is 8 sockets, i wonder if it would be better to model the topology as "2 sockets, 4 cores, 1 threads" or "2 sockets, 2 cores, 2 threads" too | |
| 16:56:19 | efried | dansmith: are there any operations other than shelve for which the swift blob would be appropriate? | |
| 16:57:15 | dansmith | well, what are you going to do about snapshot? | |
| 16:57:23 | dansmith | snapshot gets a fresh tpm? | |
| 16:57:34 | dansmith | that doesn't work well for people that use snapshot as a backup mechanism | |
| 16:57:42 | dansmith | speaking of that, the backup api :) | |
| 16:58:48 | mnaser | burn it with fire | |
| 16:58:54 | efried | dansmith: gr, config option? save_tpm_with_snapshot_note_does_not_apply_to_shelve_where_we_always_do_that_anyway = $bool | |
| 16:59:07 | dansmith | idk | |
| 16:59:18 | efried | open question I guess, sigh | |
| 16:59:19 | dansmith | I mean, not a config option, no | |
| 16:59:45 | dansmith | but not sure what the answer is | |
| 16:59:53 | sean-k-mooney | mnaser: that is becasue we use itertool.permuations so we will initally iterate over the numa nodes in order trying to fit the vitual numa node to the host | |
| 16:59:54 | efried | I think the answer has to be no | |
| 17:00:05 | efried | because we have no way to associate in that scenario | |
| 17:00:12 | dansmith | if I snapshot my instance and plan to roll ack if my OS upgrade fails, I'm going to be super surprised that I can't read my encrypted data volume (ever again) | |
| 17:00:15 | efried | because we're relying on the instance sysmeta to associate for shelve. | |
| 17:00:18 | sean-k-mooney | mnaser: so in that specific case it will work | |
| 17:00:19 | mnaser | sean-k-mooney: so maybe eventually machines might get some inefficent assignments | |
| 17:00:24 | sean-k-mooney | mnaser: but in general it wont | |
| 17:00:30 | sean-k-mooney | mnaser: yes | |
| 17:00:40 | mnaser | so its possible i hit node4+node5 which is a bad time | |
| 17:00:50 | dansmith | efried: I think it'd be good to get some jroll input on this | |
| 17:00:57 | efried | agree | |
| 17:01:01 | sean-k-mooney | if node 0,2,3 could fit it it would select 0 and 2 | |
| 17:01:06 | sean-k-mooney | and never check 3 | |
| 17:01:09 | dansmith | efried: but I think people will be pretty surprised if it's gone | |
| 17:01:15 | mnaser | sean-k-mooney: so far im seeing some numbers increasing by 25% -- and ok i see | |
| 17:01:42 | efried | dansmith: but then we go back to needing the association in the instance meta, because there's nothing else | |
| 17:02:20 | sean-k-mooney | mnaser: that is why i said we liekly could optimise this using the numa distance but its more complex then what we do today | |
| 17:02:29 | dansmith | efried: image meta you mean? I'm not saying we should, I'm just poking holes in it which.. if people lose data they get mad | |
| 17:02:30 | sean-k-mooney | mnaser: and it has implciations for modeling in plamcent | |
| 17:02:33 | dansmith | efried: I'm on a call right now though | |
| 17:02:36 | efried | sorry, yeah, image meta. | |
| 17:02:37 | mnaser | sean-k-mooney: yes that sounds massively complex to get implemented -- but would be so awesome | |
| 17:03:12 | sean-k-mooney | mnaser: what feature that relates to numa isint :) | |
| 17:03:18 | mnaser | hah, good one | |
| 17:03:30 | mnaser | sean-k-mooney: i mean in my case, the fact memory doesnt float across 8/16 numa nodes, at least 2 is better, the only "bad" potential idea is if a VM get pinned to two NUMA nodes that are pretty far apart | |
| 17:03:58 | sean-k-mooney | right as in on different sockets. | |
| 17:04:07 | mnaser | yep | |
| 17:04:17 | sean-k-mooney | but even then its still better then floating as it will be deterministic | |
| 17:04:53 | sean-k-mooney | well in that once the instnace i booted its performance should not change | |
| 17:17:50 | openstackgerrit | Merged openstack/nova stable/train: Imported Translations from Zanata https://review.opendev.org/694911 | |
| 17:32:45 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072 | |