Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-20
16:22:30 stephenfin efried: https://review.opendev.org/#/c/689461/
16:22:32 efried stephenfin: yes, gmann said something in -barbican
16:22:40 sean-k-mooney as the numa distance increase teh latnecy does too
16:22:50 stephenfin ah, maybe not the same thing
16:22:52 stephenfin nvm me
16:23:04 mnaser sean-k-mooney: i also bumped the "4-link xGMI max speed" setting to 18Gbps .. but the machine is now crashing (it was set to 10.667 earlier)
16:23:09 efried stephenfin: yeah, he pointed to https://review.opendev.org/#/c/689458/ (same thing but in train)
16:23:14 mnaser so ill reset it back to that, and hopefully the latency stays at these numbers
16:23:34 gmann stephenfin: yeah, stable/train backport will fix the baribican. but still asking the reason to infra on ML for learning.
16:23:51 stephenfin sweet
16:24:10 sean-k-mooney but it lends more evidence to the idea we should look at cache affinity and numa distance when selecting vm numa node mappings
16:24:39 mnaser sean-k-mooney: yep, giving an instance two NUMA nodes that are '32' away vs '10' ..
16:24:54 sean-k-mooney mnaser: can you dump the numa distance info
16:24:56 mnaser or rather 11/12
16:25:15 sean-k-mooney i would be gould to confimr the latency corralate with the numa distance reported
16:25:24 mnaser https://www.irccloud.com/pastebin/GwxGM24b/
16:25:47 mnaser they're not that consistent but yeah
16:26:14 sean-k-mooney ya it does if we look noes 8-15 have the same numa distacne of 32
16:26:32 mnaser which are all socket #2
16:26:36 sean-k-mooney yep
16:27:18 sean-k-mooney we see 2-7 are all reported at numa distance of 12 and have almost the same latency
16:27:24 gmann we need some review from mriedem and tonyb here to get these backport in (pinged dave-mccowan barbican stable core also) - we need to get these merged as soon as we can - https://review.opendev.org/#/q/I24a46d0d7476203feccb1250d4ce3ad94b2e0ecd
16:27:38 mnaser sean-k-mooney: struggling what the 11/12 are there for
16:29:00 sean-k-mooney ya its not fully clear but we could use this info to optimise slightly
16:29:25 sean-k-mooney i suspect that is related to how the dies are grouped in pairs
16:29:35 sean-k-mooney distance of 11 is to the other ccx in the pari
16:29:49 sean-k-mooney 12 is the other ccx on the socket no in the pair
16:29:56 sean-k-mooney and 32 is cross socket in this case
16:32:09 mnaser sean-k-mooney: ok that makes sense then, i will try to reset the infinity fabric speed down to see if the machine stops crashing :\
16:34:28 Sundar sean-k-mooney: Thanks a lot for reviewing the Cyborg spec despite your busy schedule. All the best.
16:40:15 mriedem gmann: the only one passing is stable/train
16:42:30 efried mriedem: it sounds like if that one merges it might fix master (I still don't understand how though)
16:42:59 gmann mriedem: yeah some unrelated failure. i did recheck on stein. after train cherry pick merged we can check if problem is sovled. otherwise need to get all in
16:43:30 gmann it all depends on how zuul pick the job definition due to those branche variant.
16:44:16 gmann we can try one by one. last option to avoid barbican gate is to make grenade job as py2 explicitly.
16:44:25 gmann *gate break
16:47:01 mriedem efried: because the job definitions are branch specific and grenade deploys the old side using stable/train and the new side using master
16:47:17 mriedem so sometimes to fix a thing on master for grenade you need to do something in master-1
16:47:48 mnaser sean-k-mooney: ok making good progress, according to numastat it ended up scheduling to numa cell0 and cell1 which means this is the most "optimized" setup
16:48:22 mnaser topology is 8 sockets, i wonder if it would be better to model the topology as "2 sockets, 4 cores, 1 threads" or "2 sockets, 2 cores, 2 threads" too
16:56:19 efried dansmith: are there any operations other than shelve for which the swift blob would be appropriate?
16:57:15 dansmith well, what are you going to do about snapshot?
16:57:23 dansmith snapshot gets a fresh tpm?
16:57:34 dansmith that doesn't work well for people that use snapshot as a backup mechanism
16:57:42 dansmith speaking of that, the backup api :)
16:58:48 mnaser burn it with fire
16:58:54 efried dansmith: gr, config option? save_tpm_with_snapshot_note_does_not_apply_to_shelve_where_we_always_do_that_anyway = $bool
16:59:07 dansmith idk
16:59:18 efried open question I guess, sigh
16:59:19 dansmith I mean, not a config option, no
16:59:45 dansmith but not sure what the answer is
16:59:53 sean-k-mooney mnaser: that is becasue we use itertool.permuations so we will initally iterate over the numa nodes in order trying to fit the vitual numa node to the host
16:59:54 efried I think the answer has to be no
17:00:05 efried because we have no way to associate in that scenario
17:00:12 dansmith if I snapshot my instance and plan to roll ack if my OS upgrade fails, I'm going to be super surprised that I can't read my encrypted data volume (ever again)
17:00:15 efried because we're relying on the instance sysmeta to associate for shelve.
17:00:18 sean-k-mooney mnaser: so in that specific case it will work
17:00:19 mnaser sean-k-mooney: so maybe eventually machines might get some inefficent assignments
17:00:24 sean-k-mooney mnaser: but in general it wont
17:00:30 sean-k-mooney mnaser: yes
17:00:40 mnaser so its possible i hit node4+node5 which is a bad time
17:00:50 dansmith efried: I think it'd be good to get some jroll input on this
17:00:57 efried agree
17:01:01 sean-k-mooney if node 0,2,3 could fit it it would select 0 and 2
17:01:06 sean-k-mooney and never check 3
17:01:09 dansmith efried: but I think people will be pretty surprised if it's gone
17:01:15 mnaser sean-k-mooney: so far im seeing some numbers increasing by 25% -- and ok i see
17:01:42 efried dansmith: but then we go back to needing the association in the instance meta, because there's nothing else
17:02:20 sean-k-mooney mnaser: that is why i said we liekly could optimise this using the numa distance but its more complex then what we do today
17:02:29 dansmith efried: image meta you mean? I'm not saying we should, I'm just poking holes in it which.. if people lose data they get mad
17:02:30 sean-k-mooney mnaser: and it has implciations for modeling in plamcent
17:02:33 dansmith efried: I'm on a call right now though
17:02:36 efried sorry, yeah, image meta.
17:02:37 mnaser sean-k-mooney: yes that sounds massively complex to get implemented -- but would be so awesome
17:03:12 sean-k-mooney mnaser: what feature that relates to numa isint :)
17:03:18 mnaser hah, good one
17:03:30 mnaser sean-k-mooney: i mean in my case, the fact memory doesnt float across 8/16 numa nodes, at least 2 is better, the only "bad" potential idea is if a VM get pinned to two NUMA nodes that are pretty far apart
17:03:58 sean-k-mooney right as in on different sockets.
17:04:07 mnaser yep
17:04:17 sean-k-mooney but even then its still better then floating as it will be deterministic
17:04:53 sean-k-mooney well in that once the instnace i booted its performance should not change
17:17:50 openstackgerrit Merged openstack/nova stable/train: Imported Translations from Zanata https://review.opendev.org/694911
17:32:45 openstackgerrit Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
17:34:27 KeithMnemonic any change to please get some reviews on the "marker" patches mriedem was working on ? https://review.opendev.org/#/c/690721/4 Thanks!
17:36:59 dustinc efried, gibi, dansmith: RE Provider Config: Just got caught up on this morning's convo. Am I right in seeing that you guys want to A) drop NAME as identification method, and B) Allow both $UUID and $COMPUTE_NODE to identify same RP but $UUID takes precedence? If so I am 100% on board.
18:06:45 dansmith dustinc: I dunno about a conflict between a fixed uuid and $compute_node, but I would recommend handling that the same way I suggested for the name/uuid conflict, which is to log error and ignore both
18:33:04 openstackgerrit Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825
18:33:04 openstackgerrit Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
18:49:19 sean-k-mooney by the way i just looked at the implemented spec folder for train. looks like we have not moved them https://github.com/openstack/nova-specs/tree/master/specs/train/implemented
19:09:08 mriedem sean-k-mooney: you can run this and post the results https://github.com/openstack/nova-specs/blob/master/tox.ini#L39
19:14:48 sean-k-mooney sure just leaving to grab dinner but ill give it a try when i get back
19:15:49 sean-k-mooney ok so that checks with launchpad to determin if they were finsihed and then updates them
19:16:03 sean-k-mooney cool
19:54:26 openstackgerrit Mark Goddard proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/695187
19:54:27 openstackgerrit Mark Goddard proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/695188
19:54:27 openstackgerrit Mark Goddard proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/694802
19:54:28 openstackgerrit Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189
20:38:41 efried dustinc: The way I suggested yesterday still works IMO, and it requires less churn in both design and already-written code.
20:40:13 efried With dansmith's addendum
20:41:29 efried Which is as follows:

Earlier   Later