| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-20 | |||
| 16:03:38 | dansmith | efried: of course, I'm saying having "the plan" be a thousand places where seemingly fundamental operations can fail with the appropriate combination of obscure backend configs sucks from the user's perspective | |
| 16:03:40 | efried | anyway, in this case we're not talking API changes, and we're talking blocking at boot rather than (well, in addition to) shelve, so I'm good. | |
| 16:03:51 | efried | dansmith: totally agree. | |
| 16:03:56 | efried | block it if you know it's broken. | |
| 16:04:09 | dansmith | ...but try not to rely on that in design | |
| 16:04:27 | efried | now I just hope I can detect a swift | |
| 16:04:46 | efried | oh, | |
| 16:05:03 | efried | cause I wanted to add a check for "key manager configured" to the "do we enable vTPM in the first place" logic | |
| 16:05:11 | efried | but couldn't find a reliable way to inspect CONF for that | |
| 16:05:20 | dansmith | you need the catalog, no? | |
| 16:05:24 | efried | but yeah, | |
| 16:05:36 | efried | I can at least look in the catalog to make sure there's a keymgr endpoint. | |
| 16:05:40 | efried | um | |
| 16:05:59 | efried | except I'm doing this on init_host and don't have a context? | |
| 16:06:05 | dansmith | no, | |
| 16:06:10 | dansmith | you do this at boot time for the instance | |
| 16:06:17 | dansmith | non-tpm instances don't care about this | |
| 16:06:37 | dansmith | what you want to avoid is agreeing to create an instance for the user which in six weeks will be unsnapshottable | |
| 16:06:44 | efried | Sorry, I'm talking about something slightly different: the part where the libvirt driver decides whether to set the "can I vTPM?" flag. | |
| 16:06:53 | dansmith | I'm not talking about that | |
| 16:06:55 | efried | ...causing it to expose the traits | |
| 16:06:57 | efried | I know, I am. | |
| 16:07:02 | efried | I understand and agree with what you're talking about. | |
| 16:07:16 | dansmith | that too may be good, but expecting keystone is up and reachable when compute starts is not okay I think | |
| 16:07:33 | efried | orly? ... okay. | |
| 16:07:34 | dansmith | computes on the other side of a network partition shouldn't just fail to start I think | |
| 16:07:42 | efried | fair enough | |
| 16:07:54 | dansmith | they'll currently wait for conductor if it's not up yet, so... | |
| 16:08:25 | efried | yeah, I don't want to be the guy to introduce a network ping into init_host. | |
| 16:08:45 | dansmith | we have that :) | |
| 16:08:49 | dansmith | I added it | |
| 16:09:01 | efried | from compute driver or from compute manager? | |
| 16:09:04 | dansmith | ...but it's just a ping() rpc method compute makes to conductor to see if it's alive before it goes :) | |
| 16:09:08 | dansmith | manager | |
| 16:09:22 | efried | yeah, that's in manager. I'm talking about libvirt compute driver's init_host, which seems... dirtier. | |
| 16:09:37 | dansmith | yes, I'm just pointing out that I called it ping :) | |
| 16:10:42 | efried | okay, so lacking a jroll for live discussion, I'll mod the spec as if this is the plan, and ping for reviews. | |
| 16:10:45 | efried | Thanks dansmith | |
| 16:11:14 | dansmith | sounds good | |
| 16:13:27 | mnaser | sean-k-mooney: so my new map is 3C per NUMA node (6 threads total), ~32GB per NUMA node, with quite interesting latency numbers | |
| 16:13:37 | mnaser | https://www.irccloud.com/pastebin/91D5sGuS/ | |
| 16:13:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: zuul: Remove unnecessary 'tox_install_siblings' https://review.opendev.org/695235 | |
| 16:16:19 | efried | mriedem: the barbican-gating-on-py3 thing from yesterday seems to be busted https://review.opendev.org/#/c/695052/ -- would you mind having a quick look at some point? | |
| 16:17:51 | mnaser | w | |
| 16:22:13 | stephenfin | efried: didn't AJaeger have a fix for that? | |
| 16:22:26 | sean-k-mooney | mnaser: that kindof look like what i would expect | |
| 16:22:30 | stephenfin | efried: https://review.opendev.org/#/c/689461/ | |
| 16:22:32 | efried | stephenfin: yes, gmann said something in -barbican | |
| 16:22:40 | sean-k-mooney | as the numa distance increase teh latnecy does too | |
| 16:22:50 | stephenfin | ah, maybe not the same thing | |
| 16:22:52 | stephenfin | nvm me | |
| 16:23:04 | mnaser | sean-k-mooney: i also bumped the "4-link xGMI max speed" setting to 18Gbps .. but the machine is now crashing (it was set to 10.667 earlier) | |
| 16:23:09 | efried | stephenfin: yeah, he pointed to https://review.opendev.org/#/c/689458/ (same thing but in train) | |
| 16:23:14 | mnaser | so ill reset it back to that, and hopefully the latency stays at these numbers | |
| 16:23:34 | gmann | stephenfin: yeah, stable/train backport will fix the baribican. but still asking the reason to infra on ML for learning. | |
| 16:23:51 | stephenfin | sweet | |
| 16:24:10 | sean-k-mooney | but it lends more evidence to the idea we should look at cache affinity and numa distance when selecting vm numa node mappings | |
| 16:24:39 | mnaser | sean-k-mooney: yep, giving an instance two NUMA nodes that are '32' away vs '10' .. | |
| 16:24:54 | sean-k-mooney | mnaser: can you dump the numa distance info | |
| 16:24:56 | mnaser | or rather 11/12 | |
| 16:25:15 | sean-k-mooney | i would be gould to confimr the latency corralate with the numa distance reported | |
| 16:25:24 | mnaser | https://www.irccloud.com/pastebin/GwxGM24b/ | |
| 16:25:47 | mnaser | they're not that consistent but yeah | |
| 16:26:14 | sean-k-mooney | ya it does if we look noes 8-15 have the same numa distacne of 32 | |
| 16:26:32 | mnaser | which are all socket #2 | |
| 16:26:36 | sean-k-mooney | yep | |
| 16:27:18 | sean-k-mooney | we see 2-7 are all reported at numa distance of 12 and have almost the same latency | |
| 16:27:24 | gmann | we need some review from mriedem and tonyb here to get these backport in (pinged dave-mccowan barbican stable core also) - we need to get these merged as soon as we can - https://review.opendev.org/#/q/I24a46d0d7476203feccb1250d4ce3ad94b2e0ecd | |
| 16:27:38 | mnaser | sean-k-mooney: struggling what the 11/12 are there for | |
| 16:29:00 | sean-k-mooney | ya its not fully clear but we could use this info to optimise slightly | |
| 16:29:25 | sean-k-mooney | i suspect that is related to how the dies are grouped in pairs | |
| 16:29:35 | sean-k-mooney | distance of 11 is to the other ccx in the pari | |
| 16:29:49 | sean-k-mooney | 12 is the other ccx on the socket no in the pair | |
| 16:29:56 | sean-k-mooney | and 32 is cross socket in this case | |
| 16:32:09 | mnaser | sean-k-mooney: ok that makes sense then, i will try to reset the infinity fabric speed down to see if the machine stops crashing :\ | |
| 16:34:28 | Sundar | sean-k-mooney: Thanks a lot for reviewing the Cyborg spec despite your busy schedule. All the best. | |
| 16:40:15 | mriedem | gmann: the only one passing is stable/train | |
| 16:42:30 | efried | mriedem: it sounds like if that one merges it might fix master (I still don't understand how though) | |
| 16:42:59 | gmann | mriedem: yeah some unrelated failure. i did recheck on stein. after train cherry pick merged we can check if problem is sovled. otherwise need to get all in | |
| 16:43:30 | gmann | it all depends on how zuul pick the job definition due to those branche variant. | |
| 16:44:16 | gmann | we can try one by one. last option to avoid barbican gate is to make grenade job as py2 explicitly. | |
| 16:44:25 | gmann | *gate break | |
| 16:47:01 | mriedem | efried: because the job definitions are branch specific and grenade deploys the old side using stable/train and the new side using master | |
| 16:47:17 | mriedem | so sometimes to fix a thing on master for grenade you need to do something in master-1 | |
| 16:47:48 | mnaser | sean-k-mooney: ok making good progress, according to numastat it ended up scheduling to numa cell0 and cell1 which means this is the most "optimized" setup | |
| 16:48:22 | mnaser | topology is 8 sockets, i wonder if it would be better to model the topology as "2 sockets, 4 cores, 1 threads" or "2 sockets, 2 cores, 2 threads" too | |
| 16:56:19 | efried | dansmith: are there any operations other than shelve for which the swift blob would be appropriate? | |
| 16:57:15 | dansmith | well, what are you going to do about snapshot? | |
| 16:57:23 | dansmith | snapshot gets a fresh tpm? | |
| 16:57:34 | dansmith | that doesn't work well for people that use snapshot as a backup mechanism | |
| 16:57:42 | dansmith | speaking of that, the backup api :) | |
| 16:58:48 | mnaser | burn it with fire | |
| 16:58:54 | efried | dansmith: gr, config option? save_tpm_with_snapshot_note_does_not_apply_to_shelve_where_we_always_do_that_anyway = $bool | |
| 16:59:07 | dansmith | idk | |
| 16:59:18 | efried | open question I guess, sigh | |
| 16:59:19 | dansmith | I mean, not a config option, no | |
| 16:59:45 | dansmith | but not sure what the answer is | |
| 16:59:53 | sean-k-mooney | mnaser: that is becasue we use itertool.permuations so we will initally iterate over the numa nodes in order trying to fit the vitual numa node to the host | |
| 16:59:54 | efried | I think the answer has to be no | |
| 17:00:05 | efried | because we have no way to associate in that scenario | |