| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 17:14:41 | dansmith | efried: uh what? | |
| 17:14:43 | efried | sean-k-mooney: yeah, so what I'm saying is, having daxio -z'd the thing, you would need specialized hardware to try to uncover the ghost data | |
| 17:14:44 | mriedem | artom: i've piled on | |
| 17:14:56 | efried | dansmith: what what? | |
| 17:15:05 | sean-k-mooney | efried: you would be surpriesed | |
| 17:15:09 | dansmith | efried: unless you scrub every byte in what you hand to the guest, it can find it | |
| 17:15:24 | efried | scrub meaning overwrite multiple times? | |
| 17:15:28 | sean-k-mooney | efried: yes | |
| 17:15:53 | efried | and... is that not a problem for regular ol ram and disk too? | |
| 17:15:59 | sean-k-mooney | at least with magnetic media there was a bias that a singel over write was not enough | |
| 17:16:13 | efried | do we srsly shred every byte of those between VMs? | |
| 17:16:16 | dansmith | efried: certainly not for ram because it's paged in | |
| 17:16:16 | sean-k-mooney | ram is not persetend so no power no data | |
| 17:16:38 | dansmith | efried: the kernel can zero a page before it lets you read from it if you haven't written to it | |
| 17:17:02 | dansmith | but with media that's not the case | |
| 17:17:20 | dansmith | I dunno what the semantics of the daxio thing are, but in general you have to be careful about that stuff | |
| 17:17:37 | efried | -z, --zero Zero the output device for len size, or the entire device if no length was provided. The output device must be a Device DAX device. | |
| 17:17:47 | dansmith | and there have been exploits where even dropping the map could be circumvented by tricking the thing into granting you access to a region again without it being zeroed | |
| 17:18:03 | efried | I mean, it doesn't say "Really really zero the output device" | |
| 17:18:09 | efried | but still | |
| 17:18:15 | efried | you would think it would be set up to prevent contamination | |
| 17:18:19 | dansmith | efried: and for silicon, it almost never does because it costs money (i.e. wears the media) | |
| 17:18:20 | efried | otherwise what good is it? | |
| 17:18:51 | sean-k-mooney | efried: anyway were are side tracking | |
| 17:18:58 | efried | agreed | |
| 17:19:01 | dansmith | efried: you said "regular deleted data" above, which is what I took exception to | |
| 17:19:30 | efried | ack | |
| 17:19:38 | sean-k-mooney | right its specal hadware that needs special handeling | |
| 17:19:39 | dansmith | but I will say, all the spectre stuff is about sussing out data via sidechannel which is not reading it direclty | |
| 17:19:44 | dansmith | so let's not pretend that's not a thing :) | |
| 17:19:52 | efried | Yeah, of course if you just trash the inode and don't actually zero the content | |
| 17:20:29 | efried | I'm talking about: if you overwrite with zeros, but just once, don't you need something special (that VMs don't have) to read the ghost data? | |
| 17:20:46 | dansmith | not necessarily | |
| 17:20:53 | efried | okay | |
| 17:20:55 | dansmith | you're thinking about low-level hardware detection of things, | |
| 17:21:04 | dansmith | but that's not the only way to get access to that "ghost" data | |
| 17:21:17 | dansmith | that's the whole point of sidechannel attacks | |
| 17:22:02 | alex_xu | efried: do we have something to ignore a host for placement allocation_candidates call now? | |
| 17:22:04 | efried | alex_xu, sean-k-mooney: looks like the CI fixups have merged now https://github.com/intel/Intel-OpenStack-CI-jobs/pull/2 | |
| 17:22:04 | efried | We still have some more minor deltas to do right? | |
| 17:22:26 | efried | alex_xu: you mean something like in_tree=!$uuid ? | |
| 17:22:32 | alex_xu | yes | |
| 17:23:10 | sean-k-mooney | i havent done a full review but the inital change i have looked at seam correct | |
| 17:23:35 | efried | alex_xu: I don't think so. cdent, we haven't done !in_tree yet, right? | |
| 17:23:48 | efried | alex_xu: why? | |
| 17:24:06 | efried | oh | |
| 17:24:08 | alex_xu | efried: for his https://review.opendev.org/#/c/681383/5/nova/scheduler/manager.py@155 | |
| 17:24:36 | alex_xu | if we have that, then we can ignore the src host for the same host resize | |
| 17:24:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 17:24:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 17:25:14 | alex_xu | efried: emm...or we do a manually filter after that | |
| 17:25:18 | sean-k-mooney | all the numa jobs just got killed | |
| 17:25:52 | artom | mriedem, dansmith ^^ | |
| 17:26:13 | efried | alex_xu: Isn't this the famous "doubled allocations" bug? | |
| 17:26:30 | alex_xu | efried: no, it isn't | |
| 17:27:07 | alex_xu | efried: when a host enable same host resize, and that host is the only host has available PCPU now. | |
| 17:27:22 | sean-k-mooney | alex_xu: cant ever ignore the source host for same host reszie or rebuild | |
| 17:27:40 | efried | yeah, I would think for same-host resize we would want to restrict to *only* the current host | |
| 17:27:53 | efried | ...having calculated the positive difference from our current allocations | |
| 17:27:53 | alex_xu | efried: we will try to allocate PCPU in placement, and yes, placement tell you the source host has PCPU, but later the scheduler will check ignore host, the source host is in the ignore host. so the scheduling failed. | |
| 17:28:06 | dansmith | efried: you don't know you're doing a same-host resize | |
| 17:28:19 | efried | oh, bummer | |
| 17:28:23 | alex_xu | sean-k-mooney: sorry, I mean the enable same host resize | |
| 17:28:27 | alex_xu | oops | |
| 17:28:35 | alex_xu | disalbe the same host resize | |
| 17:28:40 | sean-k-mooney | same host resize with pcpus is way more involved | |
| 17:28:47 | efried | so what we should really do is two GET /a_c calls | |
| 17:28:48 | sean-k-mooney | well not pcpu numa in general | |
| 17:29:24 | efried | one with in_tree=!$cur_host with the full set of resources | |
| 17:29:24 | efried | one with in_tree=$cur_host with the diff'd set of resources | |
| 17:29:24 | efried | and then combine the results. | |
| 17:29:56 | mriedem | umm, you're not talking about adding that into the series for train now are you? | |
| 17:30:05 | sean-k-mooney | that still has the double allocation proablem | |
| 17:30:12 | alex_xu | dansmith: I know the ignored_host from request spec, so I can check that, if the allocation_candidates return the host in the ignore_hosts, then fallback to VCPU | |
| 17:30:17 | efried | sean-k-mooney: how would it? | |
| 17:30:23 | mriedem | are you essentially trying to say, avoid the same host for a resize if the instance has PCPU resources? | |
| 17:30:35 | dansmith | alex_xu: you want to fall back to vcpu allocation if the only host that gets returned is the one you're on? | |
| 17:30:39 | dansmith | alex_xu: that's....crazy | |
| 17:30:54 | sean-k-mooney | the in_tree=$cur_host call would require double the resouces | |
| 17:31:01 | sean-k-mooney | the current resouce and the resize to resouces | |
| 17:31:07 | dansmith | alex_xu: or are you saying fall back to vcpu allocation but with the current host excluded? | |
| 17:31:26 | sean-k-mooney | anyway the numa toplogy filter requires double the resouces so its not a new thing | |
| 17:31:32 | efried | sean-k-mooney: no, that's why the in_tree=$cur_host is made with the *diff* of the current and new resources (only the positive ones) | |
| 17:31:45 | efried | (this is all theoretical btw) | |
| 17:31:52 | alex_xu | dansmith: fallback to vcpu if the only host get returned is the one you are on. After fallback vcpu, I won't get VCPu availalbe on the source host again | |
| 17:32:01 | sean-k-mooney | efried: i mention this on irc last week | |
| 17:32:06 | sean-k-mooney | in placment channel | |
| 17:32:23 | sean-k-mooney | efried: i have a customer request to fix inplemce rebuild with numa toplogy | |
| 17:32:37 | efried | sean-k-mooney: Yeah, I thought it sounded familiar. | |
| 17:32:39 | sean-k-mooney | so i or someone on my team needs to actully make this work in U | |
| 17:32:47 | cdent | i confirm that in_tree=! hasn't happened yet (sorry was away from computer for a few minutes doing network management) | |
| 17:32:53 | sean-k-mooney | and backport somthi esle to queens | |
| 17:33:01 | mriedem | sean-k-mooney: there is another nickel | |
| 17:33:18 | mriedem | sean-k-mooney: remember when you said you'd make sriov pci device live migration claims part of the actual claims code in U too/ | |
| 17:33:19 | mriedem | ? | |
| 17:33:32 | sean-k-mooney | mriedem: yes but this time we have the customer wanting use to ship them the backported code in october | |
| 17:33:34 | dansmith | U is going to be a big cycle.. HUGE. YUGE. | |
| 17:33:42 | dtroyer | efried, sean-k-mooney, alex_xu: status update: we've re-triggered the pmem job on Intel CI with the changes on PS22, should have some results in about 45 min… | |
| 17:33:52 | mriedem | sean-k-mooney: backported code for PCPU? | |
| 17:33:54 | efried | dansmith: no it's not. | |