Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-01
19:46:58 mriedem dansmith: MessagingTimeouts in nova-next in the latest failure in https://review.opendev.org/#/c/691390/
19:47:07 mriedem looks like just a slow node?
19:47:24 mriedem dstat shows ram usage around 6.5gb for most of the run
19:47:40 mriedem well, at least around the time it was timing out
19:48:44 mriedem load average really spikes around the time of the timeouts
19:59:37 sean-k-mooney mriedem: you ruled out memory right
20:00:15 sean-k-mooney i allocated a little too many hugpage when playing with dpdk today and when i almost exausted memory in the vm i was getting big cpu spikes
20:00:46 sean-k-mooney like all core 100% for 3 secons then normal then spike then normal
20:08:15 mriedem peakmemtracker shows mysql as the top memory user but it doesn't look like it's running out
20:08:27 mriedem should be 8gb of ram on the node
20:08:40 sean-k-mooney ya you are proably fine
20:09:32 sean-k-mooney i had no swap and allcoated 4G of hugepages instead of 2 by mistake so i was running out
20:21:14 efried jroll: You available to brainstorm some of this vTPM stuff?
20:21:50 jroll efried: maybe?
20:22:05 jroll I'm not really an expert on the TPM bits, but I have a few minutes to chat, yes
20:22:40 efried jroll: I'm just trying to understand how the nova flow needs to work.
20:22:55 jroll sure, I can try to help.
20:23:18 efried If each boot needs to be able to specify an independent value, that's a bigger change than other things.
20:24:23 efried Basically what I'm trying to figure out is where the value of `secret` comes from in
20:24:23 efried <tpm model='tpm-tis'>
20:24:23 efried <backend type='emulator' version='2.0'>
20:24:23 efried <encryption secret='6dd3e4a5-1d76-44ce-961f-f119f5aad935'/>
20:24:23 efried </backend>
20:24:23 efried </tpm>
20:24:35 jroll right
20:24:43 jroll so take the fact that's it's a tpm out of your head
20:24:51 efried hah, that's easy enough
20:24:56 efried I can't even spell tpm yet.
20:24:58 jroll basically, there's a file for the instance on the hypervisor's disk, which needs to be encrypted
20:25:05 jroll to encrypt a file, you need a secret
20:25:30 efried to your understanding, is the value of `secret` the actual secret, or is it an identifier for a secret that needs to be looked up?
20:25:35 jroll the 6dd3e4a5-1d76-44ce-961f-f119f5aad935 in your paste is a reference to a secret object stored in libvirt (i.e. virsh secret-set 6dd3e4a5-1d76-44ce-961f-f119f5aad935 myawesomesecret)
20:25:46 efried "a reference" <= ✔
20:26:05 jroll see https://libvirt.org/formatsecret.html#vTPMUsageType
20:26:11 efried "stored in libvirt" -- from the host or from within the VM
20:26:13 efried ...okay...
20:26:20 jroll from the host, yeah
20:26:26 jroll the VM knows nothing about this file
20:26:27 efried oh, look, a vTPM usage man page, that's convenient.
20:26:30 jroll :D
20:26:50 jroll I found it from a libvirt page you linked in that spec :P
20:26:56 efried butbutbut...
20:27:10 efried I thought the whole point of this was that someone who walks away with the disk can't get at the TPM
20:27:31 jroll correct, because the file representing the TPM is encrypted
20:27:45 efried with a key that's available on the host tho?
20:27:47 jroll I *assume*, but have not verified, that this secret is stored in memory, not on disk
20:28:18 jroll which is why I mentioned we may want to call virSecretSetValue on every instance start()
20:28:42 jroll as if I'm correct about it being in-memory, a host reboot or instance migration would mean we need to set that secret again
20:28:54 efried I thought I read somewhere that it's in memory
20:28:57 efried https://github.com/qemu/qemu/blob/master/docs/specs/tpm.txt
20:29:08 efried L14
20:29:14 efried and 31
20:29:52 jroll The TIS interface makes a memory mapped IO region in the area 0xfed40000 -
20:29:54 jroll 0xfed44fff available to the guest operating system.
20:30:17 jroll I read that as it maps the decrypted version of the file on disk into memory, but I'm not good with terms at this level
20:31:12 jroll I thought I read somewhere that swtpm keeps the state on disk, but I don't see it now
20:31:20 efried "...reboot or ... migration ... set that secret again"
20:31:45 jroll so if swtpm keeps it in memory, and encrypted, that's even better
20:32:07 efried If the host has possession of the master secret, everything sounds easy. But it also sounds like it doesn't satisfy your use case, which is "must not be compromised if someone walks off with the disk".
20:32:36 jroll if the host keeps the master secret in memory only, that means it is not compromised if someone walks off with the disk
20:32:58 efried okay, how does the master secret get into memory?
20:33:07 jroll via virSecretSetValue
20:33:13 jroll per https://libvirt.org/formatsecret.html#vTPMUsageType
20:33:25 efried ...with a value or file that comes from where?
20:33:37 jroll that's the TODO in this spec that I don't have an answer to
20:33:41 jroll :)
20:34:28 efried Because I don't expect a flow like
20:34:28 efried - Host boots
20:34:29 efried - Operator intervention is required to type in a password or load up a file
20:34:29 efried - now your VMs can start
20:34:29 efried to be acceptable in general
20:34:47 jroll I agree
20:35:12 jroll so we'd have to have something in the instance database record, which is either a secret, or a reference to a secret
20:35:25 jroll ideally the secret would be in barbican or equivalent
20:35:55 jroll and that secret could be fetched by nova-compute as needed
20:36:26 efried and you're allowed to fetch it by virtue of... being an authed user?
20:37:17 jroll I would assume by being the nova user
20:37:35 jroll or having access to the rabbit bus, if we had nova-compute ask nova-conductor for it
20:38:07 efried <red flag> upcall </red flag>
20:38:20 jroll heh
20:38:40 efried I was looking over some stuff about trusted certs for image signatures
20:38:57 jroll do computes not ever ask the conductor for info? what about all the database reads?
20:39:03 efried https://docs.openstack.org/nova/latest/user/certificate-validation.html
20:39:24 efried jroll: at least in theory, traffic only ever flows one way, from the conductor down to the compute
20:39:52 efried There may be one upcall left over, I can't remember, but we've been working hard to purge them and not allow any new ones.
20:40:12 jroll erm, how does a compute read from the database?
20:40:28 efried I think there are two databases
20:40:30 jroll it isn't an RPC upcall, I get that, but it's still a call to the conductor, just buried in o.vo
20:40:56 efried I'm not real well versed on this stuff I'm afraid
20:41:04 jroll ok
20:41:04 umbSublime I'm not sure that would apply, but say the secret were stored in barbican couldn't the compute host query barbican by means of the service_token ?
20:41:09 jroll just sayin': https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L53
20:41:24 jroll umbSublime: probably, yeah
20:42:33 efried jroll: I don't know how relevant the certificate validation thing is; I was just looking at how it tries to bootstrap secrets.
20:43:41 efried unfortunately it seems to have added new CLI/API fields -- that would be the "specify a secret to nova boot" route. I would hate to have to do that because it blows up the feature scope significantly.
20:43:55 jroll right
20:44:10 efried without that, we like add a new extra spec and parlay it into xml and we're done.
20:44:27 efried with it, we've got microversions and client/CLI enhancements and on and on.
20:44:31 jroll it looks similar to how TPM secret storage could work, in that nova-compute has to fetch a secret from barbican to boot the instance
20:44:55 jroll in the image validation case, the user provides the secret reference, because the user signs the image

Earlier   Later