| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-01 | |||
| 17:40:36 | dansmith | and it looked overly normal to me | |
| 17:40:53 | dansmith | mysql wasn't spiking that I could tell, it was the big memory user, but only like 400MB at the time | |
| 17:41:02 | dansmith | IO wasn't crazy, etc | |
| 17:41:30 | mriedem | yeah nothing obvious like the 0 i/ops in the other thing earlier today | |
| 17:42:06 | dansmith | I didn't catch the zero iops thing | |
| 17:42:08 | mriedem | and there is no rabbit involved b/c we're going straight to the db yeah? | |
| 17:42:14 | dansmith | you were assuming zero iops meant bad? | |
| 17:42:30 | mriedem | the 0 iops thing was related to another bug in the ML where we get long timeouts | |
| 17:42:42 | mriedem | which triggered the resize long_rpc_timeout discussion | |
| 17:43:08 | mriedem | and a very obvious message in the mysql error log | |
| 17:43:29 | dansmith | you mean zero iops for a long period of time or something I assume/ | |
| 17:43:35 | mriedem | 3 minutes yeah | |
| 17:43:54 | mriedem | while placement was processing a POST /allocations request | |
| 17:43:57 | dansmith | okay, yeah, one or two samples with zero iops wouldn't be alarming to me, but minutes of it, for sure | |
| 17:46:49 | mriedem | and https://bugs.launchpad.net/nova/+bug/1825584 shouldn't be related to the scheduler timing out waiting for a response from cell1 b/c there is no uwsgi involved there | |
| 17:46:49 | openstack | Launchpad bug 1825584 in OpenStack Compute (nova) "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Undecided,Confirmed] | |
| 17:46:53 | mriedem | just scheduler running with eventlet | |
| 18:30:42 | openstackgerrit | Merged openstack/nova stable/rocky: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/691283 | |
| 18:36:07 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/691284 | |
| 18:48:25 | mriedem | interesting that we don't have a task_state change while confirming a resized server | |
| 18:48:35 | mriedem | to prevent other actions from happening while still confirming and cleaning up the source host | |
| 18:57:59 | mriedem | but we do have task_states.RESIZE_REVERTING for reverting a resize | |
| 19:36:52 | openstackgerrit | Merged openstack/nova master: doc: link to nova code review guide from dev policies https://review.opendev.org/692569 | |
| 19:36:57 | openstackgerrit | Merged openstack/nova master: Document CD mentality policy for nova contributors https://review.opendev.org/692572 | |
| 19:46:58 | mriedem | dansmith: MessagingTimeouts in nova-next in the latest failure in https://review.opendev.org/#/c/691390/ | |
| 19:47:07 | mriedem | looks like just a slow node? | |
| 19:47:24 | mriedem | dstat shows ram usage around 6.5gb for most of the run | |
| 19:47:40 | mriedem | well, at least around the time it was timing out | |
| 19:48:44 | mriedem | load average really spikes around the time of the timeouts | |
| 19:59:37 | sean-k-mooney | mriedem: you ruled out memory right | |
| 20:00:15 | sean-k-mooney | i allocated a little too many hugpage when playing with dpdk today and when i almost exausted memory in the vm i was getting big cpu spikes | |
| 20:00:46 | sean-k-mooney | like all core 100% for 3 secons then normal then spike then normal | |
| 20:08:15 | mriedem | peakmemtracker shows mysql as the top memory user but it doesn't look like it's running out | |
| 20:08:27 | mriedem | should be 8gb of ram on the node | |
| 20:08:40 | sean-k-mooney | ya you are proably fine | |
| 20:09:32 | sean-k-mooney | i had no swap and allcoated 4G of hugepages instead of 2 by mistake so i was running out | |
| 20:21:14 | efried | jroll: You available to brainstorm some of this vTPM stuff? | |
| 20:21:50 | jroll | efried: maybe? | |
| 20:22:05 | jroll | I'm not really an expert on the TPM bits, but I have a few minutes to chat, yes | |
| 20:22:40 | efried | jroll: I'm just trying to understand how the nova flow needs to work. | |
| 20:22:55 | jroll | sure, I can try to help. | |
| 20:23:18 | efried | If each boot needs to be able to specify an independent value, that's a bigger change than other things. | |
| 20:24:23 | efried | Basically what I'm trying to figure out is where the value of `secret` comes from in | |
| 20:24:23 | efried | <tpm model='tpm-tis'> | |
| 20:24:23 | efried | <backend type='emulator' version='2.0'> | |
| 20:24:23 | efried | <encryption secret='6dd3e4a5-1d76-44ce-961f-f119f5aad935'/> | |
| 20:24:23 | efried | </backend> | |
| 20:24:23 | efried | </tpm> | |
| 20:24:35 | jroll | right | |
| 20:24:43 | jroll | so take the fact that's it's a tpm out of your head | |
| 20:24:51 | efried | hah, that's easy enough | |
| 20:24:56 | efried | I can't even spell tpm yet. | |
| 20:24:58 | jroll | basically, there's a file for the instance on the hypervisor's disk, which needs to be encrypted | |
| 20:25:05 | jroll | to encrypt a file, you need a secret | |
| 20:25:30 | efried | to your understanding, is the value of `secret` the actual secret, or is it an identifier for a secret that needs to be looked up? | |
| 20:25:35 | jroll | the 6dd3e4a5-1d76-44ce-961f-f119f5aad935 in your paste is a reference to a secret object stored in libvirt (i.e. virsh secret-set 6dd3e4a5-1d76-44ce-961f-f119f5aad935 myawesomesecret) | |
| 20:25:46 | efried | "a reference" <= ✔ | |
| 20:26:05 | jroll | see https://libvirt.org/formatsecret.html#vTPMUsageType | |
| 20:26:11 | efried | "stored in libvirt" -- from the host or from within the VM | |
| 20:26:13 | efried | ...okay... | |
| 20:26:20 | jroll | from the host, yeah | |
| 20:26:26 | jroll | the VM knows nothing about this file | |
| 20:26:27 | efried | oh, look, a vTPM usage man page, that's convenient. | |
| 20:26:30 | jroll | :D | |
| 20:26:50 | jroll | I found it from a libvirt page you linked in that spec :P | |
| 20:26:56 | efried | butbutbut... | |
| 20:27:10 | efried | I thought the whole point of this was that someone who walks away with the disk can't get at the TPM | |
| 20:27:31 | jroll | correct, because the file representing the TPM is encrypted | |
| 20:27:45 | efried | with a key that's available on the host tho? | |
| 20:27:47 | jroll | I *assume*, but have not verified, that this secret is stored in memory, not on disk | |
| 20:28:18 | jroll | which is why I mentioned we may want to call virSecretSetValue on every instance start() | |
| 20:28:42 | jroll | as if I'm correct about it being in-memory, a host reboot or instance migration would mean we need to set that secret again | |
| 20:28:54 | efried | I thought I read somewhere that it's in memory | |
| 20:28:57 | efried | https://github.com/qemu/qemu/blob/master/docs/specs/tpm.txt | |
| 20:29:08 | efried | L14 | |
| 20:29:14 | efried | and 31 | |
| 20:29:52 | jroll | The TIS interface makes a memory mapped IO region in the area 0xfed40000 - | |
| 20:29:54 | jroll | 0xfed44fff available to the guest operating system. | |
| 20:30:17 | jroll | I read that as it maps the decrypted version of the file on disk into memory, but I'm not good with terms at this level | |
| 20:31:12 | jroll | I thought I read somewhere that swtpm keeps the state on disk, but I don't see it now | |
| 20:31:20 | efried | "...reboot or ... migration ... set that secret again" | |
| 20:31:45 | jroll | so if swtpm keeps it in memory, and encrypted, that's even better | |
| 20:32:07 | efried | If the host has possession of the master secret, everything sounds easy. But it also sounds like it doesn't satisfy your use case, which is "must not be compromised if someone walks off with the disk". | |
| 20:32:36 | jroll | if the host keeps the master secret in memory only, that means it is not compromised if someone walks off with the disk | |
| 20:32:58 | efried | okay, how does the master secret get into memory? | |
| 20:33:07 | jroll | via virSecretSetValue | |
| 20:33:13 | jroll | per https://libvirt.org/formatsecret.html#vTPMUsageType | |
| 20:33:25 | efried | ...with a value or file that comes from where? | |
| 20:33:37 | jroll | that's the TODO in this spec that I don't have an answer to | |
| 20:33:41 | jroll | :) | |
| 20:34:28 | efried | Because I don't expect a flow like | |
| 20:34:28 | efried | - Host boots | |
| 20:34:29 | efried | - Operator intervention is required to type in a password or load up a file | |
| 20:34:29 | efried | - now your VMs can start | |
| 20:34:29 | efried | to be acceptable in general | |
| 20:34:47 | jroll | I agree | |
| 20:35:12 | jroll | so we'd have to have something in the instance database record, which is either a secret, or a reference to a secret | |
| 20:35:25 | jroll | ideally the secret would be in barbican or equivalent | |
| 20:35:55 | jroll | and that secret could be fetched by nova-compute as needed | |
| 20:36:26 | efried | and you're allowed to fetch it by virtue of... being an authed user? | |