| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-11 | |||
| 15:11:01 | sean-k-mooney | or suspend/pause | |
| 15:11:09 | sean-k-mooney | maybe even stop | |
| 15:11:31 | efried | sean-k-mooney: the secret in the domain xml is just the uuid of the secret. So presumably (hopefully) the decryption happens in-memory when the xml is loaded up to actually start the guest. | |
| 15:11:36 | sean-k-mooney | start calls power on which calls reboot which recreates the domain | |
| 15:12:20 | sean-k-mooney | sure but the issue is the secret is not encrypted on disk | |
| 15:12:29 | efried | so regardless of whether the domain xml is there or needs to be recreated, it's when the guest starts up that we would need the secret to be present. | |
| 15:12:53 | lyarwood | again, the secret isn't provided to QEMU as a plaintext string | |
| 15:13:01 | sean-k-mooney | efried: im wondering is there a way to pass it at that point and never save it to disk | |
| 15:13:27 | efried | lyarwood: but it's on disk in cleartext. (base-64 encoded, but effectively plaintext) | |
| 15:13:34 | sean-k-mooney | lyarwood: sure but efried just said it in /etc/libvirt/secrets/ and is just base64 encoded | |
| 15:13:46 | lyarwood | right and that's pretty locked down given the listing above | |
| 15:13:47 | efried | lyarwood: I suspect I'm missing what you're getting at. | |
| 15:14:00 | efried | lyarwood: you mean 600 owned by root locked down? | |
| 15:14:03 | lyarwood | if someone had access to root it's already game over | |
| 15:14:10 | sean-k-mooney | lyarwood: that not good enough in my view | |
| 15:14:22 | efried | pretty sure in jroll's view too. | |
| 15:14:23 | sean-k-mooney | the operator shoudl not be able to see the secreate | |
| 15:14:28 | lyarwood | they could already get in and decrypt various things from the running QEMU process | |
| 15:14:46 | canori01 | Hello, would I evacuate an instance that's running into 'AllocationUpdateFailed: Failed to update allocations for consumer' running on Stein | |
| 15:15:00 | lyarwood | well that just isn't within libvirt's security model AFAIK | |
| 15:15:02 | canori01 | I mean, how would I recover from that | |
| 15:15:31 | efried | lyarwood: Actually I think we're drawing the line at "compromised root is acceptable risk; stolen disk must be secure". | |
| 15:16:07 | jroll | more like we realize compromised root on a running hypervisor is game over anyway, but yeah | |
| 15:16:27 | lyarwood | efried: kk, danpb had a blog about secret handling a while ago that might cover a way around that ./me checks | |
| 15:16:27 | efried | IOW I think it's the nova admin creds that are the root of trust. | |
| 15:16:32 | efried | thanks lyarwood | |
| 15:16:43 | sean-k-mooney | efried: https://libvirt.org/formatsecret.html#SecretAttributes | |
| 15:16:53 | jroll | lyarwood: thanks, I was just going to ask that | |
| 15:16:55 | sean-k-mooney | so it look like you can mare a secret as ephemeral | |
| 15:16:56 | efried | having displayed any understanding whatsoever, lyarwood, you're now permanently on call for support. | |
| 15:17:06 | sean-k-mooney | so its only kept in memory and never on disk | |
| 15:17:17 | sean-k-mooney | so if we store it in barbican and then do that that would be better | |
| 15:17:34 | efried | oh, cool, I saw that but hadn't played with it yet, will have to try that out and see if it does the trick. | |
| 15:17:42 | sean-k-mooney | well ephemeral=yes private=yes | |
| 15:17:51 | efried | The secret I created while playing was private but not ephemeral. | |
| 15:18:14 | jroll | can confirm ephemeral=yes leaves it off of disk | |
| 15:18:35 | sean-k-mooney | so form a nova point of view can we use barbican to store it | |
| 15:18:41 | efried | cool, so the nova user is the root of trust because has access to the barbican entry. | |
| 15:18:49 | efried | yeah, what sean-k-mooney said. | |
| 15:18:56 | jroll | and restarting libvirt nukes an ephemeral secret | |
| 15:18:59 | jroll | \o/ | |
| 15:19:22 | sean-k-mooney | jroll: that is proably ok/good | |
| 15:19:25 | efried | now we just gotta hope you don't have to restart libvirt to make it pick up the secret :P | |
| 15:19:29 | lyarwood | thanks sean-k-mooney | |
| 15:19:46 | jroll | efried: fwiw, I didn't need to restart, but I'm on older libvirt playing with tls secrets, not tpm | |
| 15:20:10 | sean-k-mooney | jroll: i dont know if libvirt makes a difference | |
| 15:20:21 | efried | jroll: didn't need to restart to do what? | |
| 15:20:56 | efried | in my case everything appeared to be moving along fine until I tried booting the guest, then the nova log got an exception complaining it couldn't find my secret. | |
| 15:21:27 | efried | anyway, ephemeral would be entirely useless in that case, so it MUST work (famous last words) | |
| 15:21:31 | jroll | efried: ah, I was just doing 'virsh secret-list' | |
| 15:21:33 | jroll | heh | |
| 15:21:46 | jroll | sean-k-mooney: if there's a bug in newer libvirt it might :) | |
| 15:22:04 | sean-k-mooney | oh i ment between secrete usage/types | |
| 15:22:18 | sean-k-mooney | i think it just has a secret that is use for many things | |
| 15:22:21 | efried | jroll: btw, there was a question as to whether we needed to bother providing the non-encrypted version of vTPM support at all. | |
| 15:22:23 | jroll | older libvirt won't let me create a vtpm secret type ¯\_(ツ)_/¯ | |
| 15:22:44 | efried | right, that needs to be >=5.6.0, which afaict you have to compile yourself. | |
| 15:22:46 | jroll | efried: I'd be pro-not-supporting that, I don't like footguns | |
| 15:22:51 | sean-k-mooney | e.g. i dont think a secreate for tls or tpm is a different thing internally in libvirt | |
| 15:23:09 | efried | penick indicated the same. Just that the requirements are going to be way more stringent for it if so. | |
| 15:23:52 | sean-k-mooney | for what its worth rhel/fedora will be shiping it relitvly quickly | |
| 15:23:59 | sean-k-mooney | im not sure when exactly | |
| 15:24:08 | sean-k-mooney | but we have an advance virt stream we use for openstack | |
| 15:24:17 | efried | I guess "with ussuri" is soon enough <shrug> | |
| 15:24:30 | sean-k-mooney | that ships a newer version then base rhel | |
| 15:24:37 | efried | but also the barbican requirement | |
| 15:25:00 | efried | anyway, glad you agree jroll, it makes the code & test surface easier. | |
| 15:25:37 | efried | lyarwood: since you mentioned the qemu side -- do you have any idea what's the minimum qemu version to support encrypted vTPM? I couldn't find that in the docs anywhere. | |
| 15:26:56 | efried | canori01: it would help to know why the allocation update is failing (out of resources? provider conflict? other?), and what action is being performed when that happens. | |
| 15:28:38 | canori01 | efried: I am replacing old hardware. So I powered off the old hypervisor and then did a nova-evacuate. All the guests evacuated except for two which gave me the provider conflict | |
| 15:29:03 | lyarwood | efried: I don't sorry | |
| 15:29:31 | lyarwood | efried: 2.12 is listed in the Stein spec FWIW | |
| 15:30:02 | efried | lyarwood: yeah, that was for non-encrypted, and idk if there's a higher req for encrypted. | |
| 15:30:23 | efried | canori01: If it was a provider conflict, you should just be able to "try again". | |
| 15:30:33 | lyarwood | efried: no idea sorry, kashyap is out today otherwise I'd ask him if he could work that out for us. | |
| 15:30:49 | efried | ah, good plan. I'll stalk him, thanks. | |
| 15:30:52 | sean-k-mooney | efried: tpm emulation was added in 2.11 but im not sure if that included encryption | |
| 15:31:10 | efried | it did not | |
| 15:31:39 | efried | But I don't even know whether qemu required any update for that aspect. It might not have. | |
| 15:31:49 | sean-k-mooney | also this is an intersting site https://repology.org/project/libvirt/versions | |
| 15:33:22 | sean-k-mooney | look like apline has the depencyies you need but very little else at teh moment | |
| 15:34:28 | sean-k-mooney | Fedora Rawhide would "work" too although i suspect nova wont be happy with it in general | |
| 15:35:06 | efried | sean-k-mooney: oh, that reminds me, I ran into another weird quirk with the home-built libvirt... | |
| 15:35:49 | efried | sean-k-mooney: When I restarted the libvirt service, it blew up because it was looking in /usr/lib/x86_64-linux-gnu/ which contained the old libvirt 4.xx | |
| 15:36:06 | efried | I had to go into the service and set LD_LIBRARY_PATH=/usr/lib explicitly to make it work. | |
| 15:36:11 | efried | which seems... weird. | |
| 15:36:17 | efried | had to do the same with n-cpu. | |
| 15:36:52 | sean-k-mooney | ya that does seam strange | |
| 15:37:02 | sean-k-mooney | well clearly i have more work to do | |
| 15:37:56 | dansmith | efried: why is that surprising? | |
| 15:38:19 | dansmith | efried: if you have an old version installed in the standard system library path, and start a new daemon binary that requires the new libs, | |
| 15:38:24 | dansmith | you have to tell it where they are | |
| 15:38:33 | dansmith | (if I'm understanding what you're saying) | |
| 15:38:36 | efried | dansmith: Is /usr/lib/x86_64-linux-gnu/ the "standard system library path"? | |
| 15:38:47 | efried | That was the surprising part to me. I would have thought /usr/lib was. | |
| 15:39:08 | dansmith | efried: for multi-arch stuff on fedora-like distros it is I think yeah | |
| 15:39:12 | sean-k-mooney | dansmith: well the devstack plugin uninstalles the packages and libvirt was compiles with --system | |
| 15:39:34 | efried | sean-k-mooney: Doesn't your plugin just use `make install`? | |
| 15:39:35 | dansmith | sean-k-mooney: clearly not if the old version was still there | |
| 15:39:54 | efried | I would have thought that should have covered symlinking or otherwise replacing versions in any "standard" paths. | |