| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-20 | |||
| 15:21:37 | efried | but otherwise it's a pretty similar model. | |
| 15:21:55 | sean-k-mooney | im not sure there is an existing freature for that | |
| 15:22:04 | sean-k-mooney | if there was we would have suggested using it for cyborg | |
| 15:22:16 | dansmith | it's similar, but I would expect that glance would gain a new container format for that, | |
| 15:22:24 | dansmith | whereas this is more "please store this binary blob for me" | |
| 15:22:54 | sean-k-mooney | well that is what the bitstream kindo of is but sure | |
| 15:22:59 | efried | sean-k-mooney: we're talking about the emulated TPM device, which libvirt stores outside of the instance dir. When we shelve-offload, we need to carry that vdev along somehow. We're brainstorming ways to make that happen. | |
| 15:22:59 | efried | see https://review.opendev.org/#/c/686804/6/specs/ussuri/approved/add-emulated-virtual-tpm.rst@197 | |
| 15:23:19 | sean-k-mooney | ah ok | |
| 15:23:31 | dansmith | sean-k-mooney: it's not if there's some agreed-upon format for containing those bitstreams, even though the bitstream themselves aren't standard | |
| 15:24:07 | sean-k-mooney | dansmith: well for the bitstream i think the plan was intilaly to use raw | |
| 15:24:16 | sean-k-mooney | but there isnt a standard format | |
| 15:24:18 | dansmith | sean-k-mooney: you mean "bare"? | |
| 15:24:29 | dansmith | or you mean no additional wrapper around the bitstream? | |
| 15:24:33 | sean-k-mooney | i mean image type raw container type bare | |
| 15:24:46 | dansmith | yeah, so that doesn't seem like a good idea to me | |
| 15:24:55 | dansmith | because nova thinks it can boot those kinds of things | |
| 15:24:57 | efried | sean-k-mooney: the spec had originally been proposing to copy the vdev into the instance dir so it would be carried along for live migration, and then unpack it on the other side. But the instance dir is blown away for offload, so that model doesn't carry. | |
| 15:25:00 | dansmith | also, the other thing to keep in mind here, | |
| 15:25:05 | sean-k-mooney | well the bitstream format is vendeor depenent | |
| 15:25:13 | sean-k-mooney | it would be nice to have a sperate format for it | |
| 15:25:20 | dansmith | is that for things like glance multi-store, you have to know that those things are linked together when you move/copy between stores | |
| 15:25:28 | dansmith | and, | |
| 15:25:39 | dansmith | if you use that snapshot to spawn to a ceph-using compute node, | |
| 15:25:45 | dansmith | what do you do with that tpm image? | |
| 15:25:50 | dansmith | copy to everyone? ignore? | |
| 15:26:00 | sean-k-mooney | efried: right we would need to snapshot it and store it somewhere | |
| 15:26:09 | dansmith | "copy to everyone" meaning.. any instance that CoWs from that should get a copy of the tpm | |
| 15:26:47 | efried | good question, I hope if you're using a snapshot to create multiple VMs from, you would want them to start with a clean TPM, but mebbe not. | |
| 15:26:58 | dansmith | efried: but how do you know? | |
| 15:27:08 | dansmith | efried: if I snapshot my instance and delete it, then come back a year later for black friday, | |
| 15:27:10 | sean-k-mooney | dansmith: by the way this topic in general sound more like the goal of glare to solve. e.g. a generic artifact store | |
| 15:27:15 | dansmith | you don't know if I'm going to create one or a hundred of those | |
| 15:27:18 | sean-k-mooney | but i dont think glare is a thing anymore | |
| 15:27:34 | efried | dansmith: your point is that we can't differentiate between shelve-offload-unshelve and snapshot-restore? | |
| 15:27:38 | dansmith | sean-k-mooney: maybe, not sure glare was really for keeping this kind of thing, vs things like template configs.. but maybe | |
| 15:28:12 | dansmith | efried: no we definitely can, I'm saying snapshot-restore has a similar problem if you store the thing in glance or otherwise link it via the snapshot itself, vs. the image record in nova | |
| 15:28:26 | sean-k-mooney | so i guess there are different usecase but i would not assuem that we want to share tpm snapshot betwen multipel instnaces | |
| 15:29:01 | sean-k-mooney | that said i can see a use case for that where i store secure boot keys or something in the tpm snapshot | |
| 15:29:04 | dansmith | sean-k-mooney: agree | |
| 15:29:11 | efried | sean-k-mooney: I think dansmith is saying that, even if we put that stake in the ground, we can't necessarily know when we go to use a snapshot | |
| 15:29:21 | dansmith | right, | |
| 15:29:27 | sean-k-mooney | ya | |
| 15:29:36 | efried | but dansmith I would think we would only ever restore the vTPM if the operation is unshelve. If it's just "boot with this image" we don't. | |
| 15:29:43 | dansmith | so it would be better if we have a solution where we the linkage is between the instance and the tpm, not the snapshot image and the tpm | |
| 15:30:15 | efried | yeah, that works. I was actually thinking to use the instance UUID as the link | |
| 15:30:27 | dansmith | eh? | |
| 15:30:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: PoC for using COMPUTE_SAME_HOST_COLD_MIGRATE https://review.opendev.org/695220 | |
| 15:30:45 | efried | in the metadata of both images, store vtpm_original_instance_uuid=$instance_uuid | |
| 15:30:54 | dansmith | that's a bad idea, IMHO | |
| 15:31:01 | dansmith | that opens up an attack vector | |
| 15:31:17 | efried | on unshelve, I need to make sure that the instance I'm unshelving is $instance_uuid, and I also need to suck in the vtpm image with that $instance_Uuid | |
| 15:31:43 | dansmith | if I can create an image, then I could create a tpm image for your instance and when you go to unshelve, you use my tpm instead of nothing beause you didn't have one before, so I can inject keys and get you to use them potentially | |
| 15:31:48 | sean-k-mooney | dansmith: because the uuid can be updated | |
| 15:32:07 | sean-k-mooney | we likely need to sotre this nova side to be safe | |
| 15:32:12 | dansmith | if you're going to store the image in a glance image, storing the id of that in sysmeta would be much stronger a link I think | |
| 15:32:23 | sean-k-mooney | ya that is what i was thinking | |
| 15:32:36 | dansmith | and maybe the sha256 of it too | |
| 15:32:51 | efried | dansmith: Seems like that attack vector is open anyway, just narrower... oh, yeah, a signature ought to close it. | |
| 15:33:19 | dansmith | efried: it means you have to have the ability to modify the internals of nova and create a malicious image.. it's much narrower | |
| 15:33:43 | dansmith | creating images is done via the api.. sysmeta is only at the db layer so requires a much larger exploit | |
| 15:34:17 | sean-k-mooney | ya also if the tpm is encypted | |
| 15:34:21 | efried | dansmith: also keep in mind the file is useless without... yeah. | |
| 15:34:29 | dansmith | note that I don't *know* that abusing glance in this way is legit | |
| 15:34:32 | sean-k-mooney | you not only need to replace the iamge it has to be encypted with teh right key | |
| 15:34:47 | dansmith | efried: it's not useless, it's still a dictionary attack vector, but I understand | |
| 15:35:17 | dansmith | sean-k-mooney: right, but the key is get-able from barbican via the api | |
| 15:35:21 | efried | dansmith: so the instance record sticks around when offloaded? | |
| 15:35:29 | dansmith | efried: that's the whole point of shelve | |
| 15:35:35 | efried | okay | |
| 15:35:56 | sean-k-mooney | true | |
| 15:35:58 | efried | keep the instance record, blow away the allocations etc. | |
| 15:36:13 | efried | and I assume we can unshelve onto any host, doesn't have to be the original. | |
| 15:36:22 | sean-k-mooney | yes | |
| 15:36:28 | efried | (within the bounds of scheduler restrictions of course) | |
| 15:36:30 | sean-k-mooney | we go to the schduler to select the hosts | |
| 15:36:39 | dansmith | likely not the original | |
| 15:36:56 | dansmith | shelve was intended as "cold storage" so some significant amount of time would have passed before you unshelve, so very unlikely you land on the same host | |
| 15:37:20 | dansmith | I will not always be used that way, but.. can't make any assumptions | |
| 15:37:24 | efried | okay, cool, I'm going to write this up and let the stakeholders know we've doubled the complexity again. | |
| 15:37:52 | efried | And maybe go ask -glance for recommendations on how to store the thingy. | |
| 15:38:03 | dansmith | efried: we might ask jroll and the Verizon Media (tm) people what they think about storing the tpm in glance | |
| 15:38:06 | dansmith | security-wise | |
| 15:38:25 | efried | Right. | |
| 15:38:47 | efried | Thanks for the talk dansmith sean-k-mooney | |
| 15:39:01 | sean-k-mooney | the other usecase for shelve is public cloud to avoid billing but ya unell you have a server affinity group or somethin its very unlikly to land on the same host | |
| 15:39:08 | dansmith | efried: already been asking some questions in -glance | |
| 15:39:18 | sean-k-mooney | no worres i need to review your spec and code too | |
| 15:39:25 | sean-k-mooney | ill try to do that this week | |
| 15:40:06 | efried | cool | |
| 15:46:01 | bauzas | mriedem: want me to care on backporting https://review.opendev.org/#/c/695145/ to stable branches ? | |
| 15:46:13 | bauzas | or you rather and me reviewing ? | |
| 15:46:29 | mriedem | can you do both? | |
| 15:46:51 | bauzas | both backporting and reviewing ? well, is that a thing ? | |
| 15:47:03 | bauzas | we have elod and lyarwood we can bug | |
| 15:47:51 | mriedem | i have things in stable/train you could review | |
| 15:48:12 | bauzas | mriedem: sure thing | |
| 15:48:21 | mriedem | i guess i don't know if you're just talking about that specific change or reviewing code in general | |
| 15:48:29 | mriedem | because both would be good | |
| 15:48:38 | bauzas | mriedem: no, sorry, was talking of proposing backports of this change | |