| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-12 | |||
| 00:30:00 | efried | okay. | |
| 00:30:21 | efried | dansmith: typos courtesy of "shift right"? | |
| 00:30:32 | dansmith | yep | |
| 00:30:52 | dansmith | efried: also, shelve_offload() is a cast, so no error result or moving instance to ERROR if it fails due to a down compute | |
| 00:30:52 | efried | I use dvorak | |
| 00:31:03 | efried | so if I shift right, I just look like I'm having a stroke. | |
| 00:31:16 | dansmith | heh | |
| 00:31:31 | efried | dansmith: that was my next question: offload kinda happens "later" sometimes, right? | |
| 00:31:46 | efried | so... does it still have the context (auth etc) of the user who did the original shelve? | |
| 00:32:00 | dansmith | it can be requested via the API, or via periodic after a timeout yeah | |
| 00:32:11 | dansmith | efried: no it could be the operator via the API, or no context via periodic | |
| 00:32:32 | efried | ugh | |
| 00:32:53 | dansmith | in fact, I can't think of any reason a regular user would ever manually offload | |
| 00:33:09 | efried | make space? | |
| 00:33:26 | efried | but anyway, it's another monkey wrench for me. | |
| 00:33:27 | dansmith | shelve is what causes them to start saving money, offload is the cache purge for the operator so something else can go there, but shouldn't really impact the user, other than to make unshelve take longer | |
| 00:33:41 | dansmith | right, the user doesn't care to make space, they're already saving the money | |
| 00:33:51 | efried | oh, yeah, I guess the space is reclaimed at shelve time. | |
| 00:33:58 | dansmith | the operator might offload a shelved instance to get the space back before the timer | |
| 00:34:09 | dansmith | well, not really, | |
| 00:34:22 | efried | shitballs, I need to go back and rewrite this rewrite. | |
| 00:34:33 | dansmith | the user doesn't get any space (quota) back, but they do get charged less (theoretically) | |
| 00:34:43 | dansmith | the operator doesn't get space (i.e. room for another instance on that host) until the offload | |
| 00:35:01 | efried | oh | |
| 00:35:08 | efried | allocations etc only go away at offload? | |
| 00:35:27 | dansmith | I would think.. the virt driver doesn't know a shelve has happened until the offload | |
| 00:35:30 | efried | so yeah, that's what I meant by "make space". But again yeah, that's something the operator cares about, not the user. | |
| 00:35:35 | dansmith | right | |
| 00:35:50 | efried | the virt driver just gets told to snapshot(). I guess they could figure it out from a task state? | |
| 00:35:56 | efried | but yeah, they don't. | |
| 00:36:13 | dansmith | well, it would be super problematic if they did, | |
| 00:36:37 | efried | Well, on that note... | |
| 00:36:37 | efried | what's the task state during the virt driver's destroy() during the offload? | |
| 00:36:43 | dansmith | because the point of the shelved state is that it is still ready to be powered back on, which means that like POWERED_OFF state, it still takes the same amount of resources as ACTIVE | |
| 00:36:51 | efried | (I'm going to have to find out anyway, but if you know off the top...) | |
| 00:37:18 | dansmith | I dunno, but why? the virt driver shouldn't be implying anything from task_state during an op like that | |
| 00:37:52 | dansmith | the only time virt should ever really look at that stuff is when it's recovering and trying to figure out if something should be deleted or ignored or something like that | |
| 00:37:58 | efried | I was going to save the vTPM backing file during the offload. But that has to be done by the virt driver, so the virt driver was going to have to have some way of knowing it needed to do that. | |
| 00:38:17 | efried | But the fact that the offload is ever (usually) done with no (or admin?) context blows that out of the water. | |
| 00:38:24 | efried | I'm going to have to do it during shelve. | |
| 00:38:30 | dansmith | task_state is really a lock for the api and downstream services to avoid taking two simultaneous actions on an instance | |
| 00:38:37 | dansmith | then you need to tell the virt driver to do it, | |
| 00:38:41 | efried | Which means similarly the virt driver needs to be able to know it should do that during snapshot | |
| 00:38:45 | dansmith | not have the virt driver imply that it needs to do it from the state | |
| 00:38:57 | dansmith | because, for example, shelve is used under the covers for cross-cell-migration | |
| 00:38:59 | efried | so add a kwarg to snapshot()? | |
| 00:39:10 | efried | or a new virt driver method, prep_shelve()? | |
| 00:39:14 | dansmith | yeah, but also, shouldn't that be the job of the compute manager anyway? | |
| 00:39:23 | dansmith | like "give me the tpm so I can save it for you" | |
| 00:39:24 | efried | no, can't be. | |
| 00:39:31 | efried | oh, that could work, yeah. | |
| 00:39:51 | efried | but obv we can't have the compute manager knowing libvirt internals | |
| 00:40:17 | efried | i.e. where that file lives. | |
| 00:40:21 | dansmith | no, of course not, but it can ask for the tpm in a portable form | |
| 00:40:36 | efried | like... an IMAGE!!! (not) | |
| 00:40:40 | dansmith | or even "give me things I should stash for you during a shelve-like snapshot" | |
| 00:41:45 | efried | oo, nothing today. | |
| 00:41:49 | dansmith | then you let compute manager decide whether it is going to save or "lose" those things, because compute manager knows what high-level operation is happening | |
| 00:41:52 | dansmith | virt does not really | |
| 00:42:45 | efried | So could that be a file-like that compute manager is responsible for streaming from? | |
| 00:42:46 | dansmith | and, you let compute manager decide *how* to save those things, instead of teaching every virt driver about all the ways things can be saved (swift, glance, etc) | |
| 00:43:36 | dansmith | efried: I would call it a BytesIO yeah | |
| 00:43:54 | efried | okay, that can work. | |
| 00:43:57 | dansmith | or IOBase or whatever the right bytes-based primitive is | |
| 00:44:34 | efried | That's what I meant: "the appropriate streamable thing that I'm going to have to go learn about cause I clearly don't know it yet" | |
| 00:44:51 | dansmith | BytesIO for your unit test | |
| 00:45:08 | dansmith | file(name, 'rb') for your libvirt from-file implementation | |
| 00:45:18 | dansmith | er, open(name, 'rb') | |
| 00:46:24 | efried | That's gonna be a tarball that I need to clean up afterward. How do I do that? | |
| 00:46:30 | dansmith | which is an io.BufferedReader | |
| 00:46:30 | efried | (Musing to self) | |
| 00:46:49 | dansmith | then just spawn tar and read from the tar pipe, don't create it on disk first | |
| 00:46:56 | efried | I imagine there must be a way to just create the ... yeah, pipe. | |
| 00:47:19 | dansmith | p = subprocess.Popen('tar c - file1 file2', stdout=subprocess.PIPE) | |
| 00:47:31 | dansmith | return p.stdout | |
| 00:48:13 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804 | |
| 00:49:49 | efried | dansmith: thanks for the help, this is gonna make some (more) things easier, again. | |
| 00:49:52 | efried | or at least simpler | |
| 00:50:03 | efried | And with that, I'm outta here. | |
| 00:50:06 | efried | o/ | |
| 00:55:42 | melwitt | TheJulia: around? | |
| 01:05:26 | mriedem | efried: fwiw reading scrollback, the libvirt driver does check task_state to flatten rbd images while unshelving i think | |
| 01:06:18 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9246 | |
| 01:06:56 | mriedem | efried: also btw, immediate shelve offload is the default behavior and there is a context | |
| 01:07:09 | mriedem | the periodic or shelveOffload api only kick in if you've configured the computes to not immediately offload | |
| 01:07:21 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.shelved_poll_interval | |
| 01:07:57 | openstackgerrit | Eric Xie proposed openstack/nova master: Report trait 'COMPUTE_IMAGE_TYPE_PLOOP' https://review.opendev.org/698132 | |
| 01:08:05 | mriedem | dansmith: btw, "because, for example, shelve is used under the covers for cross-cell-migration" is not true - it was for the original poc, but not what i have now | |
| 01:08:12 | mriedem | parts are similar though | |
| 01:09:19 | dansmith | mriedem: well, yeah, sorry I'm collapsing shelve-like snapshot behavior into it | |
| 01:09:50 | dansmith | mriedem: and the task_state check down in the rbd stuff is just a total hack, I hope you'd agree | |
| 01:10:00 | dansmith | because of the layers of plumbing that would be required | |
| 01:10:05 | mriedem | it is | |
| 01:11:09 | mriedem | we're the ones that approved it too :) | |
| 01:11:25 | dansmith | mriedem: given that a snapshot for migration would be very shelve-like, but a task-state check for shelving to save the tpm would be the exact kind of guessing the virt driver shouldn't be doing because composite operations on top of the primitives can't rely on predictable behavior | |
| 01:12:20 | mriedem | yup, case in point https://review.opendev.org/#/c/691991/ | |
| 01:12:24 | dansmith | very shelve-like, meaning it would *need* to store the tpm, but wouldn't if the task_state wasn't shelving, so migrations that happened to cross cells would end up with an empty tpm | |
| 01:12:28 | dansmith | which is exactly my point | |
| 01:12:45 | dansmith | ha yeah | |
| 01:13:57 | dansmith | this marks my 12th hour behind the keyboard today | |