| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-11 | |||
| 23:17:18 | openstackgerrit | Merged openstack/nova master: Add resource provider allocation unset example to troubleshooting doc https://review.opendev.org/696582 | |
| 23:17:25 | openstackgerrit | Merged openstack/nova master: nova-net: Convert remaining API tests to use neutron https://review.opendev.org/696509 | |
| 23:38:17 | pmatulis | how do i map the hypervisor long ID (server show) to the hypervisor hostname (hypervisor list/show)? | |
| 23:40:05 | dansmith | pmatulis: if you server-show as admin, you should see the unobfuscated hostid , IIRC | |
| 23:41:56 | dansmith | pmatulis: as OS-EXT-SRV-ATTR:hypervisor_hostname | |
| #openstack-nova - 2019-12-12 | |||
| 00:01:04 | pmatulis | dansmith, yeah, i'm going to change a nova policy to make that available to non-admins | |
| 00:16:24 | efried | But do we do that, or do we just fail the offload and ERROR the instance? | |
| 00:16:24 | efried | The virt driver's destroy() would have to happen when the compute came back up later, just like we reap instances that are deleted when their compute is down. | |
| 00:16:24 | efried | dansmith: here's a weird one: If I shelve my instance, and then my compute goes down, would a subsequent offload "succeed"? | |
| 00:28:52 | dansmith | efried: offload should succeed when it comes back up, but would fail while the compute is down with just an rpc failure type thing | |
| 00:29:10 | efried | okay. thanks. | |
| 00:29:22 | dansmith | O dpm | |
| 00:29:24 | dansmith | wow. | |
| 00:29:31 | dansmith | I don't think there's a local delete sort of thing for offload | |
| 00:29:51 | dansmith | yeah, there's not | |
| 00:30:00 | efried | okay. | |
| 00:30:21 | efried | dansmith: typos courtesy of "shift right"? | |
| 00:30:32 | dansmith | yep | |
| 00:30:52 | efried | I use dvorak | |
| 00:30:52 | dansmith | efried: also, shelve_offload() is a cast, so no error result or moving instance to ERROR if it fails due to a down compute | |
| 00:31:03 | efried | so if I shift right, I just look like I'm having a stroke. | |
| 00:31:16 | dansmith | heh | |
| 00:31:31 | efried | dansmith: that was my next question: offload kinda happens "later" sometimes, right? | |
| 00:31:46 | efried | so... does it still have the context (auth etc) of the user who did the original shelve? | |
| 00:32:00 | dansmith | it can be requested via the API, or via periodic after a timeout yeah | |
| 00:32:11 | dansmith | efried: no it could be the operator via the API, or no context via periodic | |
| 00:32:32 | efried | ugh | |
| 00:32:53 | dansmith | in fact, I can't think of any reason a regular user would ever manually offload | |
| 00:33:09 | efried | make space? | |
| 00:33:26 | efried | but anyway, it's another monkey wrench for me. | |
| 00:33:27 | dansmith | shelve is what causes them to start saving money, offload is the cache purge for the operator so something else can go there, but shouldn't really impact the user, other than to make unshelve take longer | |
| 00:33:41 | dansmith | right, the user doesn't care to make space, they're already saving the money | |
| 00:33:51 | efried | oh, yeah, I guess the space is reclaimed at shelve time. | |
| 00:33:58 | dansmith | the operator might offload a shelved instance to get the space back before the timer | |
| 00:34:09 | dansmith | well, not really, | |
| 00:34:22 | efried | shitballs, I need to go back and rewrite this rewrite. | |
| 00:34:33 | dansmith | the user doesn't get any space (quota) back, but they do get charged less (theoretically) | |
| 00:34:43 | dansmith | the operator doesn't get space (i.e. room for another instance on that host) until the offload | |
| 00:35:01 | efried | oh | |
| 00:35:08 | efried | allocations etc only go away at offload? | |
| 00:35:27 | dansmith | I would think.. the virt driver doesn't know a shelve has happened until the offload | |
| 00:35:30 | efried | so yeah, that's what I meant by "make space". But again yeah, that's something the operator cares about, not the user. | |
| 00:35:35 | dansmith | right | |
| 00:35:50 | efried | the virt driver just gets told to snapshot(). I guess they could figure it out from a task state? | |
| 00:35:56 | efried | but yeah, they don't. | |
| 00:36:13 | dansmith | well, it would be super problematic if they did, | |
| 00:36:37 | efried | what's the task state during the virt driver's destroy() during the offload? | |
| 00:36:37 | efried | Well, on that note... | |
| 00:36:43 | dansmith | because the point of the shelved state is that it is still ready to be powered back on, which means that like POWERED_OFF state, it still takes the same amount of resources as ACTIVE | |
| 00:36:51 | efried | (I'm going to have to find out anyway, but if you know off the top...) | |
| 00:37:18 | dansmith | I dunno, but why? the virt driver shouldn't be implying anything from task_state during an op like that | |
| 00:37:52 | dansmith | the only time virt should ever really look at that stuff is when it's recovering and trying to figure out if something should be deleted or ignored or something like that | |
| 00:37:58 | efried | I was going to save the vTPM backing file during the offload. But that has to be done by the virt driver, so the virt driver was going to have to have some way of knowing it needed to do that. | |
| 00:38:17 | efried | But the fact that the offload is ever (usually) done with no (or admin?) context blows that out of the water. | |
| 00:38:24 | efried | I'm going to have to do it during shelve. | |
| 00:38:30 | dansmith | task_state is really a lock for the api and downstream services to avoid taking two simultaneous actions on an instance | |
| 00:38:37 | dansmith | then you need to tell the virt driver to do it, | |
| 00:38:41 | efried | Which means similarly the virt driver needs to be able to know it should do that during snapshot | |
| 00:38:45 | dansmith | not have the virt driver imply that it needs to do it from the state | |
| 00:38:57 | dansmith | because, for example, shelve is used under the covers for cross-cell-migration | |
| 00:38:59 | efried | so add a kwarg to snapshot()? | |
| 00:39:10 | efried | or a new virt driver method, prep_shelve()? | |
| 00:39:14 | dansmith | yeah, but also, shouldn't that be the job of the compute manager anyway? | |
| 00:39:23 | dansmith | like "give me the tpm so I can save it for you" | |
| 00:39:24 | efried | no, can't be. | |
| 00:39:31 | efried | oh, that could work, yeah. | |
| 00:39:51 | efried | but obv we can't have the compute manager knowing libvirt internals | |
| 00:40:17 | efried | i.e. where that file lives. | |
| 00:40:21 | dansmith | no, of course not, but it can ask for the tpm in a portable form | |
| 00:40:36 | efried | like... an IMAGE!!! (not) | |
| 00:40:40 | dansmith | or even "give me things I should stash for you during a shelve-like snapshot" | |
| 00:41:45 | efried | oo, nothing today. | |
| 00:41:49 | dansmith | then you let compute manager decide whether it is going to save or "lose" those things, because compute manager knows what high-level operation is happening | |
| 00:41:52 | dansmith | virt does not really | |
| 00:42:45 | efried | So could that be a file-like that compute manager is responsible for streaming from? | |
| 00:42:46 | dansmith | and, you let compute manager decide *how* to save those things, instead of teaching every virt driver about all the ways things can be saved (swift, glance, etc) | |
| 00:43:36 | dansmith | efried: I would call it a BytesIO yeah | |
| 00:43:54 | efried | okay, that can work. | |
| 00:43:57 | dansmith | or IOBase or whatever the right bytes-based primitive is | |
| 00:44:34 | efried | That's what I meant: "the appropriate streamable thing that I'm going to have to go learn about cause I clearly don't know it yet" | |
| 00:44:51 | dansmith | BytesIO for your unit test | |
| 00:45:08 | dansmith | file(name, 'rb') for your libvirt from-file implementation | |
| 00:45:18 | dansmith | er, open(name, 'rb') | |
| 00:46:24 | efried | That's gonna be a tarball that I need to clean up afterward. How do I do that? | |
| 00:46:30 | efried | (Musing to self) | |
| 00:46:30 | dansmith | which is an io.BufferedReader | |
| 00:46:49 | dansmith | then just spawn tar and read from the tar pipe, don't create it on disk first | |
| 00:46:56 | efried | I imagine there must be a way to just create the ... yeah, pipe. | |
| 00:47:19 | dansmith | p = subprocess.Popen('tar c - file1 file2', stdout=subprocess.PIPE) | |
| 00:47:31 | dansmith | return p.stdout | |
| 00:48:13 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804 | |
| 00:49:49 | efried | dansmith: thanks for the help, this is gonna make some (more) things easier, again. | |
| 00:49:52 | efried | or at least simpler | |
| 00:50:03 | efried | And with that, I'm outta here. | |
| 00:50:06 | efried | o/ | |
| 00:55:42 | melwitt | TheJulia: around? | |
| 01:05:26 | mriedem | efried: fwiw reading scrollback, the libvirt driver does check task_state to flatten rbd images while unshelving i think | |
| 01:06:18 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9246 | |
| 01:06:56 | mriedem | efried: also btw, immediate shelve offload is the default behavior and there is a context | |
| 01:07:09 | mriedem | the periodic or shelveOffload api only kick in if you've configured the computes to not immediately offload | |