| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-06 | |||
| 17:13:56 | sean-k-mooney | ok then we can add that as a new feature if you can explain what it is | |
| 17:14:06 | sean-k-mooney | that is not a bug it woudl be a new feature | |
| 17:14:19 | sean-k-mooney | a small one but its still a feature | |
| 17:14:39 | geguileo | sean-k-mooney: https://paste.openstack.org/show/brFgX6MgBlxjgCrE3rbg/ | |
| 17:14:52 | sean-k-mooney | geguileo: what do you mean by volume in https://bugs.launchpad.net/nova/+bug/2023079 | |
| 17:14:55 | geguileo | sean-k-mooney: but that's not the right code | |
| 17:14:59 | geguileo | it's what I used to test | |
| 17:15:27 | sean-k-mooney | oh "When doing a live volume migration " | |
| 17:15:32 | geguileo | but that has CPU implications when running, so it's best only to change it when the block migration is going to happen | |
| 17:15:44 | sean-k-mooney | geguileo: we have a differnt concept called block-migration in the context of live migrating a vm | |
| 17:15:53 | dansmith | ...yeah | |
| 17:16:03 | geguileo | sean-k-mooney: yes, I mean block live migration, I've updated the bug name, thanks | |
| 17:16:04 | dansmith | I was going to say, we should never be block migrating a volume | |
| 17:16:25 | geguileo | sean-k-mooney: ooooh, then soooooooorry for mixing terms (/me facepalms) | |
| 17:16:27 | sean-k-mooney | geguileo: block live migratin in nova mean live migrate a vm with local raw/qcow storage | |
| 17:16:43 | dansmith | it means we literally move all the data | |
| 17:16:56 | dansmith | (all the disk data) | |
| 17:17:03 | geguileo | my bad, I've updated the LP bug | |
| 17:17:10 | geguileo | dansmith: we are currently moving ALL the data | |
| 17:17:23 | sean-k-mooney | geguileo: ya so i know knwo what your trying to fix | |
| 17:17:27 | geguileo | that's why the detect_zeroes would be good for volume live migration | |
| 17:17:43 | sean-k-mooney | so we expect this to be done by cider using the driver assited migration feature | |
| 17:17:43 | geguileo | sean-k-mooney: thanks for you patience in understanding my ramblings :-) | |
| 17:17:57 | sean-k-mooney | however you want nova ot be intelegent enough | |
| 17:18:07 | sean-k-mooney | so that when we fallback to nova doing the volume migration | |
| 17:18:15 | sean-k-mooney | that we also preseve the sparceness | |
| 17:18:23 | geguileo | sean-k-mooney: problem is that driver assisted migration cannot work between different backends (and afaik it doesn't work for any driver even between volumes of the same array) | |
| 17:18:41 | geguileo | sean-k-mooney: afaik today all online volume migrations are done by nova | |
| 17:18:51 | dansmith | wait what? | |
| 17:18:54 | geguileo | I don't think any cinde driver supports it | |
| 17:19:00 | opendevreview | Amit Uniyal proposed openstack/nova master: Reproducer for dangling bdms https://review.opendev.org/c/openstack/nova/+/881457 | |
| 17:19:00 | opendevreview | Amit Uniyal proposed openstack/nova master: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284 | |
| 17:19:13 | sean-k-mooney | geguileo: i was pretty sure you fixed at least one vendor driver last year | |
| 17:19:13 | dansmith | why would we ever want to do that? unless you're crossing AZs or something | |
| 17:19:14 | geguileo | dansmith: all volumes that are attached to now are migrated by nova | |
| 17:19:27 | geguileo | s/now/nova | |
| 17:19:32 | sean-k-mooney | geguileo: if you change backend right | |
| 17:19:38 | sean-k-mooney | not just retype | |
| 17:19:45 | sean-k-mooney | within the same backend | |
| 17:19:59 | dansmith | retype is a different thing right? | |
| 17:20:16 | sean-k-mooney | yes | |
| 17:20:20 | dansmith | I'm talking about you live migrate from one server to the next one in the rack, we should not be moving all the volume data to a *new* volume... | |
| 17:20:28 | geguileo | dansmith: retype is a different think, but many times it triggers a migration | |
| 17:20:40 | geguileo | dansmith: oh, yeah, not a nova live migration | |
| 17:20:45 | geguileo | dansmith: it's a volume live migration | |
| 17:20:57 | sean-k-mooney | retypes can be within hte same backend (just different qos policy) or to a diffent backend | |
| 17:20:58 | geguileo | basically when you mirror the data from one volume to another | |
| 17:21:00 | dansmith | okay, I guess we're confusing too many things | |
| 17:21:10 | geguileo | sean-k-mooney: correct! | |
| 17:21:49 | sean-k-mooney | so i remmeber a custoemr issue like 4-6 months ago where i tought wew fixed scalio or one of the other driver to explcitly preserve sparcens when doign a driver assited voluem migration | |
| 17:21:49 | geguileo | dansmith: what's the right term for moving data from one attached volume to another while the instance is running? | |
| 17:22:04 | dansmith | swap volume I think? | |
| 17:22:14 | geguileo | sean-k-mooney: I fixed for offline and to report the value to Nova | |
| 17:22:17 | dansmith | that's the action we see, AFAIK | |
| 17:22:28 | sean-k-mooney | on the nova side its the swap volume is what is called yes | |
| 17:22:29 | geguileo | dansmith: ok, I'll try to talk about swap volume | |
| 17:22:48 | sean-k-mooney | well no volume migration is fine | |
| 17:23:05 | dansmith | geguileo: not trying to make you use our language, I just need to know that a bunch of terms have been re-used :) | |
| 17:23:19 | sean-k-mooney | but you are asserting that the driver asseited path never works for an online volume migration | |
| 17:23:22 | geguileo | dansmith: oh, I prefer to use the right language | |
| 17:23:41 | geguileo | sean-k-mooney: I don't think we have any cinder driver capable of doing it... | |
| 17:24:01 | sean-k-mooney | i see... | |
| 17:24:15 | sean-k-mooney | that kind of suck since you have to transet all the data via the compute node then | |
| 17:24:35 | dansmith | I just don't understand where that happens | |
| 17:24:48 | geguileo | maybe RBD can... | |
| 17:24:50 | dansmith | if the instance is running but nova is transferring the data between volumes.. where in nova is that happening? | |
| 17:24:58 | geguileo | sean-k-mooney: agreed, it sucks | |
| 17:25:20 | geguileo | dansmith: libvirt/QEMU supports that | |
| 17:25:25 | geguileo | by adding a mirror to the disk | |
| 17:25:38 | geguileo | and once the volumes are mirrored | |
| 17:25:44 | geguileo | nova removes the old volume | |
| 17:25:46 | dansmith | ah, okay and is that what we're poking via swap? | |
| 17:25:56 | geguileo | I think so | |
| 17:26:02 | geguileo | I believe that's the swap volume | |
| 17:26:13 | dansmith | okay I thought our swap was just "pause, change the connection, unpause" | |
| 17:26:30 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#LL2241C16-L2241C16 | |
| 17:26:32 | sean-k-mooney | there i think | |
| 17:27:02 | dansmith | oh okay so it really is us blocking on that action, interesting | |
| 17:27:04 | geguileo | sean-k-mooney: sounds about right | |
| 17:27:27 | sean-k-mooney | i dug into this a few months ago but i have mostly purged that info | |
| 17:27:29 | dansmith | man, that sucks :) | |
| 17:27:43 | sean-k-mooney | its actully libvirt doing this | |
| 17:27:48 | sean-k-mooney | but there are a few flags we can pass | |
| 17:27:54 | geguileo | yeah, Nova just waits for the job to complete | |
| 17:27:56 | sean-k-mooney | to have it sparify the zeors | |
| 17:28:11 | geguileo | yeah, it's the detect_zeroes option from https://paste.openstack.org/show/brFgX6MgBlxjgCrE3rbg/ | |
| 17:28:20 | geguileo | well, that's the brute force approach for my tests | |
| 17:28:27 | geguileo | because that enables them ALL the time | |
| 17:28:30 | geguileo | which sucks | |
| 17:28:41 | geguileo | but I was able to confirm that it works for NFS/qcow2 and RBD | |
| 17:28:48 | geguileo | doesn't work for SCSI devices though | |
| 17:28:50 | sean-k-mooney | well it depens on if there is any perfornace overhead to it normally | |
| 17:29:00 | geguileo | sean-k-mooney: there is performance overhead | |
| 17:29:15 | dansmith | definitely | |
| 17:29:18 | geguileo | so I think it only makes sense when you are going to be reading the whole thing | |
| 17:29:24 | geguileo | and then you save on network + writes | |
| 17:29:54 | geguileo | so only enable it during the volume swap operation | |
| 17:29:55 | sean-k-mooney | well we cant really change this in responce to a volume migration api request | |
| 17:29:56 | geguileo | then disable it | |
| 17:30:00 | dansmith | you're trading cpu for disk | |