| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-06 | |||
| 17:20:28 | geguileo | dansmith: retype is a different think, but many times it triggers a migration | |
| 17:20:40 | geguileo | dansmith: oh, yeah, not a nova live migration | |
| 17:20:45 | geguileo | dansmith: it's a volume live migration | |
| 17:20:57 | sean-k-mooney | retypes can be within hte same backend (just different qos policy) or to a diffent backend | |
| 17:20:58 | geguileo | basically when you mirror the data from one volume to another | |
| 17:21:00 | dansmith | okay, I guess we're confusing too many things | |
| 17:21:10 | geguileo | sean-k-mooney: correct! | |
| 17:21:49 | geguileo | dansmith: what's the right term for moving data from one attached volume to another while the instance is running? | |
| 17:21:49 | sean-k-mooney | so i remmeber a custoemr issue like 4-6 months ago where i tought wew fixed scalio or one of the other driver to explcitly preserve sparcens when doign a driver assited voluem migration | |
| 17:22:04 | dansmith | swap volume I think? | |
| 17:22:14 | geguileo | sean-k-mooney: I fixed for offline and to report the value to Nova | |
| 17:22:17 | dansmith | that's the action we see, AFAIK | |
| 17:22:28 | sean-k-mooney | on the nova side its the swap volume is what is called yes | |
| 17:22:29 | geguileo | dansmith: ok, I'll try to talk about swap volume | |
| 17:22:48 | sean-k-mooney | well no volume migration is fine | |
| 17:23:05 | dansmith | geguileo: not trying to make you use our language, I just need to know that a bunch of terms have been re-used :) | |
| 17:23:19 | sean-k-mooney | but you are asserting that the driver asseited path never works for an online volume migration | |
| 17:23:22 | geguileo | dansmith: oh, I prefer to use the right language | |
| 17:23:41 | geguileo | sean-k-mooney: I don't think we have any cinder driver capable of doing it... | |
| 17:24:01 | sean-k-mooney | i see... | |
| 17:24:15 | sean-k-mooney | that kind of suck since you have to transet all the data via the compute node then | |
| 17:24:35 | dansmith | I just don't understand where that happens | |
| 17:24:48 | geguileo | maybe RBD can... | |
| 17:24:50 | dansmith | if the instance is running but nova is transferring the data between volumes.. where in nova is that happening? | |
| 17:24:58 | geguileo | sean-k-mooney: agreed, it sucks | |
| 17:25:20 | geguileo | dansmith: libvirt/QEMU supports that | |
| 17:25:25 | geguileo | by adding a mirror to the disk | |
| 17:25:38 | geguileo | and once the volumes are mirrored | |
| 17:25:44 | geguileo | nova removes the old volume | |
| 17:25:46 | dansmith | ah, okay and is that what we're poking via swap? | |
| 17:25:56 | geguileo | I think so | |
| 17:26:02 | geguileo | I believe that's the swap volume | |
| 17:26:13 | dansmith | okay I thought our swap was just "pause, change the connection, unpause" | |
| 17:26:30 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#LL2241C16-L2241C16 | |
| 17:26:32 | sean-k-mooney | there i think | |
| 17:27:02 | dansmith | oh okay so it really is us blocking on that action, interesting | |
| 17:27:04 | geguileo | sean-k-mooney: sounds about right | |
| 17:27:27 | sean-k-mooney | i dug into this a few months ago but i have mostly purged that info | |
| 17:27:29 | dansmith | man, that sucks :) | |
| 17:27:43 | sean-k-mooney | its actully libvirt doing this | |
| 17:27:48 | sean-k-mooney | but there are a few flags we can pass | |
| 17:27:54 | geguileo | yeah, Nova just waits for the job to complete | |
| 17:27:56 | sean-k-mooney | to have it sparify the zeors | |
| 17:28:11 | geguileo | yeah, it's the detect_zeroes option from https://paste.openstack.org/show/brFgX6MgBlxjgCrE3rbg/ | |
| 17:28:20 | geguileo | well, that's the brute force approach for my tests | |
| 17:28:27 | geguileo | because that enables them ALL the time | |
| 17:28:30 | geguileo | which sucks | |
| 17:28:41 | geguileo | but I was able to confirm that it works for NFS/qcow2 and RBD | |
| 17:28:48 | geguileo | doesn't work for SCSI devices though | |
| 17:28:50 | sean-k-mooney | well it depens on if there is any perfornace overhead to it normally | |
| 17:29:00 | geguileo | sean-k-mooney: there is performance overhead | |
| 17:29:15 | dansmith | definitely | |
| 17:29:18 | geguileo | so I think it only makes sense when you are going to be reading the whole thing | |
| 17:29:24 | geguileo | and then you save on network + writes | |
| 17:29:54 | geguileo | so only enable it during the volume swap operation | |
| 17:29:55 | sean-k-mooney | well we cant really change this in responce to a volume migration api request | |
| 17:29:56 | geguileo | then disable it | |
| 17:30:00 | dansmith | you're trading cpu for disk | |
| 17:30:22 | sean-k-mooney | even if we coudl im not sure if we should | |
| 17:30:29 | geguileo | dansmith: in volume swap, we are trading cpu for network + disk + time | |
| 17:30:47 | dansmith | yeah I meant if it's enabled all the time | |
| 17:31:04 | dansmith | and yeah, network too | |
| 17:31:17 | sean-k-mooney | well it would only have an effect on livemigration and on the intal disk creation | |
| 17:31:23 | geguileo | dansmith: I don't think we should enable it all the time, because if the storage supports discard, then the disk will be recovered by periodically calling fstrimg like some OSs do | |
| 17:31:35 | dansmith | yeah | |
| 17:31:54 | geguileo | but I think this can greatly improve some volume swap cases | |
| 17:31:59 | sean-k-mooney | well discard is off by default | |
| 17:32:12 | sean-k-mooney | and currently only works if you use virtio-scsi which is not our default | |
| 17:32:23 | geguileo | sean-k-mooney: yeah, but that's something we have to improve on the cinder side | |
| 17:32:32 | sean-k-mooney | no i mean in the nova side | |
| 17:32:33 | geguileo | so we properly report the value | |
| 17:32:43 | sean-k-mooney | we have a config option to opt into allowing discard | |
| 17:32:47 | sean-k-mooney | and by default we dont | |
| 17:32:48 | geguileo | yeah, but if cinder reports it is supported then nova does the right thing | |
| 17:32:54 | geguileo | sean-k-mooney: really? | |
| 17:33:06 | geguileo | sean-k-mooney: which one? because I don't recall touching nova conf to enable that one | |
| 17:33:12 | geguileo | (maybe devstack does automatically) | |
| 17:33:17 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.hw_disk_discard | |
| 17:34:01 | sean-k-mooney | so that contols if discard works for local disk at least | |
| 17:34:14 | geguileo | sean-k-mooney: oh, but for local, not cinder | |
| 17:34:23 | geguileo | good to know about that one | |
| 17:34:24 | sean-k-mooney | i was under the impression it had an effect for cinder too but im not sure | |
| 17:34:36 | sean-k-mooney | geguileo: so the reason the discard beahvior came to my attention | |
| 17:34:40 | geguileo | sean-k-mooney: I don't think it does | |
| 17:34:44 | sean-k-mooney | was i wanted to make discard the defautl for nova | |
| 17:34:49 | sean-k-mooney | and found it broke virtio-blk | |
| 17:34:52 | geguileo | sean-k-mooney: that would be awesome!!! | |
| 17:34:54 | sean-k-mooney | so i fixed that | |
| 17:35:10 | geguileo | what did you fix? | |
| 17:35:28 | sean-k-mooney | i removed the block based on the qemu min version | |
| 17:35:32 | sean-k-mooney | and i coudl boot vms | |
| 17:35:41 | sean-k-mooney | so what i need to do is repoduce this locally again | |
| 17:35:46 | sean-k-mooney | and do some manual testing | |
| 17:36:00 | sean-k-mooney | to confirm if discard with qcow acturally works | |
| 17:36:06 | geguileo | sean-k-mooney: LVM with LIO doesn't currently support trimming | |
| 17:36:17 | geguileo | it's one of the bugs I have a local fix for | |
| 17:36:30 | sean-k-mooney | ok but that wont affect things right | |
| 17:36:37 | geguileo | so you may see fstrim telling you it has recovered space, but it hasn't really | |
| 17:36:37 | sean-k-mooney | since that driver wont report discard supprot | |
| 17:36:51 | geguileo | sean-k-mooney: we can ask cinder drivers to report discard support | |