| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-06 | |||
| 17:18:41 | geguileo | sean-k-mooney: afaik today all online volume migrations are done by nova | |
| 17:18:51 | dansmith | wait what? | |
| 17:18:54 | geguileo | I don't think any cinde driver supports it | |
| 17:19:00 | opendevreview | Amit Uniyal proposed openstack/nova master: Reproducer for dangling bdms https://review.opendev.org/c/openstack/nova/+/881457 | |
| 17:19:00 | opendevreview | Amit Uniyal proposed openstack/nova master: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284 | |
| 17:19:13 | sean-k-mooney | geguileo: i was pretty sure you fixed at least one vendor driver last year | |
| 17:19:13 | dansmith | why would we ever want to do that? unless you're crossing AZs or something | |
| 17:19:14 | geguileo | dansmith: all volumes that are attached to now are migrated by nova | |
| 17:19:27 | geguileo | s/now/nova | |
| 17:19:32 | sean-k-mooney | geguileo: if you change backend right | |
| 17:19:38 | sean-k-mooney | not just retype | |
| 17:19:45 | sean-k-mooney | within the same backend | |
| 17:19:59 | dansmith | retype is a different thing right? | |
| 17:20:16 | sean-k-mooney | yes | |
| 17:20:20 | dansmith | I'm talking about you live migrate from one server to the next one in the rack, we should not be moving all the volume data to a *new* volume... | |
| 17:20:28 | geguileo | dansmith: retype is a different think, but many times it triggers a migration | |
| 17:20:40 | geguileo | dansmith: oh, yeah, not a nova live migration | |
| 17:20:45 | geguileo | dansmith: it's a volume live migration | |
| 17:20:57 | sean-k-mooney | retypes can be within hte same backend (just different qos policy) or to a diffent backend | |
| 17:20:58 | geguileo | basically when you mirror the data from one volume to another | |
| 17:21:00 | dansmith | okay, I guess we're confusing too many things | |
| 17:21:10 | geguileo | sean-k-mooney: correct! | |
| 17:21:49 | sean-k-mooney | so i remmeber a custoemr issue like 4-6 months ago where i tought wew fixed scalio or one of the other driver to explcitly preserve sparcens when doign a driver assited voluem migration | |
| 17:21:49 | geguileo | dansmith: what's the right term for moving data from one attached volume to another while the instance is running? | |
| 17:22:04 | dansmith | swap volume I think? | |
| 17:22:14 | geguileo | sean-k-mooney: I fixed for offline and to report the value to Nova | |
| 17:22:17 | dansmith | that's the action we see, AFAIK | |
| 17:22:28 | sean-k-mooney | on the nova side its the swap volume is what is called yes | |
| 17:22:29 | geguileo | dansmith: ok, I'll try to talk about swap volume | |
| 17:22:48 | sean-k-mooney | well no volume migration is fine | |
| 17:23:05 | dansmith | geguileo: not trying to make you use our language, I just need to know that a bunch of terms have been re-used :) | |
| 17:23:19 | sean-k-mooney | but you are asserting that the driver asseited path never works for an online volume migration | |
| 17:23:22 | geguileo | dansmith: oh, I prefer to use the right language | |
| 17:23:41 | geguileo | sean-k-mooney: I don't think we have any cinder driver capable of doing it... | |
| 17:24:01 | sean-k-mooney | i see... | |
| 17:24:15 | sean-k-mooney | that kind of suck since you have to transet all the data via the compute node then | |
| 17:24:35 | dansmith | I just don't understand where that happens | |
| 17:24:48 | geguileo | maybe RBD can... | |
| 17:24:50 | dansmith | if the instance is running but nova is transferring the data between volumes.. where in nova is that happening? | |
| 17:24:58 | geguileo | sean-k-mooney: agreed, it sucks | |
| 17:25:20 | geguileo | dansmith: libvirt/QEMU supports that | |
| 17:25:25 | geguileo | by adding a mirror to the disk | |
| 17:25:38 | geguileo | and once the volumes are mirrored | |
| 17:25:44 | geguileo | nova removes the old volume | |
| 17:25:46 | dansmith | ah, okay and is that what we're poking via swap? | |
| 17:25:56 | geguileo | I think so | |
| 17:26:02 | geguileo | I believe that's the swap volume | |
| 17:26:13 | dansmith | okay I thought our swap was just "pause, change the connection, unpause" | |
| 17:26:30 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#LL2241C16-L2241C16 | |
| 17:26:32 | sean-k-mooney | there i think | |
| 17:27:02 | dansmith | oh okay so it really is us blocking on that action, interesting | |
| 17:27:04 | geguileo | sean-k-mooney: sounds about right | |
| 17:27:27 | sean-k-mooney | i dug into this a few months ago but i have mostly purged that info | |
| 17:27:29 | dansmith | man, that sucks :) | |
| 17:27:43 | sean-k-mooney | its actully libvirt doing this | |
| 17:27:48 | sean-k-mooney | but there are a few flags we can pass | |
| 17:27:54 | geguileo | yeah, Nova just waits for the job to complete | |
| 17:27:56 | sean-k-mooney | to have it sparify the zeors | |
| 17:28:11 | geguileo | yeah, it's the detect_zeroes option from https://paste.openstack.org/show/brFgX6MgBlxjgCrE3rbg/ | |
| 17:28:20 | geguileo | well, that's the brute force approach for my tests | |
| 17:28:27 | geguileo | because that enables them ALL the time | |
| 17:28:30 | geguileo | which sucks | |
| 17:28:41 | geguileo | but I was able to confirm that it works for NFS/qcow2 and RBD | |
| 17:28:48 | geguileo | doesn't work for SCSI devices though | |
| 17:28:50 | sean-k-mooney | well it depens on if there is any perfornace overhead to it normally | |
| 17:29:00 | geguileo | sean-k-mooney: there is performance overhead | |
| 17:29:15 | dansmith | definitely | |
| 17:29:18 | geguileo | so I think it only makes sense when you are going to be reading the whole thing | |
| 17:29:24 | geguileo | and then you save on network + writes | |
| 17:29:54 | geguileo | so only enable it during the volume swap operation | |
| 17:29:55 | sean-k-mooney | well we cant really change this in responce to a volume migration api request | |
| 17:29:56 | geguileo | then disable it | |
| 17:30:00 | dansmith | you're trading cpu for disk | |
| 17:30:22 | sean-k-mooney | even if we coudl im not sure if we should | |
| 17:30:29 | geguileo | dansmith: in volume swap, we are trading cpu for network + disk + time | |
| 17:30:47 | dansmith | yeah I meant if it's enabled all the time | |
| 17:31:04 | dansmith | and yeah, network too | |
| 17:31:17 | sean-k-mooney | well it would only have an effect on livemigration and on the intal disk creation | |
| 17:31:23 | geguileo | dansmith: I don't think we should enable it all the time, because if the storage supports discard, then the disk will be recovered by periodically calling fstrimg like some OSs do | |
| 17:31:35 | dansmith | yeah | |
| 17:31:54 | geguileo | but I think this can greatly improve some volume swap cases | |
| 17:31:59 | sean-k-mooney | well discard is off by default | |
| 17:32:12 | sean-k-mooney | and currently only works if you use virtio-scsi which is not our default | |
| 17:32:23 | geguileo | sean-k-mooney: yeah, but that's something we have to improve on the cinder side | |
| 17:32:32 | sean-k-mooney | no i mean in the nova side | |
| 17:32:33 | geguileo | so we properly report the value | |
| 17:32:43 | sean-k-mooney | we have a config option to opt into allowing discard | |
| 17:32:47 | sean-k-mooney | and by default we dont | |
| 17:32:48 | geguileo | yeah, but if cinder reports it is supported then nova does the right thing | |
| 17:32:54 | geguileo | sean-k-mooney: really? | |
| 17:33:06 | geguileo | sean-k-mooney: which one? because I don't recall touching nova conf to enable that one | |
| 17:33:12 | geguileo | (maybe devstack does automatically) | |
| 17:33:17 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.hw_disk_discard | |
| 17:34:01 | sean-k-mooney | so that contols if discard works for local disk at least | |
| 17:34:14 | geguileo | sean-k-mooney: oh, but for local, not cinder | |
| 17:34:23 | geguileo | good to know about that one | |
| 17:34:24 | sean-k-mooney | i was under the impression it had an effect for cinder too but im not sure | |
| 17:34:36 | sean-k-mooney | geguileo: so the reason the discard beahvior came to my attention | |
| 17:34:40 | geguileo | sean-k-mooney: I don't think it does | |
| 17:34:44 | sean-k-mooney | was i wanted to make discard the defautl for nova | |