| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-06 | |||
| 17:23:05 | dansmith | geguileo: not trying to make you use our language, I just need to know that a bunch of terms have been re-used :) | |
| 17:23:19 | sean-k-mooney | but you are asserting that the driver asseited path never works for an online volume migration | |
| 17:23:22 | geguileo | dansmith: oh, I prefer to use the right language | |
| 17:23:41 | geguileo | sean-k-mooney: I don't think we have any cinder driver capable of doing it... | |
| 17:24:01 | sean-k-mooney | i see... | |
| 17:24:15 | sean-k-mooney | that kind of suck since you have to transet all the data via the compute node then | |
| 17:24:35 | dansmith | I just don't understand where that happens | |
| 17:24:48 | geguileo | maybe RBD can... | |
| 17:24:50 | dansmith | if the instance is running but nova is transferring the data between volumes.. where in nova is that happening? | |
| 17:24:58 | geguileo | sean-k-mooney: agreed, it sucks | |
| 17:25:20 | geguileo | dansmith: libvirt/QEMU supports that | |
| 17:25:25 | geguileo | by adding a mirror to the disk | |
| 17:25:38 | geguileo | and once the volumes are mirrored | |
| 17:25:44 | geguileo | nova removes the old volume | |
| 17:25:46 | dansmith | ah, okay and is that what we're poking via swap? | |
| 17:25:56 | geguileo | I think so | |
| 17:26:02 | geguileo | I believe that's the swap volume | |
| 17:26:13 | dansmith | okay I thought our swap was just "pause, change the connection, unpause" | |
| 17:26:30 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#LL2241C16-L2241C16 | |
| 17:26:32 | sean-k-mooney | there i think | |
| 17:27:02 | dansmith | oh okay so it really is us blocking on that action, interesting | |
| 17:27:04 | geguileo | sean-k-mooney: sounds about right | |
| 17:27:27 | sean-k-mooney | i dug into this a few months ago but i have mostly purged that info | |
| 17:27:29 | dansmith | man, that sucks :) | |
| 17:27:43 | sean-k-mooney | its actully libvirt doing this | |
| 17:27:48 | sean-k-mooney | but there are a few flags we can pass | |
| 17:27:54 | geguileo | yeah, Nova just waits for the job to complete | |
| 17:27:56 | sean-k-mooney | to have it sparify the zeors | |
| 17:28:11 | geguileo | yeah, it's the detect_zeroes option from https://paste.openstack.org/show/brFgX6MgBlxjgCrE3rbg/ | |
| 17:28:20 | geguileo | well, that's the brute force approach for my tests | |
| 17:28:27 | geguileo | because that enables them ALL the time | |
| 17:28:30 | geguileo | which sucks | |
| 17:28:41 | geguileo | but I was able to confirm that it works for NFS/qcow2 and RBD | |
| 17:28:48 | geguileo | doesn't work for SCSI devices though | |
| 17:28:50 | sean-k-mooney | well it depens on if there is any perfornace overhead to it normally | |
| 17:29:00 | geguileo | sean-k-mooney: there is performance overhead | |
| 17:29:15 | dansmith | definitely | |
| 17:29:18 | geguileo | so I think it only makes sense when you are going to be reading the whole thing | |
| 17:29:24 | geguileo | and then you save on network + writes | |
| 17:29:54 | geguileo | so only enable it during the volume swap operation | |
| 17:29:55 | sean-k-mooney | well we cant really change this in responce to a volume migration api request | |
| 17:29:56 | geguileo | then disable it | |
| 17:30:00 | dansmith | you're trading cpu for disk | |
| 17:30:22 | sean-k-mooney | even if we coudl im not sure if we should | |
| 17:30:29 | geguileo | dansmith: in volume swap, we are trading cpu for network + disk + time | |
| 17:30:47 | dansmith | yeah I meant if it's enabled all the time | |
| 17:31:04 | dansmith | and yeah, network too | |
| 17:31:17 | sean-k-mooney | well it would only have an effect on livemigration and on the intal disk creation | |
| 17:31:23 | geguileo | dansmith: I don't think we should enable it all the time, because if the storage supports discard, then the disk will be recovered by periodically calling fstrimg like some OSs do | |
| 17:31:35 | dansmith | yeah | |
| 17:31:54 | geguileo | but I think this can greatly improve some volume swap cases | |
| 17:31:59 | sean-k-mooney | well discard is off by default | |
| 17:32:12 | sean-k-mooney | and currently only works if you use virtio-scsi which is not our default | |
| 17:32:23 | geguileo | sean-k-mooney: yeah, but that's something we have to improve on the cinder side | |
| 17:32:32 | sean-k-mooney | no i mean in the nova side | |
| 17:32:33 | geguileo | so we properly report the value | |
| 17:32:43 | sean-k-mooney | we have a config option to opt into allowing discard | |
| 17:32:47 | sean-k-mooney | and by default we dont | |
| 17:32:48 | geguileo | yeah, but if cinder reports it is supported then nova does the right thing | |
| 17:32:54 | geguileo | sean-k-mooney: really? | |
| 17:33:06 | geguileo | sean-k-mooney: which one? because I don't recall touching nova conf to enable that one | |
| 17:33:12 | geguileo | (maybe devstack does automatically) | |
| 17:33:17 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.hw_disk_discard | |
| 17:34:01 | sean-k-mooney | so that contols if discard works for local disk at least | |
| 17:34:14 | geguileo | sean-k-mooney: oh, but for local, not cinder | |
| 17:34:23 | geguileo | good to know about that one | |
| 17:34:24 | sean-k-mooney | i was under the impression it had an effect for cinder too but im not sure | |
| 17:34:36 | sean-k-mooney | geguileo: so the reason the discard beahvior came to my attention | |
| 17:34:40 | geguileo | sean-k-mooney: I don't think it does | |
| 17:34:44 | sean-k-mooney | was i wanted to make discard the defautl for nova | |
| 17:34:49 | sean-k-mooney | and found it broke virtio-blk | |
| 17:34:52 | geguileo | sean-k-mooney: that would be awesome!!! | |
| 17:34:54 | sean-k-mooney | so i fixed that | |
| 17:35:10 | geguileo | what did you fix? | |
| 17:35:28 | sean-k-mooney | i removed the block based on the qemu min version | |
| 17:35:32 | sean-k-mooney | and i coudl boot vms | |
| 17:35:41 | sean-k-mooney | so what i need to do is repoduce this locally again | |
| 17:35:46 | sean-k-mooney | and do some manual testing | |
| 17:36:00 | sean-k-mooney | to confirm if discard with qcow acturally works | |
| 17:36:06 | geguileo | sean-k-mooney: LVM with LIO doesn't currently support trimming | |
| 17:36:17 | geguileo | it's one of the bugs I have a local fix for | |
| 17:36:30 | sean-k-mooney | ok but that wont affect things right | |
| 17:36:37 | sean-k-mooney | since that driver wont report discard supprot | |
| 17:36:37 | geguileo | so you may see fstrim telling you it has recovered space, but it hasn't really | |
| 17:36:51 | geguileo | sean-k-mooney: we can ask cinder drivers to report discard support | |
| 17:37:06 | sean-k-mooney | well again i dont care about the cinder case im trying to fix discard support for non-cinder storage | |
| 17:37:20 | geguileo | using the report_discard_supported backend option | |
| 17:37:39 | geguileo | sean-k-mooney: I was digging into the discard case for cinder lol | |
| 17:37:52 | geguileo | and forgot to look into the ephemeral case | |
| 17:37:52 | sean-k-mooney | yep i know :) | |
| 17:38:07 | sean-k-mooney | ephmeral means somethign esle in nova | |
| 17:38:16 | geguileo | sean-k-mooney: I'm doing a writeup on my findings, so I'll send you the link later so you can add yours as well | |
| 17:38:24 | sean-k-mooney | acl | |
| 17:38:27 | sean-k-mooney | *ack | |
| 17:41:09 | geguileo | sean-k-mooney: dansmith thank you both for your time :-) | |
| 17:41:23 | dansmith | same :) | |
| 18:08:01 | carloss | o/ bauzas - what are your thoughts on having a follow-up cross-project session between Nova and Manila in the PTG next week? we can use it to chat about gouthamr's specs | |
| 20:07:49 | opendevreview | Merged openstack/nova master: Add debug logging when Instance raises OrphanedObjectError https://review.opendev.org/c/openstack/nova/+/883325 | |
| #openstack-nova - 2023-06-07 | |||
| 06:08:18 | opendevreview | Amit Uniyal proposed openstack/nova master: Reproducer for dangling bdms https://review.opendev.org/c/openstack/nova/+/881457 | |
| 06:08:18 | opendevreview | Amit Uniyal proposed openstack/nova master: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284 | |