| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-06 | |||
| 17:05:20 | geguileo | sean-k-mooney: did you actually check that the size was reduced? | |
| 17:05:44 | sean-k-mooney | no but if its not that a qemu bug | |
| 17:05:55 | geguileo | sean-k-mooney: but then the log should remain | |
| 17:06:13 | sean-k-mooney | why it would not be correct | |
| 17:06:41 | geguileo | just because fstrim says it has freed space within the guest OS it doesn't mean that it has actually hapened | |
| 17:07:11 | sean-k-mooney | sure there are several layeres at play here | |
| 17:07:24 | geguileo | sean-k-mooney: oh, I'm sure there are many, but they are beyond my expertise | |
| 17:07:39 | geguileo | I'm just reporting as a storage guy saying, don't know why I can't make it work | |
| 17:07:41 | geguileo | lol | |
| 17:07:53 | sean-k-mooney | so you were using local sotrage | |
| 17:07:55 | sean-k-mooney | no cinder | |
| 17:07:57 | sean-k-mooney | booted a vm | |
| 17:08:02 | sean-k-mooney | allcoated space | |
| 17:08:11 | geguileo | I was booting a VM from RBD, iSCSI, NFS | |
| 17:08:12 | sean-k-mooney | and then deleteed it an ddid a trim | |
| 17:08:45 | geguileo | Then I did a live volume migration, which made the disk loose sparseness (became thick) | |
| 17:08:54 | geguileo | then I issued the "fstrim -v --all" | |
| 17:08:59 | sean-k-mooney | well i filed https://bugs.launchpad.net/nova/+bug/2013123 for local storage | |
| 17:09:17 | geguileo | sean-k-mooney: yeah, I replied in that LP bug | |
| 17:09:45 | geguileo | sean-k-mooney: I'm working on a sparseness document, because this is a CF | |
| 17:09:59 | geguileo | (including the cinder side) | |
| 17:10:11 | sean-k-mooney | well discard is not just about sparceness | |
| 17:10:27 | sean-k-mooney | but for what its worth we do not make any statement about spacencie at the api level | |
| 17:11:22 | dansmith | AFAIK, with various file formats you can only expect the used size to decrease if you fully discard a block that covers an extent | |
| 17:11:23 | geguileo | sean-k-mooney: yeah, it's also about SSDs optimization, power consumption, etc | |
| 17:11:36 | dansmith | vmware, TMK, only actually reclaims space when the guest is shutdown | |
| 17:11:51 | geguileo | dansmith: true, but even then I can clearly see the reduction | |
| 17:12:11 | geguileo | I mean, the size goes down from 1GB to 100MB or so... | |
| 17:12:14 | sean-k-mooney | so for example i dont knwo if qcow or raw files will actully reduce space | |
| 17:12:27 | geguileo | sean-k-mooney: qcow2 does | |
| 17:12:32 | sean-k-mooney | for qcow i woudl expect it to be reduced for raw proably not | |
| 17:12:33 | geguileo | it things are set correctly | |
| 17:12:38 | geguileo | (aka all the starts align) | |
| 17:12:51 | geguileo | s/starts/stars | |
| 17:13:12 | geguileo | and I can even make NFS/qcow2 and RBD preserve sparseness on live migration | |
| 17:13:19 | geguileo | changing the nova code | |
| 17:13:33 | sean-k-mooney | how did you chagne the nova code | |
| 17:13:35 | geguileo | to use the detect_zeroes feature | |
| 17:13:52 | geguileo | I created this LP bug for that one https://bugs.launchpad.net/nova/+bug/2023079 | |
| 17:13:56 | sean-k-mooney | ok then we can add that as a new feature if you can explain what it is | |
| 17:14:06 | sean-k-mooney | that is not a bug it woudl be a new feature | |
| 17:14:19 | sean-k-mooney | a small one but its still a feature | |
| 17:14:39 | geguileo | sean-k-mooney: https://paste.openstack.org/show/brFgX6MgBlxjgCrE3rbg/ | |
| 17:14:52 | sean-k-mooney | geguileo: what do you mean by volume in https://bugs.launchpad.net/nova/+bug/2023079 | |
| 17:14:55 | geguileo | sean-k-mooney: but that's not the right code | |
| 17:14:59 | geguileo | it's what I used to test | |
| 17:15:27 | sean-k-mooney | oh "When doing a live volume migration " | |
| 17:15:32 | geguileo | but that has CPU implications when running, so it's best only to change it when the block migration is going to happen | |
| 17:15:44 | sean-k-mooney | geguileo: we have a differnt concept called block-migration in the context of live migrating a vm | |
| 17:15:53 | dansmith | ...yeah | |
| 17:16:03 | geguileo | sean-k-mooney: yes, I mean block live migration, I've updated the bug name, thanks | |
| 17:16:04 | dansmith | I was going to say, we should never be block migrating a volume | |
| 17:16:25 | geguileo | sean-k-mooney: ooooh, then soooooooorry for mixing terms (/me facepalms) | |
| 17:16:27 | sean-k-mooney | geguileo: block live migratin in nova mean live migrate a vm with local raw/qcow storage | |
| 17:16:43 | dansmith | it means we literally move all the data | |
| 17:16:56 | dansmith | (all the disk data) | |
| 17:17:03 | geguileo | my bad, I've updated the LP bug | |
| 17:17:10 | geguileo | dansmith: we are currently moving ALL the data | |
| 17:17:23 | sean-k-mooney | geguileo: ya so i know knwo what your trying to fix | |
| 17:17:27 | geguileo | that's why the detect_zeroes would be good for volume live migration | |
| 17:17:43 | geguileo | sean-k-mooney: thanks for you patience in understanding my ramblings :-) | |
| 17:17:43 | sean-k-mooney | so we expect this to be done by cider using the driver assited migration feature | |
| 17:17:57 | sean-k-mooney | however you want nova ot be intelegent enough | |
| 17:18:07 | sean-k-mooney | so that when we fallback to nova doing the volume migration | |
| 17:18:15 | sean-k-mooney | that we also preseve the sparceness | |
| 17:18:23 | geguileo | sean-k-mooney: problem is that driver assisted migration cannot work between different backends (and afaik it doesn't work for any driver even between volumes of the same array) | |
| 17:18:41 | geguileo | sean-k-mooney: afaik today all online volume migrations are done by nova | |
| 17:18:51 | dansmith | wait what? | |
| 17:18:54 | geguileo | I don't think any cinde driver supports it | |
| 17:19:00 | opendevreview | Amit Uniyal proposed openstack/nova master: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284 | |
| 17:19:00 | opendevreview | Amit Uniyal proposed openstack/nova master: Reproducer for dangling bdms https://review.opendev.org/c/openstack/nova/+/881457 | |
| 17:19:13 | dansmith | why would we ever want to do that? unless you're crossing AZs or something | |
| 17:19:13 | sean-k-mooney | geguileo: i was pretty sure you fixed at least one vendor driver last year | |
| 17:19:14 | geguileo | dansmith: all volumes that are attached to now are migrated by nova | |
| 17:19:27 | geguileo | s/now/nova | |
| 17:19:32 | sean-k-mooney | geguileo: if you change backend right | |
| 17:19:38 | sean-k-mooney | not just retype | |
| 17:19:45 | sean-k-mooney | within the same backend | |
| 17:19:59 | dansmith | retype is a different thing right? | |
| 17:20:16 | sean-k-mooney | yes | |
| 17:20:20 | dansmith | I'm talking about you live migrate from one server to the next one in the rack, we should not be moving all the volume data to a *new* volume... | |
| 17:20:28 | geguileo | dansmith: retype is a different think, but many times it triggers a migration | |
| 17:20:40 | geguileo | dansmith: oh, yeah, not a nova live migration | |
| 17:20:45 | geguileo | dansmith: it's a volume live migration | |
| 17:20:57 | sean-k-mooney | retypes can be within hte same backend (just different qos policy) or to a diffent backend | |
| 17:20:58 | geguileo | basically when you mirror the data from one volume to another | |
| 17:21:00 | dansmith | okay, I guess we're confusing too many things | |
| 17:21:10 | geguileo | sean-k-mooney: correct! | |
| 17:21:49 | geguileo | dansmith: what's the right term for moving data from one attached volume to another while the instance is running? | |
| 17:21:49 | sean-k-mooney | so i remmeber a custoemr issue like 4-6 months ago where i tought wew fixed scalio or one of the other driver to explcitly preserve sparcens when doign a driver assited voluem migration | |
| 17:22:04 | dansmith | swap volume I think? | |
| 17:22:14 | geguileo | sean-k-mooney: I fixed for offline and to report the value to Nova | |
| 17:22:17 | dansmith | that's the action we see, AFAIK | |
| 17:22:28 | sean-k-mooney | on the nova side its the swap volume is what is called yes | |
| 17:22:29 | geguileo | dansmith: ok, I'll try to talk about swap volume | |
| 17:22:48 | sean-k-mooney | well no volume migration is fine | |
| 17:23:05 | dansmith | geguileo: not trying to make you use our language, I just need to know that a bunch of terms have been re-used :) | |
| 17:23:19 | sean-k-mooney | but you are asserting that the driver asseited path never works for an online volume migration | |
| 17:23:22 | geguileo | dansmith: oh, I prefer to use the right language | |
| 17:23:41 | geguileo | sean-k-mooney: I don't think we have any cinder driver capable of doing it... | |