| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-06 | |||
| 16:52:45 | bauzas | and thanks gibi for the chair | |
| 16:52:50 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-06-16.00.log.html | |
| 16:52:50 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-06-16.00.txt | |
| 16:52:50 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-06-16.00.html | |
| 16:52:50 | opendevmeet | Meeting ended Tue Jun 6 16:52:50 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:52:50 | bauzas | #endmeeting | |
| 16:53:02 | bauzas | gibi: there you go, you have your logs :p | |
| 16:54:11 | gibi | thank | |
| 16:54:25 | gibi | and it was the moment when my connection dropped first | |
| 16:54:34 | gibi | so it was not that flaky after all | |
| 16:55:13 | dansmith | are we canceling the nova meeting next week? | |
| 16:55:27 | bauzas | dansmith: damn shit, forgot to tell it | |
| 16:55:43 | bauzas | unless people wanna run it, which I'm cool wiht | |
| 16:56:29 | geguileo | dansmith: sean-k-mooney last week I mentioned the os-brick idempotency and we talked about reconstructing the disk XML on some operations. I've just opened a bug (#2023078) where Nova is not rebuilding the disk XML after block migration. Don't know nova enough to know if this would be fixed with the changes to the os-brick idempotency thingy. | |
| 16:57:14 | sean-k-mooney | the xml for live migration is built on the souce node based on info passed back form the destination host | |
| 16:57:22 | sean-k-mooney | for cold migration it build on the dest host | |
| 16:57:52 | sean-k-mooney | i assume you are refering to live-migration local block devices? | |
| 16:57:58 | geguileo | sean-k-mooney: the XML after live migration is wrong | |
| 16:58:18 | sean-k-mooney | that xml must be generated on the source host | |
| 16:58:20 | geguileo | sean-k-mooney: it doesn't have the right discard=unmap value even when the destination is saying that it is supported | |
| 16:58:38 | sean-k-mooney | is it enabeld on the source | |
| 16:59:11 | geguileo | sean-k-mooney: but if the source didn't support it and now it does? Then it cannot be supported until reboot? | |
| 16:59:28 | geguileo | Because then I think Cinder should start reporting that everything supports discard... | |
| 16:59:57 | sean-k-mooney | geguileo: correct | |
| 17:00:12 | sean-k-mooney | so this is not somethign that should be chnaging durign a live migrate | |
| 17:00:31 | geguileo | ok, so then why do we even report the support for this thing? | |
| 17:00:45 | geguileo | We should always set discard=unmap in the XML | |
| 17:01:01 | sean-k-mooney | well we cant because our min libvirt/qemu did not support it | |
| 17:01:05 | geguileo | if it works, nice, if it doesn't it will not prevent it from working after a live migration | |
| 17:01:22 | sean-k-mooney | that may have changed recently but it was a limitation in the past | |
| 17:01:37 | opendevreview | sean mooney proposed openstack/nova master: Allow discard with virtio-blk https://review.opendev.org/c/openstack/nova/+/878795 | |
| 17:01:42 | geguileo | sean-k-mooney: but if it does now then maybe we have to give it another go at this whole thing | |
| 17:01:52 | sean-k-mooney | geguileo: without ^ we also dont support discard for all disk buses | |
| 17:02:18 | sean-k-mooney | geguileo: sure but we have to be careful to ensure that the move ops work properly | |
| 17:02:22 | geguileo | sean-k-mooney: that patch should be abandoned... I added a comment to the LP bug | |
| 17:02:33 | sean-k-mooney | that is not your patch | |
| 17:02:38 | sean-k-mooney | its my patch to fix the bug | |
| 17:02:45 | sean-k-mooney | that for some reason was not in launchpad | |
| 17:02:49 | geguileo | sean-k-mooney: how is it different to mine? | |
| 17:03:02 | geguileo | it's literaly the same | |
| 17:03:06 | geguileo | and it doesn't work | |
| 17:03:07 | sean-k-mooney | i make discard work for disk buses | |
| 17:03:20 | geguileo | it just removes a debug log message | |
| 17:03:28 | geguileo | and the log is correct | |
| 17:03:32 | geguileo | discard doesn't work with virtio | |
| 17:03:37 | geguileo | don't know why | |
| 17:03:37 | sean-k-mooney | yes it does | |
| 17:03:49 | sean-k-mooney | it requires a min version fo qemu and libvirt | |
| 17:04:02 | geguileo | I could only make it work with IDE, SCSI, and SATA | |
| 17:04:29 | sean-k-mooney | i can try and find the downstream bz for virtio-blk again one sec | |
| 17:04:38 | geguileo | sean-k-mooney: I have the downstream BZ | |
| 17:04:45 | geguileo | sean-k-mooney: I'm just telling you I can't make it work | |
| 17:04:47 | sean-k-mooney | for qemu supprot of virtio-blk | |
| 17:04:51 | geguileo | (maybe I'm dumb) | |
| 17:04:54 | sean-k-mooney | *trim with virtio--blk | |
| 17:05:06 | geguileo | sean-k-mooney: sure, it says it supports it... I can't make it work without using IDE, SCSI or SATA | |
| 17:05:08 | sean-k-mooney | well it worked in our ci | |
| 17:05:16 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/879077/1 | |
| 17:05:20 | geguileo | sean-k-mooney: did you actually check that the size was reduced? | |
| 17:05:44 | sean-k-mooney | no but if its not that a qemu bug | |
| 17:05:55 | geguileo | sean-k-mooney: but then the log should remain | |
| 17:06:13 | sean-k-mooney | why it would not be correct | |
| 17:06:41 | geguileo | just because fstrim says it has freed space within the guest OS it doesn't mean that it has actually hapened | |
| 17:07:11 | sean-k-mooney | sure there are several layeres at play here | |
| 17:07:24 | geguileo | sean-k-mooney: oh, I'm sure there are many, but they are beyond my expertise | |
| 17:07:39 | geguileo | I'm just reporting as a storage guy saying, don't know why I can't make it work | |
| 17:07:41 | geguileo | lol | |
| 17:07:53 | sean-k-mooney | so you were using local sotrage | |
| 17:07:55 | sean-k-mooney | no cinder | |
| 17:07:57 | sean-k-mooney | booted a vm | |
| 17:08:02 | sean-k-mooney | allcoated space | |
| 17:08:11 | geguileo | I was booting a VM from RBD, iSCSI, NFS | |
| 17:08:12 | sean-k-mooney | and then deleteed it an ddid a trim | |
| 17:08:45 | geguileo | Then I did a live volume migration, which made the disk loose sparseness (became thick) | |
| 17:08:54 | geguileo | then I issued the "fstrim -v --all" | |
| 17:08:59 | sean-k-mooney | well i filed https://bugs.launchpad.net/nova/+bug/2013123 for local storage | |
| 17:09:17 | geguileo | sean-k-mooney: yeah, I replied in that LP bug | |
| 17:09:45 | geguileo | sean-k-mooney: I'm working on a sparseness document, because this is a CF | |
| 17:09:59 | geguileo | (including the cinder side) | |
| 17:10:11 | sean-k-mooney | well discard is not just about sparceness | |
| 17:10:27 | sean-k-mooney | but for what its worth we do not make any statement about spacencie at the api level | |
| 17:11:22 | dansmith | AFAIK, with various file formats you can only expect the used size to decrease if you fully discard a block that covers an extent | |
| 17:11:23 | geguileo | sean-k-mooney: yeah, it's also about SSDs optimization, power consumption, etc | |
| 17:11:36 | dansmith | vmware, TMK, only actually reclaims space when the guest is shutdown | |
| 17:11:51 | geguileo | dansmith: true, but even then I can clearly see the reduction | |
| 17:12:11 | geguileo | I mean, the size goes down from 1GB to 100MB or so... | |
| 17:12:14 | sean-k-mooney | so for example i dont knwo if qcow or raw files will actully reduce space | |
| 17:12:27 | geguileo | sean-k-mooney: qcow2 does | |
| 17:12:32 | sean-k-mooney | for qcow i woudl expect it to be reduced for raw proably not | |
| 17:12:33 | geguileo | it things are set correctly | |
| 17:12:38 | geguileo | (aka all the starts align) | |
| 17:12:51 | geguileo | s/starts/stars | |
| 17:13:12 | geguileo | and I can even make NFS/qcow2 and RBD preserve sparseness on live migration | |
| 17:13:19 | geguileo | changing the nova code | |
| 17:13:33 | sean-k-mooney | how did you chagne the nova code | |
| 17:13:35 | geguileo | to use the detect_zeroes feature | |
| 17:13:52 | geguileo | I created this LP bug for that one https://bugs.launchpad.net/nova/+bug/2023079 | |
| 17:13:56 | sean-k-mooney | ok then we can add that as a new feature if you can explain what it is | |
| 17:14:06 | sean-k-mooney | that is not a bug it woudl be a new feature | |
| 17:14:19 | sean-k-mooney | a small one but its still a feature | |