| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-17 | |||
| 11:08:36 | gmann | yea | |
| 11:08:45 | gmann | kashyap: sure take your time. | |
| 11:09:07 | mdbooth | gmann: That could be tough | |
| 11:10:14 | mdbooth | gmann: I think qemu would have to do explicit zero detection during the rebase operation | |
| 11:11:13 | mdbooth | gmann: When we do this for ephemeral storage, we get sparseness from metadata | |
| 11:11:57 | mdbooth | kashyap: Do you know qemu supports doing a block rebase with explicit zero detection? | |
| 11:12:06 | mdbooth | And if libvirt exposes that? | |
| 11:12:06 | lyarwood | gmann: which volumebackend is this with btw? | |
| 11:12:25 | kashyap | mdbooth: I saw some 'detect zeroes' stuff fly by on the QEMU list. Don't recall top off my head, need some digging | |
| 11:16:24 | gmann | mdbooth: kashyap ok. please let us know if something going to be supported or already there. it will really help | |
| 11:16:33 | gmann | lyarwood: lvm | |
| 11:17:01 | mdbooth | gmann: So, for ephemeral storage block migration with qcow2 we use VIR_MIGRATE_NON_SHARED_INC | |
| 11:17:31 | mdbooth | This basically means we copy the backing file | |
| 11:17:52 | mdbooth | Then the live migration copies just the 'overlay' | |
| 11:18:20 | mdbooth | IOW qemu is relying on qcow2 metadata | |
| 11:19:15 | gmann | k | |
| 11:19:22 | mdbooth | For volume copy we don't have this | |
| 11:20:31 | gmann | can it be done with VIR_DOMAIN_BLOCK_REBASE_SHALLOW when same backing file (not sure it can) ? | |
| 11:20:47 | mdbooth | https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockRebaseFlags | |
| 11:20:56 | mdbooth | gmann: There's no backing file for bolumes | |
| 11:21:00 | mdbooth | volumes | |
| 11:21:42 | mdbooth | gmann: It doesn't look to me like any of the above options support zero-detection during copy | |
| 11:21:51 | gmann | humm | |
| 11:21:54 | gmann | yea | |
| 11:22:30 | mdbooth | gmann: I'd be moderately confident that this can't be done right now | |
| 11:22:47 | kashyap | mdbooth: gmann: I was quickly scanning QEMU block schema | |
| 11:23:02 | kashyap | I see a 'detect_zeroes' option that could be specificed when _configuring_ a block device | |
| 11:23:31 | mdbooth | Incidentally, what *could* be done is this: | |
| 11:23:50 | kashyap | And a QEMU block developer confirms: If an area is allocated nowhere in the backing chain, block-stream won't allocate it | |
| 11:24:21 | mdbooth | 1. cinder tells nova to make an atomic local snapshot of the volume | |
| 11:24:32 | mdbooth | 2. Nova makes atomic snapshot and starts writing to snapshot | |
| 11:24:51 | mdbooth | 3. Cinder does fast backend-specific copy of volume from A to B | |
| 11:25:20 | mdbooth | 4. Cinder tells Nova to rebase its snapshot on to B, which is identical to A at the point Nova snapshotted it | |
| 11:25:31 | mdbooth | 5. Nova commits its snapshot to B | |
| 11:25:36 | mdbooth | 6. Profit | |
| 11:26:07 | mdbooth | That's a way different control flow, though | |
| 11:27:48 | mdbooth | kashyap: That's qcow2 thought, right? | |
| 11:28:02 | mdbooth | though | |
| 11:28:10 | gmann | mdbooth: humm, it is on what johnthetubaguy proposal on cinder nova flow ? | |
| 11:28:14 | kashyap | mdbooth: Yep. Still talking to Max. Let me paste my conv. snippet | |
| 11:28:32 | mdbooth | kashyap: So this is a cinder volume: lvm exported over iscsi | |
| 11:28:39 | mdbooth | qemu just sees a single, flat block device | |
| 11:29:06 | johnthetubaguy | that swap volume stuff has to follow after multi-attach I feel | |
| 11:29:24 | mdbooth | johnthetubaguy: Yep. | |
| 11:29:35 | johnthetubaguy | cool, sounds like we are on the same page there then | |
| 11:29:42 | kashyap | mdbooth: Noted | |
| 11:29:54 | kashyap | So, here we go, 8 lines of paste: | |
| 11:29:55 | kashyap | 12:21 < mreitz> Well, if an area is allocated nowhere in the backing chain, block-stream won't allocate it | |
| 11:29:59 | kashyap | 12:21 * kashyap does notice 'detect_zeroes' flag as an option for @BlockDeviceInfo | |
| 11:30:02 | kashyap | 12:22 < mreitz> At least from what I can tell from the code | |
| 11:30:04 | kashyap | 12:22 < mreitz> Yeah, you can open the target BDS with detect-zeroes=on... | |
| 11:30:07 | kashyap | 12:24 < mreitz> But as far as I can tell, copy-on-read actually always uses zero detection and converts requests accordingly | |
| 11:30:10 | kashyap | 12:24 < mreitz> And block-stream performs its writes through the copy-on-read mechanic | |
| 11:30:13 | kashyap | 12:24 < kashyap> mreitz: Ah, so, in that case (when the target block dev is opened with 'detect-zeroes=on'), thin provisioning will be retained? | |
| 11:30:16 | kashyap | 12:24 < mreitz> So I'd think that block-stream should always do zero detection | |
| 11:31:21 | mdbooth | kashyap: Ah, ok. So there's no detect zeroes option because it's always done. | |
| 11:31:45 | mdbooth | So we'd just need to ensure we're doing it on the output? | |
| 11:32:40 | kashyap | mdbooth: I think so. The experiment is: | |
| 11:32:43 | kashyap | Build a backing chain of empty images, stream and see whether something gets allocated. | |
| 11:32:43 | mdbooth | gmann: Anyway, ^^^ is orthogonal to cinderv3. | |
| 11:33:05 | mdbooth | As johnthetubaguy says. | |
| 11:33:24 | mdbooth | And I don't see how we could support swap_volume with multi_attach in any case. | |
| 11:34:21 | mdbooth | kashyap: Never mind the backing chain. Create a sparse flat file with some data allocated in the middle and stream that. | |
| 11:34:39 | mdbooth | Can you get it to create an output file which is also sparse? | |
| 11:34:55 | kashyap | Need to try it out. | |
| 11:35:09 | kashyap | mdbooth: You have a quick way to write zeroes to a qcow2 file? | |
| 11:35:32 | mdbooth | kashyap: To be a good test, I would explicitly not use qcow2 at any point. | |
| 11:35:43 | mdbooth | Should be flat at both source and dest. | |
| 11:35:48 | kashyap | [Maybe dd if=/dev/zero...] | |
| 11:35:59 | mdbooth | However, creating sparse qcow2 is trivial, right? | |
| 11:36:11 | mdbooth | Also, libguestfs if you want to write data into it. | |
| 11:36:12 | kashyap | mdbooth: Not use QCOW2 because, to remove its metadata complications? | |
| 11:36:40 | mdbooth | Yeah, the interesting test is whether sparse detection works *without* qcow2 metadata. | |
| 11:36:56 | kashyap | Oh, yes, 'guestfish'. In the past I did use it, instead of 'dd' | |
| 11:42:08 | kashyap | mdbooth: Sparse qcow2 is the default, no? (When you do `qemu-img create -f qcow2 foo.qcow2 10M`) | |
| 11:44:54 | gmann | kashyap: mdbooth seems interesting to me. but i have to go home now(its late). ll read log and catch you guys later. thanks for help btw | |
| 11:45:37 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446777 | |
| 12:18:57 | openstackgerrit | Gábor Antal proposed openstack/nova master: Add json style checking for sample notifications https://review.openstack.org/443677 | |
| 12:20:30 | openstackgerrit | Gábor Antal proposed openstack/nova master: Add json style checking for sample notifications https://review.openstack.org/443677 | |
| 12:36:55 | cdent | edleafe: not sure if this will be helpful on your functional test explorations but I was just reminded that I had this that I started before jay's big functional test he did: https://review.openstack.org/#/c/427309/ | |
| 12:55:31 | edleafe | kashyap: are you around for some questions? | |
| 12:55:47 | dims | cdent : could use a "## Need Volunteers/help" section :) thanks for the updates! | |
| 12:56:06 | openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Remove old oslo.messaging transport aliases https://review.openstack.org/446990 | |
| 12:56:06 | edleafe | cdent: my problem was the driver mocks. Started on it, but on a client call now | |
| 12:56:07 | kashyap | edleafe: Hiya, yes. I can try :-) | |
| 12:56:55 | cdent | dims: that's a good idea but I would be half inclined to put that as the top level heading and indent everything else :) | |
| 12:57:06 | dims | :) | |
| 12:58:46 | edleafe | kashyap: it's the same problem that mriedem was talking to you about - volume migration failures | |
| 12:59:27 | edleafe | kashyap: on a call with the client now - give me a few | |
| 13:00:51 | kashyap | edleafe: Sure. Just to note: I wasn't following the entire issue from Nova POV, mdbooth is more in the trenches there. I was mostly following it from libvirt / QEMU perspective | |
| 13:03:43 | edleafe | kashyap: well, there are two different errors: one from libvirt, and one from QEMU :) | |
| 13:04:06 | kashyap | edleafe: One of them is the memory corruption issue? | |
| 13:04:10 | kashyap | (The one from libvirt) | |
| 13:06:03 | kashyap | And do you also have a log for the other one from QEMU? | |
| 13:06:29 | edleafe | kashyap: I can get them | |
| 13:06:33 | mriedem | kashyap: edleafe isn't talking about the thing from yesterday | |
| 13:06:39 | mriedem | he's talking about volume swap | |
| 13:06:46 | edleafe | kashyap: the error is libvirtError: Requested operation is not valid: pivot of disk 'vdb' requires an active copy job | |
| 13:06:57 | kashyap | mriedem: Ah, a different issue | |
| 13:07:16 | edleafe | This is on a newer libvirt, using Mitaka with Matt's patches | |