| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-06 | |||
| 09:51:23 | mdbooth | kashyap: Yeah | |
| 09:51:27 | kashyap | libvirt uses QMP (the JSON RPC) with QEMU. Not HMP. (only rarely) | |
| 09:51:49 | kashyap | Yes: | |
| 09:51:49 | kashyap | 2018-06-04 11:13:18.158+0000: 32308: debug : qemuMonitorJSONCommandWithFd:301 : Send command '{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","scsi":"off","bus":"pci.0","addr" | |
| 09:51:53 | kashyap | :"0x6","share-rw":"on","drive":"drive-virtio-disk1","id":"virtio-disk1"},"id":"libvirt-20"}' for write with FD -1 | |
| 09:52:16 | mdbooth | Right, so that's 'shared', right? | |
| 09:52:25 | kashyap | mdbooth: Yes. And libvirt will _not_ migrate it. | |
| 09:52:28 | mdbooth | i.e. it's multiattach | |
| 09:52:36 | kashyap | Here is the libvirt source itself saying as such: | |
| 09:52:37 | kashyap | 288 /* Default is to migrate only non-shared non-readonly disks | |
| 09:52:37 | kashyap | 289 * with source */ | |
| 09:52:37 | kashyap | 290 return !disk->src->shared && !disk->src->readonly && | |
| 09:52:37 | kashyap | 291 !virStorageSourceIsEmpty(disk->src); | |
| 09:52:39 | kashyap | 292 } | |
| 09:52:42 | kashyap | 293 | |
| 09:52:51 | mdbooth | kashyap: Except that it does | |
| 09:53:12 | mdbooth | Keep searching down on that disk id | |
| 09:53:49 | mdbooth | 2018-06-04 10:57:05.201+0000 | |
| 09:53:53 | mdbooth | It does a drive mirror | |
| 09:54:18 | mdbooth | This isn't live migration, btw | |
| 09:54:24 | mdbooth | This is volume migration | |
| 09:54:52 | mdbooth | So the data in volume a is moved to volume b and a is seamlessly substituted for b in the domain | |
| 09:55:26 | kashyap | mdbooth: Hang on -- | |
| 09:55:29 | mdbooth | So it looks like this version of qemu/libvirt at least does permit drive mirror of a shared disk | |
| 09:55:36 | kashyap | The `drive-mirror` is done fir 'disk0' | |
| 09:55:39 | kashyap | s/fir/for/ | |
| 09:55:50 | kashyap | 2018-06-04 10:59:30.956+0000: 32306: debug : qemuMonitorJSONCommandWithFd:301 : Send command '{"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0","target":"/opt/stack/data/n | |
| 09:55:55 | kashyap | ova/instances/snapshots/tmpQeWSxD/8d2e75dbf0a4466fb31f11ca06a21a96.delta","sync":"top","mode":"existing","format":"qcow2"},"id":"libvirt-28"}' for write with FD -1 | |
| 09:56:16 | mdbooth | kashyap: That's much later | |
| 09:56:21 | mdbooth | Different operation | |
| 09:56:27 | mdbooth | Look at 2018-06-04 10:57:05.201+0000 | |
| 09:57:02 | kashyap | mdbooth: Okay, I seem to have misread the code :-( | |
| 09:57:07 | kashyap | mdbooth: libvirt _doesn't_ forbid | |
| 09:58:50 | pvc | hi | |
| 09:59:32 | kashyap | mdbooth: So libvirt should _not_ forbid because, the said shared disk might be in use at any given point. | |
| 09:59:45 | kashyap | But probably that's what you were trying to tell me. | |
| 10:00:05 | mdbooth | kashyap: Actually I think it *should* forbid it, but doesn't | |
| 10:00:11 | pvc | kashyap | |
| 10:00:25 | pvc | can help me with gpu instnace? | |
| 10:00:26 | kashyap | mdbooth: libvirt developer says the above reason is why it shouldn't forbid it | |
| 10:00:27 | mdbooth | Although... meh it should probably give us the rope to hang ourselves with | |
| 10:00:48 | kashyap | pvc: Hi, in the middle of something. Please ask such questions to the operators list or ask.openstack.org | |
| 10:01:09 | kashyap | (And I don't know the answer to it, top off my head) | |
| 10:02:03 | mdbooth | kashyap: I don't understand why what 'be in use' means. Can you explain? | |
| 10:02:13 | kashyap | mdbooth: BTW, yes. I did see the timestamp you pointed out. And indeed the shared disk is being `drive-mirror`ed: | |
| 10:02:14 | mdbooth | why what? | |
| 10:02:16 | kashyap | 2018-06-04 10:57:05.201+0000: 32306: debug : qemuMonitorJSONCommandWithFd:301 : Send command '{"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk1","target":"/dev/sdb","sync":"full","mode":"existing","format":"raw"},"id":"libvirt-23"}' for write with FD -1 | |
| 10:02:16 | mdbooth | s/why// | |
| 10:02:47 | kashyap | mdbooth: The shared disk might be in use by the other VM | |
| 10:03:11 | mdbooth | Exactly, and iiuc libvirt/qemu has no way to know if this is the case | |
| 10:03:40 | mdbooth | So doing a drive mirror isn't a safe operation, because it can't take account of changes made by another vm | |
| 10:04:12 | kashyap | (Hmm, the libvirt dev isn't sure if QEMU disallows forbids or not.) | |
| 10:04:27 | mdbooth | You'd need a side-channel between all attached vms to make that safe. | |
| 10:04:42 | mdbooth | And you can't do it in the guest os, because this operation isn't visible to the guest os | |
| 10:05:29 | kashyap | mdbooth: Mind filing a bug for this? | |
| 10:05:30 | mdbooth | kashyap: Which channel are you on? | |
| 10:05:35 | kashyap | #virt, OFTC | |
| 10:05:53 | mdbooth | kashyap: I'm not convinced it's a bug, tbh. | |
| 10:06:04 | mdbooth | Well actually I do think it's a bug, but in Nova. | |
| 10:06:20 | mdbooth | I don't think we should be attempting to do this in the first place. | |
| 10:06:22 | kashyap | (That's what I meant, Nova LP bug) | |
| 10:07:12 | mdbooth | The problem with that is that to fix it we have to break an api | |
| 10:07:24 | mdbooth | i.e. make something fail which previously succeeded | |
| 10:11:00 | kashyap | Hmm | |
| 10:13:38 | kashyap | mdbooth: Can you step back a bit and remind me the actual Nova test? | |
| 10:14:04 | kashyap | It is doing "swap volume", i.e. volume migration? But in "mult-attach" context? | |
| 10:20:29 | kashyap | mdbooth: So, the _default_ behaviour for libvirt is that it won't migrate the shared disks if you don't ask for them to be shared. | |
| 10:20:51 | kashyap | If you _do_ aske the shared disks to be migrated, libvirt will honor that. | |
| 10:21:02 | kashyap | I think Nova shouldn't do that is what you are saying. | |
| 10:21:25 | lyarwood | kashyap: volume migration in this context is swap volume, so rebasing in libvirt terms. | |
| 10:21:46 | kashyap | Yep. blockRebase()/blockCopy() --> calls QEMU 'drive-mirror' | |
| 10:22:32 | lyarwood | kashyap: right, just checking you didn't have it confused with actual block LM. | |
| 10:23:06 | kashyap | Yeah, my head already auto-maps _swap_volume() to the above APIs. But thank you :-) | |
| 10:23:35 | kashyap | lyarwood: Actual block LM also goes down to that 'drive-mirror' call, you know that? | |
| 10:24:16 | lyarwood | kashyap: yeah without going through blockRebase right? | |
| 10:24:30 | lyarwood | kashyap: it's just a copy | |
| 10:26:31 | kashyap | Not sure on that, have to check | |
| 10:48:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |
| 10:48:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267 | |
| 10:48:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847 | |
| 10:48:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 10:48:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 10:50:27 | openstackgerrit | Merged openstack/nova master: Ensure resource class cache when listing usages https://review.openstack.org/572652 | |
| 11:32:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove network info stubbing in functional test https://review.openstack.org/570018 | |
| 11:32:47 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |
| 11:32:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267 | |
| 11:32:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847 | |
| 11:32:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 11:32:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 12:21:23 | frickler | mriedem_away: yep, lgtm | |
| 12:38:06 | openstackgerrit | wanghongtao proposed openstack/nova master: Fix the metadata re to match the unicode https://review.openstack.org/536236 | |
| 12:48:36 | openstackgerrit | wanghongtao proposed openstack/nova master: Fix the metadata re to match the unicode https://review.openstack.org/536236 | |
| 13:04:28 | mriedem | lyarwood: i did find via code inspection at least one cinder bug while looking at the access mode logic in the volume manager, comments in the spec https://review.openstack.org/#/c/552078/ | |
| 13:04:59 | mriedem | lyarwood: i don't really have a great way to test this stuff until i get the stack written through the compute api, so i figure any issues that are on the cinder side with modeling per-attachment access mode will have to be resolved as dependent bugs | |
| 13:05:01 | artom | mdbooth, so, re: mailing list post about swap volume - I believe we could just change the response code in the multi-attach case to a 400 or 409 or whatever, without a new microversion | |
| 13:05:03 | artom | https://docs.openstack.org/nova/latest/contributor/microversions.html | |
| 13:05:21 | mriedem | lyarwood: beyond that, what is holding up the spec? i'll rename attach_mode to access_mode to match what's in the connection_info | |
| 13:06:22 | mdbooth | artom: I wasn't sure, but that was my intention. I don't think we should preserve this behaviour at all, even behind a microversion. | |
| 13:06:56 | mdbooth | artom: But I'm not clear on the policy here, hence the ML post. | |
| 13:07:00 | artom | mdbooth, I agree, which is why I would think using a response code that's already possible for that request would be OK | |