| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-06 | |||
| 09:29:38 | jangutter | sahid: Thanks for taking a look at https://review.openstack.org/567148 . I agree that there's "duplicate" info passed through for VIFHostDevice, but at least it's consistent with other VIF's. I think it has to do with the distinction between the VIF objects and the VIFPortProfile objects. | |
| 09:31:31 | jangutter | sahid: The os-vif VIF describes "how to wire it into the VM". The os-vif port-profile describes "how to wire it into the datapath". Because the distinction is not particularly clear with libvirt, there's some muddy overlap. | |
| 09:33:47 | openstackgerrit | Bhagyashri Shewale proposed openstack/python-novaclient master: Modify novaclient to support basic attributes https://review.openstack.org/572285 | |
| 09:38:11 | kashyap | lyarwood: Meta comment: The libvirt debug log on the URL is supposedly: 3.2M. But my download of it is still running beyond 7MB | |
| 09:38:38 | kashyap | Maybe `wget` extracts it downloads | |
| 09:40:59 | kashyap | The full download (on a remote machine) was: 47M :-) | |
| 09:46:57 | mdbooth | kashyap: I'm looking at the libvirt logs from this multiattach job: http://logs.openstack.org/58/567258/5/check/nova-multiattach/d23fad8/logs/libvirt/libvirtd.txt.gz | |
| 09:47:25 | mdbooth | Specifically I'm trying to see what goes on during the successful multiattach swap volume test | |
| 09:49:23 | mdbooth | It looks like the drive of interest is added around 2018-06-04 10:56:57.865+0000 | |
| 09:49:55 | kashyap | Can you tell the drive ID? | |
| 09:50:09 | mdbooth | drive-virtio-1 | |
| 09:50:09 | kashyap | Is it? 'drive-virtio-disk1' | |
| 09:50:38 | mdbooth | I'm slightly confused, because I don't see shared in the 'human-monitor-command' | |
| 09:51:02 | kashyap | I don't see a "drive-virtio-1". Did you mean: 'drive-virtio-disk1'? | |
| 09:51:21 | mdbooth | But there is "share-rw":"on" in what seems to be sent to qemu | |
| 09:51:23 | mdbooth | kashyap: Yeah | |
| 09:51:27 | kashyap | libvirt uses QMP (the JSON RPC) with QEMU. Not HMP. (only rarely) | |
| 09:51:49 | kashyap | 2018-06-04 11:13:18.158+0000: 32308: debug : qemuMonitorJSONCommandWithFd:301 : Send command '{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","scsi":"off","bus":"pci.0","addr" | |
| 09:51:49 | kashyap | Yes: | |
| 09:51:53 | kashyap | :"0x6","share-rw":"on","drive":"drive-virtio-disk1","id":"virtio-disk1"},"id":"libvirt-20"}' for write with FD -1 | |
| 09:52:16 | mdbooth | Right, so that's 'shared', right? | |
| 09:52:25 | kashyap | mdbooth: Yes. And libvirt will _not_ migrate it. | |
| 09:52:28 | mdbooth | i.e. it's multiattach | |
| 09:52:36 | kashyap | Here is the libvirt source itself saying as such: | |
| 09:52:37 | kashyap | 291 !virStorageSourceIsEmpty(disk->src); | |
| 09:52:37 | kashyap | 290 return !disk->src->shared && !disk->src->readonly && | |
| 09:52:37 | kashyap | 289 * with source */ | |
| 09:52:37 | kashyap | 288 /* Default is to migrate only non-shared non-readonly disks | |
| 09:52:39 | kashyap | 292 } | |
| 09:52:42 | kashyap | 293 | |
| 09:52:51 | mdbooth | kashyap: Except that it does | |
| 09:53:12 | mdbooth | Keep searching down on that disk id | |
| 09:53:49 | mdbooth | 2018-06-04 10:57:05.201+0000 | |
| 09:53:53 | mdbooth | It does a drive mirror | |
| 09:54:18 | mdbooth | This isn't live migration, btw | |
| 09:54:24 | mdbooth | This is volume migration | |
| 09:54:52 | mdbooth | So the data in volume a is moved to volume b and a is seamlessly substituted for b in the domain | |
| 09:55:26 | kashyap | mdbooth: Hang on -- | |
| 09:55:29 | mdbooth | So it looks like this version of qemu/libvirt at least does permit drive mirror of a shared disk | |
| 09:55:36 | kashyap | The `drive-mirror` is done fir 'disk0' | |
| 09:55:39 | kashyap | s/fir/for/ | |
| 09:55:50 | kashyap | 2018-06-04 10:59:30.956+0000: 32306: debug : qemuMonitorJSONCommandWithFd:301 : Send command '{"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0","target":"/opt/stack/data/n | |
| 09:55:55 | kashyap | ova/instances/snapshots/tmpQeWSxD/8d2e75dbf0a4466fb31f11ca06a21a96.delta","sync":"top","mode":"existing","format":"qcow2"},"id":"libvirt-28"}' for write with FD -1 | |
| 09:56:16 | mdbooth | kashyap: That's much later | |
| 09:56:21 | mdbooth | Different operation | |
| 09:56:27 | mdbooth | Look at 2018-06-04 10:57:05.201+0000 | |
| 09:57:02 | kashyap | mdbooth: Okay, I seem to have misread the code :-( | |
| 09:57:07 | kashyap | mdbooth: libvirt _doesn't_ forbid | |
| 09:58:50 | pvc | hi | |
| 09:59:32 | kashyap | mdbooth: So libvirt should _not_ forbid because, the said shared disk might be in use at any given point. | |
| 09:59:45 | kashyap | But probably that's what you were trying to tell me. | |
| 10:00:05 | mdbooth | kashyap: Actually I think it *should* forbid it, but doesn't | |
| 10:00:11 | pvc | kashyap | |
| 10:00:25 | pvc | can help me with gpu instnace? | |
| 10:00:26 | kashyap | mdbooth: libvirt developer says the above reason is why it shouldn't forbid it | |
| 10:00:27 | mdbooth | Although... meh it should probably give us the rope to hang ourselves with | |
| 10:00:48 | kashyap | pvc: Hi, in the middle of something. Please ask such questions to the operators list or ask.openstack.org | |
| 10:01:09 | kashyap | (And I don't know the answer to it, top off my head) | |
| 10:02:03 | mdbooth | kashyap: I don't understand why what 'be in use' means. Can you explain? | |
| 10:02:13 | kashyap | mdbooth: BTW, yes. I did see the timestamp you pointed out. And indeed the shared disk is being `drive-mirror`ed: | |
| 10:02:14 | mdbooth | why what? | |
| 10:02:16 | mdbooth | s/why// | |
| 10:02:16 | kashyap | 2018-06-04 10:57:05.201+0000: 32306: debug : qemuMonitorJSONCommandWithFd:301 : Send command '{"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk1","target":"/dev/sdb","sync":"full","mode":"existing","format":"raw"},"id":"libvirt-23"}' for write with FD -1 | |
| 10:02:47 | kashyap | mdbooth: The shared disk might be in use by the other VM | |
| 10:03:11 | mdbooth | Exactly, and iiuc libvirt/qemu has no way to know if this is the case | |
| 10:03:40 | mdbooth | So doing a drive mirror isn't a safe operation, because it can't take account of changes made by another vm | |
| 10:04:12 | kashyap | (Hmm, the libvirt dev isn't sure if QEMU disallows forbids or not.) | |
| 10:04:27 | mdbooth | You'd need a side-channel between all attached vms to make that safe. | |
| 10:04:42 | mdbooth | And you can't do it in the guest os, because this operation isn't visible to the guest os | |
| 10:05:29 | kashyap | mdbooth: Mind filing a bug for this? | |
| 10:05:30 | mdbooth | kashyap: Which channel are you on? | |
| 10:05:35 | kashyap | #virt, OFTC | |
| 10:05:53 | mdbooth | kashyap: I'm not convinced it's a bug, tbh. | |
| 10:06:04 | mdbooth | Well actually I do think it's a bug, but in Nova. | |
| 10:06:20 | mdbooth | I don't think we should be attempting to do this in the first place. | |
| 10:06:22 | kashyap | (That's what I meant, Nova LP bug) | |
| 10:07:12 | mdbooth | The problem with that is that to fix it we have to break an api | |
| 10:07:24 | mdbooth | i.e. make something fail which previously succeeded | |
| 10:11:00 | kashyap | Hmm | |
| 10:13:38 | kashyap | mdbooth: Can you step back a bit and remind me the actual Nova test? | |
| 10:14:04 | kashyap | It is doing "swap volume", i.e. volume migration? But in "mult-attach" context? | |
| 10:20:29 | kashyap | mdbooth: So, the _default_ behaviour for libvirt is that it won't migrate the shared disks if you don't ask for them to be shared. | |
| 10:20:51 | kashyap | If you _do_ aske the shared disks to be migrated, libvirt will honor that. | |
| 10:21:02 | kashyap | I think Nova shouldn't do that is what you are saying. | |
| 10:21:25 | lyarwood | kashyap: volume migration in this context is swap volume, so rebasing in libvirt terms. | |
| 10:21:46 | kashyap | Yep. blockRebase()/blockCopy() --> calls QEMU 'drive-mirror' | |
| 10:22:32 | lyarwood | kashyap: right, just checking you didn't have it confused with actual block LM. | |
| 10:23:06 | kashyap | Yeah, my head already auto-maps _swap_volume() to the above APIs. But thank you :-) | |
| 10:23:35 | kashyap | lyarwood: Actual block LM also goes down to that 'drive-mirror' call, you know that? | |
| 10:24:16 | lyarwood | kashyap: yeah without going through blockRebase right? | |
| 10:24:30 | lyarwood | kashyap: it's just a copy | |
| 10:26:31 | kashyap | Not sure on that, have to check | |
| 10:48:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |
| 10:48:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267 | |
| 10:48:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847 | |
| 10:48:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 10:48:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 10:50:27 | openstackgerrit | Merged openstack/nova master: Ensure resource class cache when listing usages https://review.openstack.org/572652 | |
| 11:32:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove network info stubbing in functional test https://review.openstack.org/570018 | |
| 11:32:47 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |