| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-17 | |||
| 13:15:37 | kashyap | Ah! | |
| 13:15:48 | kashyap | For a moment, lost track of release status | |
| 13:16:03 | kashyap | Says, it's "Legacy Release" phase https://releases.openstack.org/ | |
| 13:16:26 | kashyap | edleafe: That fix mriedem pointed out is what you're looking for. | |
| 13:17:08 | edleafe | kashyap: just checked their system, and they have those changes applied | |
| 13:17:55 | kashyap | edleafe: And they still see the same pivot failure? | |
| 13:18:35 | SzarkaAlex | @gibi Hi! Are you here? | |
| 13:20:11 | mdbooth | edleafe: Can you do me a favour. There's an edge case in that code that we knowingly glossed over, namely that we can't detect if the job fails. | |
| 13:21:19 | mdbooth | edleafe: Can you correlate the Nova log with the libvirt logs and look for a failure of the rebase job? | |
| 13:21:54 | edleafe | mdbooth: sec... | |
| 13:23:20 | edleafe | mdbooth: like this? http://paste.openstack.org/show/603131/ | |
| 13:23:38 | mdbooth | edleafe: That's the nova log. | |
| 13:23:47 | edleafe | ah | |
| 13:23:49 | mdbooth | Can you look in the libvirt log? | |
| 13:23:59 | mdbooth | /var/log/libvirt/whatever probably | |
| 13:24:36 | edleafe | looking... | |
| 13:24:55 | mdbooth | kashyap: 'requires an active copy job' | |
| 13:25:22 | mdbooth | Is that the same as the pivot-too-early problem? | |
| 13:25:43 | kashyap | mdbooth: Oh, I think vaguely recall, I think that edge case you discussed in one of the reviews... | |
| 13:25:57 | kashyap | mdbooth: Yes, it's the same problem | |
| 13:26:10 | kashyap | Wait, let me think again | |
| 13:26:29 | kashyap | mdbooth: No, I was wrong: | |
| 13:26:36 | mdbooth | _("disk '%s' not ready for pivot yet"), | |
| 13:26:46 | kashyap | Pivot too early implies that there _is_ an active block operation in progress | |
| 13:26:51 | mdbooth | if (disk->mirrorState != VIR_DOMAIN_DISK_MIRROR_STATE_READY) { | |
| 13:26:51 | mdbooth | virReportError(VIR_ERR_BLOCK_COPY_ACTIVE, | |
| 13:26:51 | mdbooth | _("disk '%s' not ready for pivot yet"), | |
| 13:26:51 | mdbooth | disk->dst); | |
| 13:26:53 | mdbooth | vs: | |
| 13:27:01 | mdbooth | if (!disk->mirror) { | |
| 13:27:01 | mdbooth | virReportError(VIR_ERR_OPERATION_INVALID, | |
| 13:27:01 | mdbooth | _("pivot of disk '%s' requires an active copy job"), | |
| 13:27:07 | mdbooth | I think there's no copy job at all | |
| 13:27:14 | kashyap | Requires an active job means: there's no block operation at all | |
| 13:27:19 | mdbooth | edleafe: Yeah, those libvirt logs would be really interesting | |
| 13:27:23 | kashyap | s/no/no _live_/ | |
| 13:29:16 | mdbooth | Incidentally, if the issue is that the job failed I think we *have* to switch to consuming events from libvirt. | |
| 13:29:21 | mdbooth | I think we're out of workarounds | |
| 13:29:33 | kashyap | edleafe: Those logs would be useful, assuming you had log filters enabled in /etc/libvirt/libvirtd.conf: | |
| 13:29:36 | kashyap | log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 13:29:39 | kashyap | log_outputs="1:file:/var/log/libvirt/libvirtd.log" | |
| 13:30:09 | kashyap | (Restart libvirtd; repeat your test; _then_ post the log.) | |
| 13:31:06 | mdbooth | kashyap: Actually looking at that code in qemuDomainBlockPivot(), I think the Nova code is wrong when it does: | |
| 13:31:13 | mdbooth | if status is None: | |
| 13:31:14 | mdbooth | return True | |
| 13:31:32 | mdbooth | Looks like it should raise an exception at that point, as it will never succeed. | |
| 13:31:37 | edleafe | mdbooth: I see the log; it's 3GB, but its last entry is about 8 hours ago?!? | |
| 13:32:06 | mdbooth | edleafe: Did it get rolled somewhere? | |
| 13:32:34 | kashyap | Oh dear | |
| 13:32:43 | edleafe | mdbooth: kashyap: let me restart libvirt, verify that it's logging, repeat the test, and then I'll post the logs and re-ping you | |
| 13:32:47 | kashyap | edleafe: If possible to repeat the test | |
| 13:33:13 | kashyap | First, please take a backup of that 3G log, _if_ you need it, and then empty it out ($ > /var/log/libvirt/libvirt.log), so we can start from a clean slate. | |
| 13:33:18 | edleafe | mdbooth: it has a current timestamp, but last entry is 8 hours ago | |
| 13:33:21 | kashyap | Then, enable the log filters, repeat the test, etc. | |
| 13:33:35 | edleafe | kashyap: I was going to roll over the log, eyah | |
| 13:33:37 | edleafe | yeah | |
| 13:34:02 | edleafe | but first another pot of coffee is a higher priority :) | |
| 13:34:26 | kashyap | Go for it | |
| 13:34:59 | kashyap | mdbooth: Sigh, it's like a landmine. Not sure how your eyes don't bleed looking at that | |
| 13:41:48 | kashyap | mdbooth: And yeah, didn't miss your quote above, from qemuDomainBlockPivot() function, it is looking for the libvirt event READY to decide whether the job is ready for pivot or not... | |
| 13:42:11 | mdbooth | kashyap: But before that it'll raise a different error if the job doesn't exist | |
| 13:42:28 | mdbooth | Specifically the error that edleafe is seeing | |
| 13:42:54 | kashyap | mdbooth: You're referring to: | |
| 13:42:55 | kashyap | if (!disk->mirror) { | |
| 13:42:55 | kashyap | virReportError(VIR_ERR_OPERATION_INVALID, | |
| 13:42:55 | kashyap | _("pivot of disk '%s' requires an active copy job"), | |
| 13:42:58 | kashyap | disk->dst); | |
| 13:43:00 | mdbooth | Yes | |
| 13:43:01 | kashyap | ? | |
| 13:43:26 | openstackgerrit | Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | |
| 13:44:57 | kashyap | mdbooth: In that case, either the job was already finished (maybe not gracefully); or it was not started. Hence, it whines "no block operation for me work on" | |
| 13:45:25 | kashyap | Let's wait for edleafe's logs (hopefully small, with that specific test) | |
| 13:45:30 | mdbooth | Either way, based on that code I belive the pivot will fail | |
| 14:10:45 | openstackgerrit | Michael Henkel proposed openstack/nova master: contrail: add vrouter VIF plugin type support https://review.openstack.org/441183 | |
| 14:11:04 | leakypipes | mhenkel: :) | |
| 14:11:26 | kashyap | mdbooth: So, if you want to hear the results (credit: Max from QEMU did the tests & reported) of earlier experiment with qcow2, I can write here. | |
| 14:11:32 | mhenkel | hey leakypipes | |
| 14:11:36 | mdbooth | kashyap: Shoot | |
| 14:11:41 | kashyap | mdbooth: When I asked about the format, he says it doesn't really matter: Detection whether the source contains zeroes -- That is independent of the format, the code actually checks the data read from the source whether it is zero. | |
| 14:11:42 | mdbooth | s/with/without/ | |
| 14:12:21 | kashyap | mdbooth: Okay, here we go: | |
| 14:12:30 | mhenkel | leakypipes: is the commit msg looking ok now? | |
| 14:12:46 | kashyap | The experiment being: build a backing chain, of depth one, of empty images, perform 'block-stream' and see whether something gets allocated. Three cases: | |
| 14:12:53 | kashyap | (1) With empty images nothing is allocated at all; | |
| 14:13:18 | kashyap | (2) With zeros (using 'qemu-io -c 'write -P 0 0 64M' base.qcow2') written to the base, the qcow2 file for some reason (there's one, but not sure what) grows by a cluster (64 kB). | |
| 14:13:26 | kashyap | (3) With non-zero data (using 'qemu-io -c 'write -P 42 0 64M' base.qcow2'), all data is written to the overlay. | |
| 14:13:45 | kashyap | * * * | |
| 14:13:47 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009 | |
| 14:14:17 | dansmith | so mriedem, we supposedly have tests for this ^ and they pass, but the two cases they check don't work in real life | |
| 14:14:21 | kashyap | mdbooth: Max tells (& I can confirm by testing as I have the QMP env. ready) running the same tests on a raw image should yield the same results. | |
| 14:14:23 | dansmith | mriedem: thoughts on how to test that change? | |
| 14:14:27 | openstackgerrit | Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | |
| 14:14:47 | mdbooth | kashyap: Is this dependent on how the target disk is initially created? | |
| 14:15:29 | kashyap | mdbooth: Hmm, that is a good question. The target image in this specific test is a simple qcow2 overlay | |
| 14:16:00 | mdbooth | So, in the case of volume migration the source and destination are both block devices from the pov of qemu | |
| 14:16:16 | edmondsw | mriedem, what is the plan/status on setting send_service_user_token conf setting to True by default (i.e. no longer experimental)? | |
| 14:16:23 | mriedem | dansmith: ooo i smell a customer bug | |
| 14:16:56 | mriedem | dansmith: tests where? novaclient? | |
| 14:17:00 | mdbooth | kashyap: I'd be interested to see the result if you create 2 sparse files with loopback devices, and copy between the loopback devices | |
| 14:17:08 | dansmith | mriedem: hang on, linking | |