| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-17 | |||
| 13:21:54 | edleafe | mdbooth: sec... | |
| 13:23:20 | edleafe | mdbooth: like this? http://paste.openstack.org/show/603131/ | |
| 13:23:38 | mdbooth | edleafe: That's the nova log. | |
| 13:23:47 | edleafe | ah | |
| 13:23:49 | mdbooth | Can you look in the libvirt log? | |
| 13:23:59 | mdbooth | /var/log/libvirt/whatever probably | |
| 13:24:36 | edleafe | looking... | |
| 13:24:55 | mdbooth | kashyap: 'requires an active copy job' | |
| 13:25:22 | mdbooth | Is that the same as the pivot-too-early problem? | |
| 13:25:43 | kashyap | mdbooth: Oh, I think vaguely recall, I think that edge case you discussed in one of the reviews... | |
| 13:25:57 | kashyap | mdbooth: Yes, it's the same problem | |
| 13:26:10 | kashyap | Wait, let me think again | |
| 13:26:29 | kashyap | mdbooth: No, I was wrong: | |
| 13:26:36 | mdbooth | _("disk '%s' not ready for pivot yet"), | |
| 13:26:46 | kashyap | Pivot too early implies that there _is_ an active block operation in progress | |
| 13:26:51 | mdbooth | disk->dst); | |
| 13:26:51 | mdbooth | _("disk '%s' not ready for pivot yet"), | |
| 13:26:51 | mdbooth | virReportError(VIR_ERR_BLOCK_COPY_ACTIVE, | |
| 13:26:51 | mdbooth | if (disk->mirrorState != VIR_DOMAIN_DISK_MIRROR_STATE_READY) { | |
| 13:26:53 | mdbooth | vs: | |
| 13:27:01 | mdbooth | _("pivot of disk '%s' requires an active copy job"), | |
| 13:27:01 | mdbooth | virReportError(VIR_ERR_OPERATION_INVALID, | |
| 13:27:01 | mdbooth | if (!disk->mirror) { | |
| 13:27:07 | mdbooth | I think there's no copy job at all | |
| 13:27:14 | kashyap | Requires an active job means: there's no block operation at all | |
| 13:27:19 | mdbooth | edleafe: Yeah, those libvirt logs would be really interesting | |
| 13:27:23 | kashyap | s/no/no _live_/ | |
| 13:29:16 | mdbooth | Incidentally, if the issue is that the job failed I think we *have* to switch to consuming events from libvirt. | |
| 13:29:21 | mdbooth | I think we're out of workarounds | |
| 13:29:33 | kashyap | edleafe: Those logs would be useful, assuming you had log filters enabled in /etc/libvirt/libvirtd.conf: | |
| 13:29:36 | kashyap | log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 13:29:39 | kashyap | log_outputs="1:file:/var/log/libvirt/libvirtd.log" | |
| 13:30:09 | kashyap | (Restart libvirtd; repeat your test; _then_ post the log.) | |
| 13:31:06 | mdbooth | kashyap: Actually looking at that code in qemuDomainBlockPivot(), I think the Nova code is wrong when it does: | |
| 13:31:13 | mdbooth | if status is None: | |
| 13:31:14 | mdbooth | return True | |
| 13:31:32 | mdbooth | Looks like it should raise an exception at that point, as it will never succeed. | |
| 13:31:37 | edleafe | mdbooth: I see the log; it's 3GB, but its last entry is about 8 hours ago?!? | |
| 13:32:06 | mdbooth | edleafe: Did it get rolled somewhere? | |
| 13:32:34 | kashyap | Oh dear | |
| 13:32:43 | edleafe | mdbooth: kashyap: let me restart libvirt, verify that it's logging, repeat the test, and then I'll post the logs and re-ping you | |
| 13:32:47 | kashyap | edleafe: If possible to repeat the test | |
| 13:33:13 | kashyap | First, please take a backup of that 3G log, _if_ you need it, and then empty it out ($ > /var/log/libvirt/libvirt.log), so we can start from a clean slate. | |
| 13:33:18 | edleafe | mdbooth: it has a current timestamp, but last entry is 8 hours ago | |
| 13:33:21 | kashyap | Then, enable the log filters, repeat the test, etc. | |
| 13:33:35 | edleafe | kashyap: I was going to roll over the log, eyah | |
| 13:33:37 | edleafe | yeah | |
| 13:34:02 | edleafe | but first another pot of coffee is a higher priority :) | |
| 13:34:26 | kashyap | Go for it | |
| 13:34:59 | kashyap | mdbooth: Sigh, it's like a landmine. Not sure how your eyes don't bleed looking at that | |
| 13:41:48 | kashyap | mdbooth: And yeah, didn't miss your quote above, from qemuDomainBlockPivot() function, it is looking for the libvirt event READY to decide whether the job is ready for pivot or not... | |
| 13:42:11 | mdbooth | kashyap: But before that it'll raise a different error if the job doesn't exist | |
| 13:42:28 | mdbooth | Specifically the error that edleafe is seeing | |
| 13:42:54 | kashyap | mdbooth: You're referring to: | |
| 13:42:55 | kashyap | _("pivot of disk '%s' requires an active copy job"), | |
| 13:42:55 | kashyap | virReportError(VIR_ERR_OPERATION_INVALID, | |
| 13:42:55 | kashyap | if (!disk->mirror) { | |
| 13:42:58 | kashyap | disk->dst); | |
| 13:43:00 | mdbooth | Yes | |
| 13:43:01 | kashyap | ? | |
| 13:43:26 | openstackgerrit | Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | |
| 13:44:57 | kashyap | mdbooth: In that case, either the job was already finished (maybe not gracefully); or it was not started. Hence, it whines "no block operation for me work on" | |
| 13:45:25 | kashyap | Let's wait for edleafe's logs (hopefully small, with that specific test) | |
| 13:45:30 | mdbooth | Either way, based on that code I belive the pivot will fail | |
| 14:10:45 | openstackgerrit | Michael Henkel proposed openstack/nova master: contrail: add vrouter VIF plugin type support https://review.openstack.org/441183 | |
| 14:11:04 | leakypipes | mhenkel: :) | |
| 14:11:26 | kashyap | mdbooth: So, if you want to hear the results (credit: Max from QEMU did the tests & reported) of earlier experiment with qcow2, I can write here. | |
| 14:11:32 | mhenkel | hey leakypipes | |
| 14:11:36 | mdbooth | kashyap: Shoot | |
| 14:11:41 | kashyap | mdbooth: When I asked about the format, he says it doesn't really matter: Detection whether the source contains zeroes -- That is independent of the format, the code actually checks the data read from the source whether it is zero. | |
| 14:11:42 | mdbooth | s/with/without/ | |
| 14:12:21 | kashyap | mdbooth: Okay, here we go: | |
| 14:12:30 | mhenkel | leakypipes: is the commit msg looking ok now? | |
| 14:12:46 | kashyap | The experiment being: build a backing chain, of depth one, of empty images, perform 'block-stream' and see whether something gets allocated. Three cases: | |
| 14:12:53 | kashyap | (1) With empty images nothing is allocated at all; | |
| 14:13:18 | kashyap | (2) With zeros (using 'qemu-io -c 'write -P 0 0 64M' base.qcow2') written to the base, the qcow2 file for some reason (there's one, but not sure what) grows by a cluster (64 kB). | |
| 14:13:26 | kashyap | (3) With non-zero data (using 'qemu-io -c 'write -P 42 0 64M' base.qcow2'), all data is written to the overlay. | |
| 14:13:45 | kashyap | * * * | |
| 14:13:47 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009 | |
| 14:14:17 | dansmith | so mriedem, we supposedly have tests for this ^ and they pass, but the two cases they check don't work in real life | |
| 14:14:21 | kashyap | mdbooth: Max tells (& I can confirm by testing as I have the QMP env. ready) running the same tests on a raw image should yield the same results. | |
| 14:14:23 | dansmith | mriedem: thoughts on how to test that change? | |
| 14:14:27 | openstackgerrit | Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | |
| 14:14:47 | mdbooth | kashyap: Is this dependent on how the target disk is initially created? | |
| 14:15:29 | kashyap | mdbooth: Hmm, that is a good question. The target image in this specific test is a simple qcow2 overlay | |
| 14:16:00 | mdbooth | So, in the case of volume migration the source and destination are both block devices from the pov of qemu | |
| 14:16:16 | edmondsw | mriedem, what is the plan/status on setting send_service_user_token conf setting to True by default (i.e. no longer experimental)? | |
| 14:16:23 | mriedem | dansmith: ooo i smell a customer bug | |
| 14:16:56 | mriedem | dansmith: tests where? novaclient? | |
| 14:17:00 | mdbooth | kashyap: I'd be interested to see the result if you create 2 sparse files with loopback devices, and copy between the loopback devices | |
| 14:17:08 | mdbooth | Is the result sparse? | |
| 14:17:08 | dansmith | mriedem: hang on, linking | |
| 14:17:13 | mriedem | edmondsw: we need some scale testing first, | |
| 14:17:37 | kashyap | Always asks tricky questions on a Friday, mdbooth... | |
| 14:17:58 | mdbooth | kashyap: Hopefully it's just a little reconfiguration of the test you've already got | |
| 14:18:04 | kashyap | mdbooth: I will let you know, once I come back from my bike ride. My eyes are aching | |
| 14:18:04 | mdbooth | I think you've already done the hard work :) | |
| 14:18:05 | mriedem | edmondsw: raj_sing- is the lead on that stuff i believe. we have service user tokens enabled in the nova-next job, but as part of the original spec we had talked about the need for some scale testing to stress it to force the usage of the service user tokens, and to stress keystone since it has to do double duty token-wise per request now. | |
| 14:18:15 | dansmith | mriedem: added a comment with links | |
| 14:18:25 | mriedem | edmondsw: https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/use-service-tokens.html#performance-impact | |