| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-17 | |||
| 13:27:07 | mdbooth | I think there's no copy job at all | |
| 13:27:14 | kashyap | Requires an active job means: there's no block operation at all | |
| 13:27:19 | mdbooth | edleafe: Yeah, those libvirt logs would be really interesting | |
| 13:27:23 | kashyap | s/no/no _live_/ | |
| 13:29:16 | mdbooth | Incidentally, if the issue is that the job failed I think we *have* to switch to consuming events from libvirt. | |
| 13:29:21 | mdbooth | I think we're out of workarounds | |
| 13:29:33 | kashyap | edleafe: Those logs would be useful, assuming you had log filters enabled in /etc/libvirt/libvirtd.conf: | |
| 13:29:36 | kashyap | log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 13:29:39 | kashyap | log_outputs="1:file:/var/log/libvirt/libvirtd.log" | |
| 13:30:09 | kashyap | (Restart libvirtd; repeat your test; _then_ post the log.) | |
| 13:31:06 | mdbooth | kashyap: Actually looking at that code in qemuDomainBlockPivot(), I think the Nova code is wrong when it does: | |
| 13:31:13 | mdbooth | if status is None: | |
| 13:31:14 | mdbooth | return True | |
| 13:31:32 | mdbooth | Looks like it should raise an exception at that point, as it will never succeed. | |
| 13:31:37 | edleafe | mdbooth: I see the log; it's 3GB, but its last entry is about 8 hours ago?!? | |
| 13:32:06 | mdbooth | edleafe: Did it get rolled somewhere? | |
| 13:32:34 | kashyap | Oh dear | |
| 13:32:43 | edleafe | mdbooth: kashyap: let me restart libvirt, verify that it's logging, repeat the test, and then I'll post the logs and re-ping you | |
| 13:32:47 | kashyap | edleafe: If possible to repeat the test | |
| 13:33:13 | kashyap | First, please take a backup of that 3G log, _if_ you need it, and then empty it out ($ > /var/log/libvirt/libvirt.log), so we can start from a clean slate. | |
| 13:33:18 | edleafe | mdbooth: it has a current timestamp, but last entry is 8 hours ago | |
| 13:33:21 | kashyap | Then, enable the log filters, repeat the test, etc. | |
| 13:33:35 | edleafe | kashyap: I was going to roll over the log, eyah | |
| 13:33:37 | edleafe | yeah | |
| 13:34:02 | edleafe | but first another pot of coffee is a higher priority :) | |
| 13:34:26 | kashyap | Go for it | |
| 13:34:59 | kashyap | mdbooth: Sigh, it's like a landmine. Not sure how your eyes don't bleed looking at that | |
| 13:41:48 | kashyap | mdbooth: And yeah, didn't miss your quote above, from qemuDomainBlockPivot() function, it is looking for the libvirt event READY to decide whether the job is ready for pivot or not... | |
| 13:42:11 | mdbooth | kashyap: But before that it'll raise a different error if the job doesn't exist | |
| 13:42:28 | mdbooth | Specifically the error that edleafe is seeing | |
| 13:42:54 | kashyap | mdbooth: You're referring to: | |
| 13:42:55 | kashyap | _("pivot of disk '%s' requires an active copy job"), | |
| 13:42:55 | kashyap | virReportError(VIR_ERR_OPERATION_INVALID, | |
| 13:42:55 | kashyap | if (!disk->mirror) { | |
| 13:42:58 | kashyap | disk->dst); | |
| 13:43:00 | mdbooth | Yes | |
| 13:43:01 | kashyap | ? | |
| 13:43:26 | openstackgerrit | Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | |
| 13:44:57 | kashyap | mdbooth: In that case, either the job was already finished (maybe not gracefully); or it was not started. Hence, it whines "no block operation for me work on" | |
| 13:45:25 | kashyap | Let's wait for edleafe's logs (hopefully small, with that specific test) | |
| 13:45:30 | mdbooth | Either way, based on that code I belive the pivot will fail | |
| 14:10:45 | openstackgerrit | Michael Henkel proposed openstack/nova master: contrail: add vrouter VIF plugin type support https://review.openstack.org/441183 | |
| 14:11:04 | leakypipes | mhenkel: :) | |
| 14:11:26 | kashyap | mdbooth: So, if you want to hear the results (credit: Max from QEMU did the tests & reported) of earlier experiment with qcow2, I can write here. | |
| 14:11:32 | mhenkel | hey leakypipes | |
| 14:11:36 | mdbooth | kashyap: Shoot | |
| 14:11:41 | kashyap | mdbooth: When I asked about the format, he says it doesn't really matter: Detection whether the source contains zeroes -- That is independent of the format, the code actually checks the data read from the source whether it is zero. | |
| 14:11:42 | mdbooth | s/with/without/ | |
| 14:12:21 | kashyap | mdbooth: Okay, here we go: | |
| 14:12:30 | mhenkel | leakypipes: is the commit msg looking ok now? | |
| 14:12:46 | kashyap | The experiment being: build a backing chain, of depth one, of empty images, perform 'block-stream' and see whether something gets allocated. Three cases: | |
| 14:12:53 | kashyap | (1) With empty images nothing is allocated at all; | |
| 14:13:18 | kashyap | (2) With zeros (using 'qemu-io -c 'write -P 0 0 64M' base.qcow2') written to the base, the qcow2 file for some reason (there's one, but not sure what) grows by a cluster (64 kB). | |
| 14:13:26 | kashyap | (3) With non-zero data (using 'qemu-io -c 'write -P 42 0 64M' base.qcow2'), all data is written to the overlay. | |
| 14:13:45 | kashyap | * * * | |
| 14:13:47 | openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009 | |
| 14:14:17 | dansmith | so mriedem, we supposedly have tests for this ^ and they pass, but the two cases they check don't work in real life | |
| 14:14:21 | kashyap | mdbooth: Max tells (& I can confirm by testing as I have the QMP env. ready) running the same tests on a raw image should yield the same results. | |
| 14:14:23 | dansmith | mriedem: thoughts on how to test that change? | |
| 14:14:27 | openstackgerrit | Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | |
| 14:14:47 | mdbooth | kashyap: Is this dependent on how the target disk is initially created? | |
| 14:15:29 | kashyap | mdbooth: Hmm, that is a good question. The target image in this specific test is a simple qcow2 overlay | |
| 14:16:00 | mdbooth | So, in the case of volume migration the source and destination are both block devices from the pov of qemu | |
| 14:16:16 | edmondsw | mriedem, what is the plan/status on setting send_service_user_token conf setting to True by default (i.e. no longer experimental)? | |
| 14:16:23 | mriedem | dansmith: ooo i smell a customer bug | |
| 14:16:56 | mriedem | dansmith: tests where? novaclient? | |
| 14:17:00 | mdbooth | kashyap: I'd be interested to see the result if you create 2 sparse files with loopback devices, and copy between the loopback devices | |
| 14:17:08 | mdbooth | Is the result sparse? | |
| 14:17:08 | dansmith | mriedem: hang on, linking | |
| 14:17:13 | mriedem | edmondsw: we need some scale testing first, | |
| 14:17:37 | kashyap | Always asks tricky questions on a Friday, mdbooth... | |
| 14:17:58 | mdbooth | kashyap: Hopefully it's just a little reconfiguration of the test you've already got | |
| 14:18:04 | kashyap | mdbooth: I will let you know, once I come back from my bike ride. My eyes are aching | |
| 14:18:04 | mdbooth | I think you've already done the hard work :) | |
| 14:18:05 | mriedem | edmondsw: raj_sing- is the lead on that stuff i believe. we have service user tokens enabled in the nova-next job, but as part of the original spec we had talked about the need for some scale testing to stress it to force the usage of the service user tokens, and to stress keystone since it has to do double duty token-wise per request now. | |
| 14:18:15 | dansmith | mriedem: added a comment with links | |
| 14:18:25 | mriedem | edmondsw: https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/use-service-tokens.html#performance-impact | |
| 14:18:53 | mriedem | "Rally benchmark tests will be ran with and without the “service_user” config options set to compare the results for long running tasks like snapshot or live migration." | |
| 14:19:14 | mriedem | edmondsw: if you have some people familiar with scale testing and rally, it would be helpful to pitch in on the testing there | |
| 14:19:25 | mriedem | i'm not sure if raj_sing- or the osic people are focusing on that right now | |
| 14:19:27 | lbragstad | aunnam around? | |
| 14:20:07 | mriedem | dansmith: i wouldn't trust the unit tests at all, | |
| 14:20:20 | mriedem | dansmith: i'd probably just write a functional test in the novaclient tree that does what you need for how this is supposed to work | |
| 14:20:27 | edmondsw | mriedem that all makes sense. We don't use rally but we do some scale testing. I can ask about trying this there | |
| 14:20:30 | mriedem | see it break on the bug, and then show it passing with the fix | |
| 14:20:30 | dansmith | mriedem: clearly not, the same tests pass before and after this | |
| 14:20:32 | kashyap | mdbooth: Quick clarification, before I head out for a few: IIUC, you want the loopback device to be the overlay, yes? | |
| 14:20:45 | mriedem | dansmith: like what i've been doing with the functional regression tests in nova lately | |
| 14:21:09 | mdbooth | kashyap: I want both source and destination to be loopback devices backed by sparse files | |
| 14:21:22 | mdbooth | kashyap: They should both be the same size | |
| 14:21:34 | mriedem | edmondsw: cool. i'm thinking a long-running volume-backed migration where the token will expire and have to be revalidated for cinder and neutron, | |
| 14:21:34 | mdbooth | kashyap: The source should contain some data, but the dest should initially be blank | |
| 14:21:47 | dansmith | mriedem: okay I forgot about the functional ones in here.. looks like we don't have any for aggregates | |
| 14:21:50 | mriedem | edmondsw: or a long-running snapshot of a large instance (note you'll need the unmerged glance patches for that) | |
| 14:22:27 | mriedem | dansmith: at first glance i'm not seeing anyway | |
| 14:22:33 | mriedem | but they are pretty simple to add | |
| 14:22:56 | mriedem | like https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/v2/legacy/test_extended_attributes.py | |
| 14:22:59 | dansmith | yeah, I'll work on it | |
| 14:23:01 | mriedem | specify the base 2.1 version, | |
| 14:23:05 | edleafe | kashyap: mdbooth: still trying to get the repro for the failure. Didn't give up (yet) :) | |