| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-28 | |||
| 08:31:41 | lyarwood | melwitt++ thanks for the ER query and other bug you raised, looks like we have hit *more* issues overnight | |
| 08:33:50 | bauzas | lyarwood: /o\ | |
| 08:35:23 | bauzas | lyarwood: mmmm, the gate looks all good for me | |
| 08:36:47 | lyarwood | https://review.opendev.org/#/c/759831/ check the failures here, a random collection of unrelated failures | |
| 08:37:03 | lyarwood | the gate isn't jammed, jobs are just failing | |
| 08:50:44 | bauzas | lyarwood: I see | |
| 08:50:57 | bauzas | we're just lucky then | |
| 08:57:25 | kashyap | lyarwood: Mornin; will check | |
| 09:01:15 | lyarwood | kashyap: ack, is that streaming somewhere? | |
| 09:01:43 | kashyap | lyarwood: It is per-email-based login; costs $25 | |
| 09:01:52 | kashyap | lyarwood: Yeah, your recollection is right | |
| 09:02:04 | kashyap | BTW, don't worry; the full talks will be available on YouTube soon | |
| 09:02:16 | lyarwood | kk no issues, I'll watch it once it's posted there | |
| 09:02:20 | kashyap | (Even the panel discussion was preprecorded) | |
| 09:04:45 | kashyap | lyarwood: So the test that's failing is live migration _without_ shared storage? | |
| 09:06:17 | frickler | gibi: lyarwood: since https://review.opendev.org/759831 failed once more on unrelated issues, how about force-merging it in order to unblock things? I think we can be pretty sure that the patch itself is working fine by now | |
| 09:06:51 | lyarwood | kashyap: https://opendev.org/openstack/tempest/src/branch/master/tempest/api/compute/admin/test_live_migration.py#L151 | |
| 09:07:12 | lyarwood | frickler: I'd be happy with that if it unblocks everyone, gibi should have the final say as PTL however. | |
| 09:07:39 | lyarwood | I'm going to spend some time this morning going over the other failures and making sure we have bugs / ER queries etc for each | |
| 09:09:21 | lyarwood | kashyap: hmm why is this even listed as live block migration? | |
| 09:09:26 | lyarwood | kashyap: it's just a paused instance | |
| 09:09:51 | kashyap | lyarwood: So it is migrating a paused instance but _with_ storage, isn't it? | |
| 09:10:18 | lyarwood | kashyap: yeah, it's just making assumptions about the env that storage isn't shared | |
| 09:10:37 | lyarwood | I guess that's what block_migration_for_live_migration is checking | |
| 09:10:37 | kashyap | Aah, so it might well be a shared storage setup, but the test is dubious | |
| 09:11:06 | kashyap | I wonder if that's why the NBD copy was cancelled - it must've detected "hey, it's shared storage, no need for copy, hence I'm cancelling it" | |
| 09:11:33 | lyarwood | kashyap: no I don't think the storage is in this job | |
| 09:11:42 | kashyap | So, it is non-shared storage? | |
| 09:11:57 | lyarwood | yeah sorry I'm just confusing the situation | |
| 09:12:16 | kashyap | No prob | |
| 09:12:23 | lyarwood | oh you know what this is | |
| 09:12:49 | lyarwood | it's the back and forth LM of a paused instance issue we saw downstream | |
| 09:13:28 | kashyap | Ohh, wait - that rings a faint bell | |
| 09:13:46 | lyarwood | 2020-10-27 16:59:16.022725 | primary | 2020-10-27 16:59:16.022 | 2020-10-27 16:59:06,664 22654 INFO [tempest.api.compute.admin.test_live_migration] Live migrate back to source ubuntu-bionic-rax-iad-0021082674 | |
| 09:13:54 | kashyap | IIRC, we ruled it out as a bogus test? | |
| 09:14:05 | lyarwood | yeah the QEMU devs said it was fubar | |
| 09:14:36 | kashyap | Do you recall that other bug no. by any chance, from your browser history? | |
| 09:14:48 | kashyap | Hang on, I found it | |
| 09:15:02 | kashyap | Because I recall "fixing" the bug description myself :D | |
| 09:15:07 | lyarwood | haha | |
| 09:21:19 | kashyap | lyarwood: https://bugzilla.redhat.com/show_bug.cgi?id=1713009 | |
| 09:21:19 | openstack | bugzilla.redhat.com bug 1713009 in qemu-kvm "[QEMU] Back-n-forth live migration of a paused VM results in QEMU crash with: "bdrv_inactivate_recurse: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed."" [Unspecified,New] - Assigned to kwolf | |
| 09:21:19 | kashyap | "Back-n-forth live migration of a paused VM results in QEMU crash ..." | |
| 09:21:19 | lyarwood | you know what, I didn't even check the QEMU logs for this failure upstream | |
| 09:21:20 | kashyap | If we see the QEMU crash, then it's this. Let's see | |
| 09:21:20 | lyarwood | I don't see this | |
| 09:21:20 | kashyap | 1 sec, on the phone | |
| 09:24:21 | kashyap | lyarwood: So the libvirtd log in the bug is from destination host, yeah? | |
| 09:24:39 | kashyap | (I think so, based on the "migrate-incoming" bit in the log) | |
| 09:25:41 | lyarwood | kashyap: yeah | |
| 09:28:44 | kashyap | lyarwood: So, to recap: should we care about this test that is "invalid" - what is the point of doing back-n-forth LM of a paused instance? | |
| 09:28:59 | kashyap | Also, have you got the QEMU command-line of the instance log in question? /me goes searching | |
| 09:31:17 | kashyap | Hm, which of the instances is it here - https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_30d/759831/4/gate/nova-grenade-multinode/30d8eb1/logs/libvirt/qemu/index.html | |
| 09:31:33 | lyarwood | instance-0000001e | |
| 09:31:46 | kashyap | Okay, I looked at the "right one", then | |
| 09:32:05 | kashyap | (Also, the "-incoming defer" confirms it's the dest host) | |
| 09:32:57 | kashyap | lyarwood: It's also in the instance log, as you surely must know - 4d564e22-8ba4-48fb-ac93-27bea660fd77 | |
| 09:34:39 | lyarwood | kashyap: right my point was that the log should use the UUID as the filename, IOW we should use it as the domain name | |
| 09:34:54 | lyarwood | kashyap: but I think there's an upgrade issue with that so we've never tried | |
| 09:34:56 | kashyap | Ah, guessed as much; consistency across the board | |
| 09:35:33 | kashyap | lyarwood: Source host logs are under subnode-2/ dir, right? | |
| 09:35:42 | kashyap | Yep | |
| 09:44:58 | kashyap | lyarwood: So, I collated the useful bits from the logs here, to check w/ the QEMU/libvirt folks - https://kashyapc.fedorapeople.org/LP-1901739/ | |
| 09:47:00 | lyarwood | ack thanks | |
| 09:48:43 | kashyap | lyarwood: So, is this an apt descrption: back-n-forth live migration (w/o shared storage) of a paused instance causes this missing block job data? | |
| 09:50:28 | lyarwood | frickler: any idea how your last recheck of https://review.opendev.org/#/c/759831/ has ended up with it in both the check and gate queues at the same time? | |
| 09:53:31 | frickler | lyarwood: I manually enqueued it into the gate directly | |
| 09:53:38 | lyarwood | frickler: ah my bad | |
| 09:54:10 | frickler | lyarwood: I did the recheck because I didn't see the earlier ones, only the -2 from gate | |
| 10:08:39 | kashyap | lyarwood: BTW, ignore my earlier wondering about the use-case for back-n-forth LM of a paused guest. I just thought of one plausible case: | |
| 10:08:52 | kashyap | You migrate a paused a instance; it succeeds. But then you suddenly realize: "oh, I actually need to do some maint work on this dest, so I need to migrate this VM _back_ to the source again" | |
| 10:18:44 | lyarwood | frickler: ack no issues, I was just confused how it ended up in both queues at the same time | |
| 10:29:02 | gibi | frickler: I have no objection force merging https://review.opendev.org/759831 | |
| 10:40:37 | openstackgerrit | Merged openstack/nova master: libvirt: Only ask tpool.Proxy to autowrap vir* classes https://review.opendev.org/759831 | |
| 10:41:27 | frickler | gibi: ^^ done | |
| 10:46:42 | lyarwood | many thanks | |
| 10:51:15 | gibi | frickler: thank you | |
| 10:52:49 | lyarwood | stephenfin: https://review.opendev.org/#/c/758971/ updated btw | |
| 10:53:15 | lyarwood | gibi: https://review.opendev.org/#/q/topic:bug/1899649 would you mind taking a look at this if you have time ahead of the ptg | |
| 10:56:29 | stephenfin | lyarwood: +2; thanks for that | |
| 11:01:02 | gibi | lyarwood: sure, looking | |
| 11:05:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tox: Stop linting as part of docs target https://review.opendev.org/735176 | |
| 11:10:01 | stephenfin | lyarwood: Think you could take a gawk at https://review.opendev.org/#/q/topic:bug/1898272+(status:open+OR+status:merged) too? | |
| 11:10:48 | lyarwood | stephenfin: sure | |
| 11:25:07 | gibi | lyarwood: I left a question in https://review.opendev.org/#/c/757894/2/nova/compute/manager.py@a2233 | |
| 11:27:20 | kashyap | gibi: [Maybe I'll get kicked off this channel for sayin it] Speaking of caffeine, I switched to decaf-only after 10:00 AM (for my afternoon coffee. | |
| 11:28:05 | kashyap | gibi: Because of a certain sleep scientist; check this 2-min msg from him: https://www.youtube.com/watch?v=KGfdR7TSJo4 | |
| 11:28:29 | gibi | kashyap: we are inclusive, no kicks due to decaf consumption :) | |
| 11:28:37 | kashyap | Haha | |
| 11:28:39 | kashyap | gibi: "If you have a cup of coffee at noon, then a quarter of that caffeine is still in your brain at _midnight_." | |
| 11:28:46 | kashyap | But --^ that was startling to me | |
| 11:29:23 | kashyap | (That said, PTG demands more caffeine) | |
| 11:31:38 | gibi | now I'm wondering about 5 o'clock tea too | |
| 11:32:14 | gibi | for me tea has a lower but more elongated effect | |
| 11:32:27 | kashyap | Yeah, I've been decaf in the afternoon for 4 days in a row. And my sleep is steadily improving already | |
| 11:33:31 | kashyap | gibi: BTW, his book (https://www.penguin.co.uk/books/295/295665/why-we-sleep/9780141983769.html) that is referenced earlier is excellent. (I've also suggested it to Eric Fried in the past, he said he thoroughly enojyed it, FWIW) | |
| 11:33:45 | lyarwood | gibi: ack thanks looking | |
| 11:36:25 | gibi | kashyap: thanks | |
| 11:57:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Clean up base migration https://review.opendev.org/758155 | |
| 11:57:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Merge in unique constraints https://review.opendev.org/758156 | |