| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 08:09:25 | lyarwood | I'm assuming that the dest tears down the connection between the two leading to the eventual `unable to connect to server` error | |
| 08:10:09 | lyarwood | instead of the source being told the migration has failed etc | |
| 08:10:47 | kashyap | lyarwood: This is the dest, right: https://zuul.opendev.org/t/openstack/build/f3b829801901417c9310ad5cc5a0e886/log/controller/logs/libvirt/libvirtd_log.txt | |
| 08:10:54 | lyarwood | yeah | |
| 08:11:15 | kashyap | Is it just me or is it loading deadly slow? I just want to pull down the entire raw file | |
| 08:11:50 | lyarwood | yeah these files are pretty large, I typically pull them down now | |
| 08:12:02 | kashyap | My browser is hung here (probably it goes into MBs). Do I clicck on the "View log" to get the raw link? | |
| 08:12:15 | lyarwood | it's awkward as the raw links don't use .gz at the end of the file names | |
| 08:12:17 | kashyap | Maybe I can simply `wget` the above URL - /me tries | |
| 08:12:37 | lyarwood | mv it to a .gz file and either gunzip or let vim unpack them btw | |
| 08:12:40 | kashyap | Right, but it _is_ a .gz file - a new user will discover it after your browser crashes :D | |
| 08:12:53 | kashyap | lyarwood: Yeah; that's become "muscle memory" now | |
| 08:12:54 | lyarwood | browsers can handle .gz | |
| 08:14:08 | kashyap | Oh, sure; sometimes, a very large file just crashed FF for me | |
| 08:15:31 | kashyap | Actual link to `wget` is: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3b/795533/5/check/nova-next/f3b8298/controller/logs/libvirt/libvirtd_log.txt | |
| 08:15:36 | kashyap | 93MB only | |
| 08:32:27 | kashyap | lyarwood: Two quick things: is this reproducible? Or is it the first you noticed? | |
| 08:32:42 | lyarwood | It's the first time I've seen this | |
| 08:32:52 | kashyap | lyarwood: Also, strangely, I don't see the migrateToURI3() failure in the source libvirtd log: | |
| 08:32:57 | kashyap | I pulled the 113MB file from here: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3b/795533/5/check/nova-next/f3b8298/compute1/logs/libvirt/libvirtd_log.txt | |
| 08:33:12 | kashyap | I wonder if it got log-rotated | |
| 08:34:39 | opendevreview | Wenping Song proposed openstack/nova master: Replace all_tenants with all_projects in List Server APIs https://review.opendev.org/c/openstack/nova/+/765311 | |
| 08:34:39 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Update Servers APIs https://review.opendev.org/c/openstack/nova/+/764292 | |
| 08:34:40 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List SG API https://review.opendev.org/c/openstack/nova/+/766726 | |
| 08:34:40 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from Rebuild Server API https://review.opendev.org/c/openstack/nova/+/766380 | |
| 08:34:41 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 08:34:41 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from Flavor Access APIs https://review.opendev.org/c/openstack/nova/+/767704 | |
| 08:34:42 | opendevreview | Wenping Song proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852 | |
| 08:34:42 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 08:34:44 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862 | |
| 08:34:44 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851 | |
| 08:34:46 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329 | |
| 08:38:27 | lyarwood | 93718 2021-06-10 06:41:58.982+0000: 58504: debug : qemuBlockJobProcessEventConcludedCopyAbort:1250 : copy job 'drive-virtio-disk0' on VM 'instance-0000001d' aborted | |
| 08:38:27 | lyarwood | 93717 2021-06-10 06:41:58.982+0000: 58504: debug : qemuBlockJobEventProcessConcluded:1489 : handling job 'drive-virtio-disk0' state '3' newstate '0' | |
| 08:38:35 | lyarwood | looks like the block job failed | |
| 08:39:36 | lyarwood | 93774 2021-06-10 06:41:59.429+0000: 58504: debug : qemuDomainObjSetJobPhase:9291 : Setting 'migration out' phase to 'confirm3_cancelled' | |
| 08:41:08 | kashyap | lyarwood: Yeah; that's a good find | |
| 08:41:47 | kashyap | The thing is - why the connection is refused? I want to think of "firewall", but I don't think that's it | |
| 08:42:02 | lyarwood | didn't we have issues with test_live_block_migration_paused before? | |
| 08:42:07 | lyarwood | kashyap: I don't think it's refused | |
| 08:42:30 | lyarwood | kashyap: the migration just ends and the python libvirt lib is just incorrectly handling the failure | |
| 08:42:32 | kashyap | lyarwood: Oh, right: | |
| 08:42:41 | kashyap | It's because the mirroring was cancelled on the source: | |
| 08:42:43 | kashyap | 2021-06-10 06:41:59.429+0000: 58504: debug : qemuDomainObjSetJobPhase:9291 : Setting 'migration out' phase to 'confirm3_cancelled' | |
| 08:42:43 | kashyap | --- | |
| 08:42:46 | kashyap | 2021-06-10 06:41:59.429+0000: 58504: debug : qemuMigrationEatCookie:1483 : cookielen=0 cookie='<null>' | |
| 08:42:49 | kashyap | 2021-06-10 06:41:59.430+0000: 58504: debug : qemuMigrationSrcNBDCopyCancel:707 : Cancelling drive mirrors for domain instance-0000001d | |
| 08:42:52 | kashyap | 2021-06-10 06:41:59.430+0000: 58504: debug : qemuMigrationSrcNBDCopyCancelled:632 : All disk mirrors are gone | |
| 08:42:55 | kashyap | --- | |
| 08:43:12 | kashyap | Yeah, having it return a graceful error is a reasonable request | |
| 08:43:21 | kashyap | lyarwood: On test_live_block_migration_paused - yes! | |
| 08:43:42 | lyarwood | https://review.opendev.org/c/openstack/nova/+/766720 | |
| 08:44:05 | kashyap | lyarwood: IIRC, it was because it was being tested on shared storage | |
| 08:44:45 | kashyap | lyarwood: Yep; that skip matches my understanding | |
| 08:44:45 | lyarwood | no I think in the end we were told that there were known issues with -drive and told to upgrade to a newer version of QEMU | |
| 08:45:10 | kashyap | So is the 'nova-next' job using Bioninc? | |
| 08:45:15 | lyarwood | I need to head offline for a haircut for ~90mins but I think we need to flag this to the QEMU folks again | |
| 08:45:19 | lyarwood | I think it's focal | |
| 08:45:36 | lyarwood | https://zuul.opendev.org/t/openstack/build/f3b829801901417c9310ad5cc5a0e886/log/zuul-info/inventory.yaml#106 | |
| 08:45:38 | lyarwood | yeah it's focal | |
| 08:45:48 | kashyap | lyarwood: I just mentioned to Dave Gilbert ... need to file a bug | |
| 08:45:54 | lyarwood | kk awesome | |
| 08:45:58 | kashyap | I also need to head out for an errand shortly | |
| 08:45:58 | lyarwood | brb in ~90 | |
| 08:47:01 | bauzas | can anyone give me a short summary about the pep8 gate issue ? saw https://review.opendev.org/c/openstack/nova/+/795533 | |
| 08:47:58 | gibi | bauzas: mypy 0.9 removed some third party type defs from the package, those needs to be installed separately | |
| 08:48:17 | gibi | bauzas: we hit by the missing paramiko typedef after 0.9 | |
| 08:48:37 | bauzas | it was a minor version upgrade from mypy ? woah | |
| 08:49:01 | bauzas | the fix is still -1 from Zuul, not related ? | |
| 08:49:55 | gibi | bauzas: the failing nova-next and nova-grenade-multinode in that patch fails with thing that I saw before on master so I rule them unrelated | |
| 08:49:56 | bauzas | gibi: can we just remove https://github.com/openstack/nova/blob/master/tox.ini#L57 for the moment ? | |
| 08:50:23 | gibi | bauzas: landing that removal need to pass the same test as the patch that is in the check queue | |
| 08:50:37 | gibi | if your tests are unstable then both equally hard to land | |
| 08:51:28 | gibi | s/your/our/ | |
| 08:52:08 | bauzas | gibi: if we would remove https://github.com/openstack/nova/blob/master/tox.ini#L57 then the jobs wouldn't be running | |
| 08:52:32 | gibi | bauzas: removing something from tox ini still triggers nova-next and nova-grenade-multinode isn't it? | |
| 08:52:34 | bauzas | gibi: so, we could fix the gate issue *and then* trying to adding this change | |
| 08:52:53 | bauzas | gibi: good question, AFAIK, I wasn't knowing | |
| 08:53:11 | stephenfin | lyarwood: artom: The reason 'mypy --install-types' wasn't enough is that that requires an existing mypy cache (.mypy_cache), which will only be created if you run mypy. So you'd have to run mypy, wait for it to potentially fail, run '--install-types', then run mypy again | |
| 08:53:20 | bauzas | gibi: lemme try to see this | |
| 08:53:32 | stephenfin | It worked locally because I had the cache already, but failed in the gate because it's a new env | |
| 08:53:51 | bauzas | gibi: stephenfin: so we could just remove mypy first, then trying https://review.opendev.org/c/openstack/nova/+/795533 to be merged, and then adding again mypy | |
| 08:54:18 | gibi | bauzas: go ahead | |
| 08:54:27 | bauzas | doing it now | |
| 08:54:32 | stephenfin | why can't https://review.opendev.org/c/openstack/nova/+/795533 merge? | |
| 08:54:37 | stephenfin | the requirements change has merged | |
| 08:54:51 | stephenfin | and that's the correct fix | |
| 08:55:25 | gibi | I don't want to block bauzas to try another angle. If nova-next and nova-grenade-mutlinode does not trigger a tox.ini change then we might faster land the tox.ini change than the requirement change. | |
| 08:55:40 | gibi | s/trigger a/ trigger on a/ | |
| 08:55:55 | stephenfin | oh, I'm possibly missing context. Is there another issue now? | |
| 08:56:17 | stephenfin | i.e. with nova-next and nova-grenade-multinode ? | |
| 08:56:24 | gibi | we needed to recheck the nova requirement patch couple of times already as it always hit someting either in those jobvs | |
| 08:56:42 | bauzas | stephenfin: the problem is that we run lots of jobs | |
| 08:56:45 | gibi | totally unrealted problems in unstable tests | |
| 08:56:45 | stephenfin | do we have a fix for those jobs? | |
| 08:56:51 | gibi | I don't think so | |
| 08:57:05 | bauzas | stephenfin: so I'm trying to see whether we could just remove the issue without running all of them | |
| 08:58:03 | kashyap | lyarwood: Oh, BTW -- when you're back: the "cancelled" in the QEMU logs has a special meaning for NBD. I forgot that I documented this myself upstream :D | |
| 08:58:20 | kashyap | lyarwood: See step (4) here: https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html#qmp-invocation-for-live-storage-migration-with-drive-mirror-nbd | |
| 08:58:28 | bauzas | gibi: stephenfin: I guess we have a bug ? | |