| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 08:34:39 | opendevreview | Wenping Song proposed openstack/nova master: Replace all_tenants with all_projects in List Server APIs https://review.opendev.org/c/openstack/nova/+/765311 | |
| 08:34:39 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Update Servers APIs https://review.opendev.org/c/openstack/nova/+/764292 | |
| 08:34:40 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List SG API https://review.opendev.org/c/openstack/nova/+/766726 | |
| 08:34:40 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from Rebuild Server API https://review.opendev.org/c/openstack/nova/+/766380 | |
| 08:34:41 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 08:34:41 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from Flavor Access APIs https://review.opendev.org/c/openstack/nova/+/767704 | |
| 08:34:42 | opendevreview | Wenping Song proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852 | |
| 08:34:42 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 08:34:44 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862 | |
| 08:34:44 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851 | |
| 08:34:46 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329 | |
| 08:38:27 | lyarwood | 93718 2021-06-10 06:41:58.982+0000: 58504: debug : qemuBlockJobProcessEventConcludedCopyAbort:1250 : copy job 'drive-virtio-disk0' on VM 'instance-0000001d' aborted | |
| 08:38:27 | lyarwood | 93717 2021-06-10 06:41:58.982+0000: 58504: debug : qemuBlockJobEventProcessConcluded:1489 : handling job 'drive-virtio-disk0' state '3' newstate '0' | |
| 08:38:35 | lyarwood | looks like the block job failed | |
| 08:39:36 | lyarwood | 93774 2021-06-10 06:41:59.429+0000: 58504: debug : qemuDomainObjSetJobPhase:9291 : Setting 'migration out' phase to 'confirm3_cancelled' | |
| 08:41:08 | kashyap | lyarwood: Yeah; that's a good find | |
| 08:41:47 | kashyap | The thing is - why the connection is refused? I want to think of "firewall", but I don't think that's it | |
| 08:42:02 | lyarwood | didn't we have issues with test_live_block_migration_paused before? | |
| 08:42:07 | lyarwood | kashyap: I don't think it's refused | |
| 08:42:30 | lyarwood | kashyap: the migration just ends and the python libvirt lib is just incorrectly handling the failure | |
| 08:42:32 | kashyap | lyarwood: Oh, right: | |
| 08:42:41 | kashyap | It's because the mirroring was cancelled on the source: | |
| 08:42:43 | kashyap | 2021-06-10 06:41:59.429+0000: 58504: debug : qemuDomainObjSetJobPhase:9291 : Setting 'migration out' phase to 'confirm3_cancelled' | |
| 08:42:43 | kashyap | --- | |
| 08:42:46 | kashyap | 2021-06-10 06:41:59.429+0000: 58504: debug : qemuMigrationEatCookie:1483 : cookielen=0 cookie='<null>' | |
| 08:42:49 | kashyap | 2021-06-10 06:41:59.430+0000: 58504: debug : qemuMigrationSrcNBDCopyCancel:707 : Cancelling drive mirrors for domain instance-0000001d | |
| 08:42:52 | kashyap | 2021-06-10 06:41:59.430+0000: 58504: debug : qemuMigrationSrcNBDCopyCancelled:632 : All disk mirrors are gone | |
| 08:42:55 | kashyap | --- | |
| 08:43:12 | kashyap | Yeah, having it return a graceful error is a reasonable request | |
| 08:43:21 | kashyap | lyarwood: On test_live_block_migration_paused - yes! | |
| 08:43:42 | lyarwood | https://review.opendev.org/c/openstack/nova/+/766720 | |
| 08:44:05 | kashyap | lyarwood: IIRC, it was because it was being tested on shared storage | |
| 08:44:45 | kashyap | lyarwood: Yep; that skip matches my understanding | |
| 08:44:45 | lyarwood | no I think in the end we were told that there were known issues with -drive and told to upgrade to a newer version of QEMU | |
| 08:45:10 | kashyap | So is the 'nova-next' job using Bioninc? | |
| 08:45:15 | lyarwood | I need to head offline for a haircut for ~90mins but I think we need to flag this to the QEMU folks again | |
| 08:45:19 | lyarwood | I think it's focal | |
| 08:45:36 | lyarwood | https://zuul.opendev.org/t/openstack/build/f3b829801901417c9310ad5cc5a0e886/log/zuul-info/inventory.yaml#106 | |
| 08:45:38 | lyarwood | yeah it's focal | |
| 08:45:48 | kashyap | lyarwood: I just mentioned to Dave Gilbert ... need to file a bug | |
| 08:45:54 | lyarwood | kk awesome | |
| 08:45:58 | kashyap | I also need to head out for an errand shortly | |
| 08:45:58 | lyarwood | brb in ~90 | |
| 08:47:01 | bauzas | can anyone give me a short summary about the pep8 gate issue ? saw https://review.opendev.org/c/openstack/nova/+/795533 | |
| 08:47:58 | gibi | bauzas: mypy 0.9 removed some third party type defs from the package, those needs to be installed separately | |
| 08:48:17 | gibi | bauzas: we hit by the missing paramiko typedef after 0.9 | |
| 08:48:37 | bauzas | it was a minor version upgrade from mypy ? woah | |
| 08:49:01 | bauzas | the fix is still -1 from Zuul, not related ? | |
| 08:49:55 | gibi | bauzas: the failing nova-next and nova-grenade-multinode in that patch fails with thing that I saw before on master so I rule them unrelated | |
| 08:49:56 | bauzas | gibi: can we just remove https://github.com/openstack/nova/blob/master/tox.ini#L57 for the moment ? | |
| 08:50:23 | gibi | bauzas: landing that removal need to pass the same test as the patch that is in the check queue | |
| 08:50:37 | gibi | if your tests are unstable then both equally hard to land | |
| 08:51:28 | gibi | s/your/our/ | |
| 08:52:08 | bauzas | gibi: if we would remove https://github.com/openstack/nova/blob/master/tox.ini#L57 then the jobs wouldn't be running | |
| 08:52:32 | gibi | bauzas: removing something from tox ini still triggers nova-next and nova-grenade-multinode isn't it? | |
| 08:52:34 | bauzas | gibi: so, we could fix the gate issue *and then* trying to adding this change | |
| 08:52:53 | bauzas | gibi: good question, AFAIK, I wasn't knowing | |
| 08:53:11 | stephenfin | lyarwood: artom: The reason 'mypy --install-types' wasn't enough is that that requires an existing mypy cache (.mypy_cache), which will only be created if you run mypy. So you'd have to run mypy, wait for it to potentially fail, run '--install-types', then run mypy again | |
| 08:53:20 | bauzas | gibi: lemme try to see this | |
| 08:53:32 | stephenfin | It worked locally because I had the cache already, but failed in the gate because it's a new env | |
| 08:53:51 | bauzas | gibi: stephenfin: so we could just remove mypy first, then trying https://review.opendev.org/c/openstack/nova/+/795533 to be merged, and then adding again mypy | |
| 08:54:18 | gibi | bauzas: go ahead | |
| 08:54:27 | bauzas | doing it now | |
| 08:54:32 | stephenfin | why can't https://review.opendev.org/c/openstack/nova/+/795533 merge? | |
| 08:54:37 | stephenfin | the requirements change has merged | |
| 08:54:51 | stephenfin | and that's the correct fix | |
| 08:55:25 | gibi | I don't want to block bauzas to try another angle. If nova-next and nova-grenade-mutlinode does not trigger a tox.ini change then we might faster land the tox.ini change than the requirement change. | |
| 08:55:40 | gibi | s/trigger a/ trigger on a/ | |
| 08:55:55 | stephenfin | oh, I'm possibly missing context. Is there another issue now? | |
| 08:56:17 | stephenfin | i.e. with nova-next and nova-grenade-multinode ? | |
| 08:56:24 | gibi | we needed to recheck the nova requirement patch couple of times already as it always hit someting either in those jobvs | |
| 08:56:42 | bauzas | stephenfin: the problem is that we run lots of jobs | |
| 08:56:45 | gibi | totally unrealted problems in unstable tests | |
| 08:56:45 | stephenfin | do we have a fix for those jobs? | |
| 08:56:51 | gibi | I don't think so | |
| 08:57:05 | bauzas | stephenfin: so I'm trying to see whether we could just remove the issue without running all of them | |
| 08:58:03 | kashyap | lyarwood: Oh, BTW -- when you're back: the "cancelled" in the QEMU logs has a special meaning for NBD. I forgot that I documented this myself upstream :D | |
| 08:58:20 | kashyap | lyarwood: See step (4) here: https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html#qmp-invocation-for-live-storage-migration-with-drive-mirror-nbd | |
| 08:58:28 | bauzas | gibi: stephenfin: I guess we have a bug ? | |
| 08:58:49 | gibi | stephenfin: we have abou 10% failing rate on master in those two jobs | |
| 08:59:05 | opendevreview | Victor Coutellier proposed openstack/nova master: Allow configuration of direct-snapshot feature https://review.opendev.org/c/openstack/nova/+/794837 | |
| 08:59:37 | kashyap | lyarwood: In short, once the mirroring from src --> dest completes, and the _READY event is emitted, source libvirtd issues QMP `block-job-cancel` to gracefully end the mirroring. | |
| 09:00:11 | gibi | so I guess we are just unlucky with the req patch | |
| 09:00:38 | stephenfin | I think so | |
| 09:01:24 | stephenfin | fwiw though I wouldn't be in favour of simply dropping the requirements fix so we can merge other stuff, if the reason the requirements patch is failing is unrelated | |
| 09:01:56 | opendevreview | Sylvain Bauza proposed openstack/nova master: Removing mypy to fix the nova CI https://review.opendev.org/c/openstack/nova/+/795744 | |
| 09:01:57 | gibi | stephenfin: not dropping the req fix | |
| 09:02:08 | gibi | stephenfin: if it lands then we are done | |
| 09:02:21 | gibi | if bauzas's patch lands first, then we revert that when yours land | |
| 09:02:33 | bauzas | gibi: stephenfin: patch is up for just removing mypy run until we fix the requirements | |
| 09:02:39 | stephenfin | that makes no sense to me though | |
| 09:02:46 | stephenfin | the mypy run is causing the other failures | |
| 09:02:48 | stephenfin | *isn't | |
| 09:03:01 | bauzas | stephenfin: sure | |
| 09:03:08 | stephenfin | so they have an equal chance of failing randomly | |
| 09:03:23 | bauzas | I'm just proposing this one because I guess nova-next WONT be running on my patch | |
| 09:03:35 | bauzas | it's just a tox.ini change | |
| 09:03:39 | gibi | stephenfin: not equal chance iff the tox.ini change does not trigger the unstable jobs | |
| 09:03:39 | stephenfin | but what does this achieve? | |
| 09:03:56 | gibi | stephenfin: if it trigger the same job, then I agree that it mypy removal patch is pointless | |