| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-09 | |||
| 22:21:29 | opendevreview | melanie witt proposed openstack/nova master: api: Log correct client IP if load balancer in use https://review.opendev.org/c/openstack/nova/+/786766 | |
| 22:25:02 | gmann | stephenfin: can you check if anything else you want to add in this spec, leaving +w to you https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| #openstack-nova - 2021-06-10 | |||
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802 | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/695188 | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/695187 | |
| 00:02:37 | opendevmeet | bug 1853009 in OpenStack Compute (nova) ussuri "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard) | |
| 00:02:37 | opendevreview | melanie witt proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/695012 | |
| 00:02:38 | opendevreview | melanie witt proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189 | |
| 06:39:39 | masterpe[m] | I have more resources declined in the placement.allocations table then actual in use. I see some instance ID there that are deleted. | |
| 06:39:56 | masterpe[m] | What shall I do with them? | |
| 06:46:16 | frickler | masterpe[m]: seems this tool was made for you https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement-audit | |
| 06:46:53 | frickler | or maybe the heal-allocations above | |
| 06:52:57 | masterpe[m] | nice thanks | |
| 08:06:14 | lyarwood | kashyap: https://review.opendev.org/c/openstack/nova/+/795533/5#message-3e7b1dc4d3f8d22b2c9a55637c7366199d00eb56 - I'll write this up in a Nova bug and likely libvirt bug later today but would you mind scanning this if you get a chance? | |
| 08:06:37 | kashyap | lyarwood: Mornin; /me clicks | |
| 08:07:30 | kashyap | lyarwood: Ah, you're debugging the informative error, "reason=failed" | |
| 08:07:45 | lyarwood | there's two parts really | |
| 08:07:51 | lyarwood | yeah the reason=failed thing | |
| 08:08:18 | lyarwood | and the way that the call to virDomainMigrateToURI3 from the source doesn't pick up the failure on the dest | |
| 08:09:23 | kashyap | Hmm, yikes | |
| 08:09:25 | lyarwood | I'm assuming that the dest tears down the connection between the two leading to the eventual `unable to connect to server` error | |
| 08:10:09 | lyarwood | instead of the source being told the migration has failed etc | |
| 08:10:47 | kashyap | lyarwood: This is the dest, right: https://zuul.opendev.org/t/openstack/build/f3b829801901417c9310ad5cc5a0e886/log/controller/logs/libvirt/libvirtd_log.txt | |
| 08:10:54 | lyarwood | yeah | |
| 08:11:15 | kashyap | Is it just me or is it loading deadly slow? I just want to pull down the entire raw file | |
| 08:11:50 | lyarwood | yeah these files are pretty large, I typically pull them down now | |
| 08:12:02 | kashyap | My browser is hung here (probably it goes into MBs). Do I clicck on the "View log" to get the raw link? | |
| 08:12:15 | lyarwood | it's awkward as the raw links don't use .gz at the end of the file names | |
| 08:12:17 | kashyap | Maybe I can simply `wget` the above URL - /me tries | |
| 08:12:37 | lyarwood | mv it to a .gz file and either gunzip or let vim unpack them btw | |
| 08:12:40 | kashyap | Right, but it _is_ a .gz file - a new user will discover it after your browser crashes :D | |
| 08:12:53 | kashyap | lyarwood: Yeah; that's become "muscle memory" now | |
| 08:12:54 | lyarwood | browsers can handle .gz | |
| 08:14:08 | kashyap | Oh, sure; sometimes, a very large file just crashed FF for me | |
| 08:15:31 | kashyap | Actual link to `wget` is: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3b/795533/5/check/nova-next/f3b8298/controller/logs/libvirt/libvirtd_log.txt | |
| 08:15:36 | kashyap | 93MB only | |
| 08:32:27 | kashyap | lyarwood: Two quick things: is this reproducible? Or is it the first you noticed? | |
| 08:32:42 | lyarwood | It's the first time I've seen this | |
| 08:32:52 | kashyap | lyarwood: Also, strangely, I don't see the migrateToURI3() failure in the source libvirtd log: | |
| 08:32:57 | kashyap | I pulled the 113MB file from here: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3b/795533/5/check/nova-next/f3b8298/compute1/logs/libvirt/libvirtd_log.txt | |
| 08:33:12 | kashyap | I wonder if it got log-rotated | |
| 08:34:39 | opendevreview | Wenping Song proposed openstack/nova master: Replace all_tenants with all_projects in List Server APIs https://review.opendev.org/c/openstack/nova/+/765311 | |
| 08:34:39 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Update Servers APIs https://review.opendev.org/c/openstack/nova/+/764292 | |
| 08:34:40 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List SG API https://review.opendev.org/c/openstack/nova/+/766726 | |
| 08:34:40 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from Rebuild Server API https://review.opendev.org/c/openstack/nova/+/766380 | |
| 08:34:41 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 08:34:41 | opendevreview | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from Flavor Access APIs https://review.opendev.org/c/openstack/nova/+/767704 | |
| 08:34:42 | opendevreview | Wenping Song proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852 | |
| 08:34:42 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 08:34:44 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862 | |
| 08:34:44 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851 | |
| 08:34:46 | opendevreview | Wenping Song proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329 | |
| 08:38:27 | lyarwood | 93718 2021-06-10 06:41:58.982+0000: 58504: debug : qemuBlockJobProcessEventConcludedCopyAbort:1250 : copy job 'drive-virtio-disk0' on VM 'instance-0000001d' aborted | |
| 08:38:27 | lyarwood | 93717 2021-06-10 06:41:58.982+0000: 58504: debug : qemuBlockJobEventProcessConcluded:1489 : handling job 'drive-virtio-disk0' state '3' newstate '0' | |
| 08:38:35 | lyarwood | looks like the block job failed | |
| 08:39:36 | lyarwood | 93774 2021-06-10 06:41:59.429+0000: 58504: debug : qemuDomainObjSetJobPhase:9291 : Setting 'migration out' phase to 'confirm3_cancelled' | |
| 08:41:08 | kashyap | lyarwood: Yeah; that's a good find | |
| 08:41:47 | kashyap | The thing is - why the connection is refused? I want to think of "firewall", but I don't think that's it | |
| 08:42:02 | lyarwood | didn't we have issues with test_live_block_migration_paused before? | |
| 08:42:07 | lyarwood | kashyap: I don't think it's refused | |
| 08:42:30 | lyarwood | kashyap: the migration just ends and the python libvirt lib is just incorrectly handling the failure | |
| 08:42:32 | kashyap | lyarwood: Oh, right: | |
| 08:42:41 | kashyap | It's because the mirroring was cancelled on the source: | |
| 08:42:43 | kashyap | 2021-06-10 06:41:59.429+0000: 58504: debug : qemuDomainObjSetJobPhase:9291 : Setting 'migration out' phase to 'confirm3_cancelled' | |
| 08:42:43 | kashyap | --- | |
| 08:42:46 | kashyap | 2021-06-10 06:41:59.429+0000: 58504: debug : qemuMigrationEatCookie:1483 : cookielen=0 cookie='<null>' | |
| 08:42:49 | kashyap | 2021-06-10 06:41:59.430+0000: 58504: debug : qemuMigrationSrcNBDCopyCancel:707 : Cancelling drive mirrors for domain instance-0000001d | |
| 08:42:52 | kashyap | 2021-06-10 06:41:59.430+0000: 58504: debug : qemuMigrationSrcNBDCopyCancelled:632 : All disk mirrors are gone | |
| 08:42:55 | kashyap | --- | |
| 08:43:12 | kashyap | Yeah, having it return a graceful error is a reasonable request | |
| 08:43:21 | kashyap | lyarwood: On test_live_block_migration_paused - yes! | |
| 08:43:42 | lyarwood | https://review.opendev.org/c/openstack/nova/+/766720 | |
| 08:44:05 | kashyap | lyarwood: IIRC, it was because it was being tested on shared storage | |
| 08:44:45 | kashyap | lyarwood: Yep; that skip matches my understanding | |
| 08:44:45 | lyarwood | no I think in the end we were told that there were known issues with -drive and told to upgrade to a newer version of QEMU | |
| 08:45:10 | kashyap | So is the 'nova-next' job using Bioninc? | |
| 08:45:15 | lyarwood | I need to head offline for a haircut for ~90mins but I think we need to flag this to the QEMU folks again | |
| 08:45:19 | lyarwood | I think it's focal | |
| 08:45:36 | lyarwood | https://zuul.opendev.org/t/openstack/build/f3b829801901417c9310ad5cc5a0e886/log/zuul-info/inventory.yaml#106 | |
| 08:45:38 | lyarwood | yeah it's focal | |
| 08:45:48 | kashyap | lyarwood: I just mentioned to Dave Gilbert ... need to file a bug | |
| 08:45:54 | lyarwood | kk awesome | |
| 08:45:58 | kashyap | I also need to head out for an errand shortly | |
| 08:45:58 | lyarwood | brb in ~90 | |
| 08:47:01 | bauzas | can anyone give me a short summary about the pep8 gate issue ? saw https://review.opendev.org/c/openstack/nova/+/795533 | |
| 08:47:58 | gibi | bauzas: mypy 0.9 removed some third party type defs from the package, those needs to be installed separately | |
| 08:48:17 | gibi | bauzas: we hit by the missing paramiko typedef after 0.9 | |
| 08:48:37 | bauzas | it was a minor version upgrade from mypy ? woah | |
| 08:49:01 | bauzas | the fix is still -1 from Zuul, not related ? | |
| 08:49:55 | gibi | bauzas: the failing nova-next and nova-grenade-multinode in that patch fails with thing that I saw before on master so I rule them unrelated | |
| 08:49:56 | bauzas | gibi: can we just remove https://github.com/openstack/nova/blob/master/tox.ini#L57 for the moment ? | |
| 08:50:23 | gibi | bauzas: landing that removal need to pass the same test as the patch that is in the check queue | |
| 08:50:37 | gibi | if your tests are unstable then both equally hard to land | |
| 08:51:28 | gibi | s/your/our/ | |
| 08:52:08 | bauzas | gibi: if we would remove https://github.com/openstack/nova/blob/master/tox.ini#L57 then the jobs wouldn't be running | |
| 08:52:32 | gibi | bauzas: removing something from tox ini still triggers nova-next and nova-grenade-multinode isn't it? | |
| 08:52:34 | bauzas | gibi: so, we could fix the gate issue *and then* trying to adding this change | |
| 08:52:53 | bauzas | gibi: good question, AFAIK, I wasn't knowing | |
| 08:53:11 | stephenfin | lyarwood: artom: The reason 'mypy --install-types' wasn't enough is that that requires an existing mypy cache (.mypy_cache), which will only be created if you run mypy. So you'd have to run mypy, wait for it to potentially fail, run '--install-types', then run mypy again | |
| 08:53:20 | bauzas | gibi: lemme try to see this | |