Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-20
13:25:17 kashyap lyarwood: Yes, the idea we discussed in the past was to listen on:
13:25:36 kashyap _BLOCK_JOB_READY events that QEMU (which in turn libvirt also relays) emits once a copy job has synced
13:26:04 kashyap lyarwood: I recall writin up that design somewhere, need to dig; but buried in a few downstream tasks before I go offline tomm & Fri
13:26:27 kashyap Give me a few, I'll find the relevant pointers :)
13:26:29 lyarwood ack np
13:26:33 lyarwood thanks
13:29:06 kashyap lyarwood: Context-1: the thorny problem CI in Nova identified was this write-up: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html ("Debugging blockRebase() - "active block copy not ready for pivot")
13:29:38 kashyap lyarwood: Context-2: from the above, I filed this libvirt bug in 2016, and is fixed (in Jan-2019): https://bugzilla.redhat.com/show_bug.cgi?id=1382165
13:29:38 openstack bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa
13:31:45 kashyap lyarwood: Context-3: https://libvirt.org/git/?p=libvirt.git;a=commit;h=73ce3911a
13:31:49 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595
13:31:49 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596
13:31:53 kashyap (There "some clients" is referring to OpenStack :-))
13:32:25 kashyap So yeah, in short: what's needed is to listen on VIR_DOMAIN_BLOCK_JOB_READY event.
13:32:49 kashyap Instead of polling (to avoid the race that we debugged in the Context-1 link)
13:34:23 lyarwood https://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainEventBlockJobCallback using this I assume?
13:34:32 lyarwood we don't want to listen over QMP
13:35:01 kashyap lyarwood: No, we're not doing over QMP, of course
13:35:17 kashyap libvirt itself emits events; there's even an XML flag for the 'ready' bit
13:35:43 lyarwood kk
13:35:52 kashyap (The actual thing is being transmitted over QMP anyway; but we don't manually do it; that's what I meant)
13:36:52 lyarwood yup understood thanks
13:37:16 kashyap And check out this bit (read it in full in the URL): "The attribute 'ready', if present, tracks progress of the job [...]": https://libvirt.org/formatdomain.html
13:37:26 kashyap [Under the 'mirror' documentation]
13:39:29 lyarwood kashyap: yeah we already poll on that
13:40:21 kashyap Ah, nod. /me bbiab; need to finish this other bit early.
13:40:26 lyarwood kashyap: I added that a while ago https://review.opendev.org/#/c/399735/
13:40:29 kashyap (But I reviewed your clean-up; looks good to me)
13:40:54 lyarwood ack thanks
13:41:06 kashyap lyarwood: Oh, yea; now I recall it. Thx
14:29:43 lyarwood artom / melwitt / stephenfin ; would you all mind taking a swing at https://review.opendev.org/#/c/631294/ sometime this week?
14:30:46 artom lyarwood, ack
14:31:55 lyarwood thanks, it's a total pig thanks to the backport requirement but all I can do at the moment
14:32:28 openstackgerrit Balazs Gibizer proposed openstack/nova-specs master: template: consider openstack client besides novaclient https://review.opendev.org/717722
14:47:53 stephenfin lyarwood: ack, I was waiting on the WIP patches theren ow
14:47:54 stephenfin *now
15:47:39 jsuchome lyarwood: fyi, I missed broken tests, will have to rewrite it once more...
15:50:15 lyarwood jsuchome: ack np I was waiting on CI for that very reason :)
15:54:49 jsuchome I accidentally ran wrong tests localy and was surprised it was passing :-)
16:05:29 openstackgerrit Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301
16:11:52 bauzas gibi: fwiw, I'll be off tomorrow and Friday
16:12:19 bauzas May the Force be with French folks
16:15:20 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove 'NovaObjectDictCompat' from 'Migration' https://review.opendev.org/723572
16:15:21 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove 'NovaObjectDictCompat' from 'InstancePCIRequest' https://review.opendev.org/723573
16:16:59 openstackgerrit Merged openstack/nova stable/stein: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/726862
16:17:18 openstackgerrit Merged openstack/nova master: Add admin doc information about image cache resource accounting https://review.opendev.org/729350
16:17:33 openstackgerrit Merged openstack/nova master: Increase code reuse in test_numa_servers https://review.opendev.org/714125
16:30:21 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595
16:30:22 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596
16:33:16 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Remove log translations https://review.opendev.org/728966
16:33:16 openstackgerrit Stephen Finucane proposed openstack/nova master: hacking: Modify checks for translated logs https://review.opendev.org/728967
16:33:17 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove hooks https://review.opendev.org/728968
16:33:17 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Remove workaround for unsupported libvirt version https://review.opendev.org/729656
16:52:52 stephenfin lyarwood: one request on https://review.opendev.org/#/c/631294/ to keep me happy
16:56:05 lyarwood stephenfin: ack thanks
17:16:55 openstackgerrit Lee Yarwood proposed openstack/nova master: Add functional test for bug 1550919 https://review.opendev.org/631294
17:16:55 openstack bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Lee Yarwood (lyarwood)
17:17:25 lyarwood stephenfin: ^ thanks again, I'll comment on the fix in the morning
17:52:51 openstackgerrit Elod Illes proposed openstack/nova stable/stein: Reproduce bug 1869050 https://review.opendev.org/729530
17:52:51 openstack bug 1869050 in OpenStack Compute (nova) stein "migration of anti-affinity server fails due to stale scheduler instance info" [Low,In progress] https://launchpad.net/bugs/1869050 - Assigned to Balazs Gibizer (balazs-gibizer)
18:03:32 openstackgerrit Elod Illes proposed openstack/nova stable/stein: Update scheduler instance info at confirm resize https://review.opendev.org/729538
18:18:57 openstackgerrit Elod Illes proposed openstack/nova stable/train: Make quotas respect instance_list_per_project_cells https://review.opendev.org/729087
19:57:10 openstackgerrit Andreas Jaeger proposed openstack/nova master: Fix pygments_style https://review.opendev.org/729747
19:57:35 melwitt thanks
20:09:27 openstackgerrit Merged openstack/nova master: trivial: Remove remaining '_LW' instances https://review.opendev.org/728964
21:15:51 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763
#openstack-nova - 2020-05-21
04:32:07 openstackgerrit Nalini Varshney proposed openstack/nova master: Add migration to make key field type VARBINARY in aggregate_metadata table, https://review.opendev.org/725522
05:07:33 openstackgerrit Merged openstack/nova stable/train: Make quotas respect instance_list_per_project_cells https://review.opendev.org/729087
05:44:32 openstackgerrit Huaqiang Wang proposed openstack/nova master: test_hardware: show test parameter details for causing a test failure https://review.opendev.org/692205
07:10:54 openstackgerrit Balazs Gibizer proposed openstack/nova stable/ussuri: Add admin doc information about image cache resource accounting https://review.opendev.org/729853
07:23:36 openstackgerrit Andreas Jaeger proposed openstack/nova-specs master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728611
07:23:52 openstackgerrit Andreas Jaeger proposed openstack/os-vif master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728612
08:34:05 openstackgerrit Balazs Gibizer proposed openstack/nova stable/queens: Reproduce bug 1862633 https://review.opendev.org/729539
08:34:05 openstack bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer)
08:35:18 openstackgerrit Balazs Gibizer proposed openstack/nova stable/queens: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/729540
08:35:44 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595
08:35:44 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596
08:36:03 lyarwood Morning all
08:36:54 gibi lyarwood: good morning
08:45:23 openstackgerrit junbo proposed openstack/nova master: use dpdk interface type for ovs dpdk representor port https://review.opendev.org/729874
08:47:31 lyarwood Random question to start the day, some folks downstream noticed that after an ungraceful shutdown and reboot of a compute that instances remained in ACTIVE with a vm_state of active and a power_state of NOSTATE. Can anyone recall why we don't just mark things as SHUTOFF etc?
08:52:06 gibi lyarwood: in my experience we mark them SHUTOFF after a compute host restart
08:52:50 lyarwood gibi: this might be a regression then as I can reproduce this in devstack
08:53:01 gibi interestign
08:57:37 openstackgerrit zhufl proposed openstack/nova master: [Trivial]Add missing white spaces between words in log messages https://review.opendev.org/729880
08:58:26 lyarwood gibi: http://paste.openstack.org/show/793830/ for example
08:59:18 gibi lyarwood: this part is strange "[stack@localhost devstack]$ sudo virsh undefine 7b0bd9f9-d74f-4c1c-bd27-a4bba44c391e"
08:59:26 gibi a simple compute host reboot does not undefine the domain
08:59:50 lyarwood ah true
08:59:56 lyarwood but it wouldn't change this
08:59:58 lyarwood I don't think
09:02:21 lyarwood ah it does
09:02:26 lyarwood it's SHUTOFF now
09:03:00 gibi then I guess not finding the domain in libvirt is not handled during the power synch
09:06:32 openstackgerrit Vladyslav Drok proposed openstack/nova master: Clean up instance on target node if evacuate fails https://review.opendev.org/375623
09:06:51 lyarwood https://github.com/openstack/nova/blob/bfdca19bb85a1a3d8238af267d3784b7cd3d93ff/nova/compute/manager.py#L9459-L9460 yeah I think it's this
09:08:03 lyarwood brb
09:09:23 gibi the code seems to set the NOSTATE intentionally
09:11:15 gibi and it is there at least for 6 years now
09:18:21 lyarwood yeah I can see the logic in this but if instance.host == the current host and the instance is ACTIVE we can still mark it as SHUTOFF tbh

Earlier   Later