Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-09-25
13:25:35 CeeMac lyarwood: sounds like you're on to something
13:25:58 lyarwood CeeMac: kk, I think the issue is that libvirt leaves the new volume connected after this failure, the instance keeps writing to the new volume until we rip out the volume from underneath it likely sending the disk into a read-only state within the instance. By the time the instance reboots it's out of sync with the original volume that is now reconnected and that's where the failure comes
13:25:59 lyarwood from.
13:26:46 CeeMac lyarwood: during the swap_volume process you mean?
13:26:50 lyarwood let me see if I can reproduce this on master with a Fedora VM so I can debug this a little more and prove that's the case
13:27:04 CeeMac lyarwood: so the new device is attached for a period until the job fails and is reverted?
13:27:06 lyarwood CeeMac: so we fail *after* the swap volume operation has completed
13:27:23 lyarwood CeeMac: yeah I believe so
13:27:46 lyarwood CeeMac: there's nothing in the failure path within Nova that would revert the active domain configuration AFAICT
13:33:35 openstackgerrit Merged openstack/nova master: libvirt: make mdev types name attribute be optional https://review.opendev.org/753574
13:34:34 CeeMac lyarwood: that could explain it. Appreciate you looking into this so diligently :)
13:47:39 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224
13:47:55 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757
13:48:46 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777
13:49:12 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094
13:50:26 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109
13:54:41 openstackgerrit Takashi Natsume proposed openstack/nova master: Add placeholder migrations for Victoria backports https://review.opendev.org/754396
14:05:18 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117
14:16:52 openstackgerrit Stephen Finucane proposed openstack/nova stable/victoria: libvirt: make mdev types name attribute be optional https://review.opendev.org/754401
14:17:00 stephenfin bauzas: ^
14:18:37 lyarwood stephenfin: do we want to land that in stable before we are certain we don't want another rc?
14:19:06 stephenfin um, I don't understand the question
14:19:13 stephenfin are you saying we should wait to land this in 22.0.1?
14:19:13 gibi if we merge someting meaningful to stable/victoria now then we have to cut RC2 I guess
14:19:17 stephenfin ah
14:19:19 stephenfin gotcha
14:19:35 stephenfin then no, that's not critical IMO
14:19:47 stephenfin I just know bauzas was eager for it to land at some point
14:20:00 gibi yeah I would only land critical things to stable/v at the moment
14:20:06 stephenfin makes sense
14:21:47 gibi also landing things on master should be limited to make a possible critical bug backport easy
14:50:59 melwitt lyarwood: hey, did you miss adding a NOTE here for the backport conflicts? https://review.opendev.org/747978
14:52:08 lyarwood melwitt: looking
14:52:53 lyarwood melwitt: oh is this the pike one
14:53:18 melwitt yeah, I wasn't sure if a NOTE was left out intentionally or not
14:53:34 lyarwood melwitt: I didn't write them up for this sorry, I posted it as a courtesy for someone
14:53:58 lyarwood that and the Newton patch attached to the bug
14:54:06 melwitt ah gotcha
15:20:58 openstackgerrit Takashi Natsume proposed openstack/nova master: Update contributor guide for Wallaby https://review.opendev.org/754427
15:26:28 openstackgerrit melanie witt proposed openstack/nova stable/pike: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747978
15:34:19 openstackgerrit Merged openstack/os-vif stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/750924
15:41:14 openstackgerrit Merged openstack/nova master: Use absolute path during qemu img rebase https://review.opendev.org/739246
15:41:21 openstackgerrit Merged openstack/nova stable/ussuri: Move revert resize under semaphore https://review.opendev.org/751354
15:41:32 openstackgerrit Merged openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982
15:49:23 openstackgerrit George Melikov proposed openstack/nova master: Instance snapshot: fallback to usual upload on ImageNotAuthorized https://review.opendev.org/754442
16:00:41 openstackgerrit melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055
16:00:42 openstackgerrit melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444
16:02:27 openstackgerrit melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444
16:02:28 openstackgerrit melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055
16:13:08 stephenfin melwitt: Care to review this functional reproducer for me, please? https://review.opendev.org/#/c/749713/
16:13:41 melwitt stephenfin: will do
16:13:45 stephenfin thanks :)
16:14:08 stephenfin lyarwood: If you're still about, this is a follow-up to a series you looked at https://review.opendev.org/#/c/749319/
16:15:53 lyarwood stephenfin: yup still here, if I don't get to today I'll make sure I do on Monday
16:16:02 stephenfin Good enough for me. Thanks
16:23:01 openstackgerrit Stephen Finucane proposed openstack/nova master: Rework how we check for neutron extensions https://review.opendev.org/705792
16:23:01 openstackgerrit Stephen Finucane proposed openstack/nova master: neutron: Remove references to out-of-tree 'qos-queue' extension https://review.opendev.org/754448
16:24:28 openstackgerrit Merged openstack/python-novaclient stable/ussuri: Add a cleanup for a server in a functional test https://review.opendev.org/743756
16:28:34 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Don't use generic 'Field' container https://review.opendev.org/738239
16:28:35 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove unnecessary type aliases, exceptions https://review.opendev.org/738240
16:28:35 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Use imports instead of type aliases https://review.opendev.org/738018
16:28:36 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove wrappers around ovo mixins https://review.opendev.org/738019
16:28:36 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: objects: Add type hints https://review.opendev.org/738020
16:31:44 lyarwood CeeMac: took a while but I've confirmed what I said before is what is happening btw, writing it up now.
16:32:27 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove use of pkg_resources https://review.opendev.org/740661
16:35:00 CeeMac lyarwood: good work, thanks
16:36:36 openstackgerrit Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci' https://review.opendev.org/676209
16:36:37 openstackgerrit Stephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console https://review.opendev.org/705657
16:36:37 openstackgerrit Stephen Finucane proposed openstack/nova master: rpc: Rework 'get_notifier', 'wrap_exception' https://review.opendev.org/741663
16:36:38 openstackgerrit Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to top-level modules https://review.opendev.org/705658
16:36:38 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Expand type hints for nova.block_device https://review.opendev.org/742170
16:38:51 stephenfin melwitt: Oh, also, if you have a chance to look at https://review.opendev.org/#/c/743976/ again and see if you agree with my thoughts, that'd be lovely
16:50:01 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Add type hints for 'nova.compute.api' https://review.opendev.org/741623
16:50:02 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Duplicate '_test_resize' to '_test_migrate' https://review.opendev.org/743206
16:50:02 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Remove unnecessary logic from resize, migrate tests https://review.opendev.org/743207
16:50:03 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Add 'nova.compute.API.migrate' https://review.opendev.org/741624
16:50:03 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Split 'nova.compute.API._resize' https://review.opendev.org/743208
16:53:07 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Add type hints https://review.opendev.org/744869
16:53:07 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Remove 'kwargs' from 'get_instance_nw_info' https://review.opendev.org/744872
16:53:08 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Remove unused 'affect_auto_assigned' parameter https://review.opendev.org/744871
16:53:08 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Add type hints for various allocate functions https://review.opendev.org/744870
17:02:54 CeeMac lyarwood: thanks again for spending your time on this, it is much appreciated
18:06:54 melwitt stephenfin: question for you re: whether we could add the correct asserts commented out or not https://review.opendev.org/#/c/749713/2/nova/tests/functional/libvirt/test_numa_live_migration.py@421
18:15:51 openstackgerrit Merged openstack/nova stable/victoria: [stable-only] Update .gitreview for stable/victoria https://review.opendev.org/754319
18:16:00 openstackgerrit Merged openstack/nova stable/victoria: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320
19:40:17 openstackgerrit Merged openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883
23:35:43 openstackgerrit Merged openstack/nova stable/pike: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747978
#openstack-nova - 2020-09-26
08:22:32 openstackgerrit Takashi Natsume proposed openstack/nova stable/ussuri: doc: correct the link to user/flavor.rst https://review.opendev.org/754495
#openstack-nova - 2020-09-27
14:31:57 openstackgerrit George Melikov proposed openstack/nova master: Instance snapshot: fallback to usual upload on ImageNotAuthorized https://review.opendev.org/754442
18:23:24 openstackgerrit George Melikov proposed openstack/nova master: Instance snapshot: fallback to usual upload on ImageNotAuthorized https://review.opendev.org/754442
#openstack-nova - 2020-09-28
04:11:21 openstackgerrit Luyao Zhong proposed openstack/nova master: Rollback when reset running live migration https://review.opendev.org/747899
04:11:22 openstackgerrit Luyao Zhong proposed openstack/nova master: Improve orphans tracking in resource tracker https://review.opendev.org/745282
07:11:52 gibi good motning
08:06:49 aarents good morning
08:09:07 aarents lyarwood when you will have time for that, I added devstack parameter: https://review.opendev.org/#/c/752463/ ans run https://review.opendev.org/#/c/752463/ on top of it, it seems ok
09:27:09 openstackgerrit Stephen Finucane proposed openstack/nova master: Add reproducer for bug #1894095 https://review.opendev.org/749713
09:27:09 openstack bug 1894095 in OpenStack Compute (nova) "Running periodic task during live migration results in incorrect usage" [Medium,Confirmed] https://launchpad.net/bugs/1894095 - Assigned to Stephen Finucane (stephenfinucane)
10:05:46 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Add type hints for 'nova.compute.api' https://review.opendev.org/741623
10:05:47 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Duplicate '_test_resize' to '_test_migrate' https://review.opendev.org/743206
10:05:47 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Remove unnecessary logic from resize, migrate tests https://review.opendev.org/743207

Earlier   Later