| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-04 | |||
| 02:53:27 | mriedem | we didn't have this problem around ~newton when i wrote the original tempest test for multiattach | |
| 02:53:40 | mriedem | you could do the 2 attachments fine, it's just that nova didn't orchestrate the detach properly | |
| 02:53:45 | melwitt | oh :( | |
| 02:53:48 | mriedem | now we're in a case with newer qemu where the 2nd attach fails | |
| 02:54:07 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: log test: use fixtures.StandardLogging in setUp https://review.openstack.org/531065 | |
| 02:54:29 | mriedem | so apparently fixed in libvirt-3.9.0-3.el7 | |
| 02:54:53 | mriedem | we are testing against 3.6.0 | |
| 02:55:34 | mriedem | https://bugzilla.redhat.com/show_bug.cgi?id=1378242#c13 | |
| 02:55:35 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:55:38 | melwitt | and this is the qemu bug that the libvirt bug was cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1378241 | |
| 02:55:38 | openstack | bugzilla.redhat.com bug 1378241 in qemu-kvm-rhev "QEMU image file locking" [Unspecified,Verified] - Assigned to famz | |
| 02:55:38 | mriedem | same thing we're hitting | |
| 02:56:21 | melwitt | that says qemu-kvm-rhev though. | |
| 02:57:11 | mriedem | yeah so that caused the bug we had with 2.10 where we were hitting the same issue with qemu-img info | |
| 02:57:17 | mriedem | and had to start using the force flag for that | |
| 02:57:24 | mriedem | https://bugs.launchpad.net/nova/+bug/1718295 | |
| 02:57:26 | openstack | Launchpad bug 1718295 in OpenStack Compute (nova) "Live migration fails with qemu-img >= 2.10: "Failed to get shared "write" lock\nIs another process using the image?"" [High,Fix released] - Assigned to Sean Dague (sdague) | |
| 02:57:48 | mriedem | so now our support matrix is likely something like, | |
| 02:57:55 | mriedem | 1. do it the old way if qemu<2.10, else | |
| 02:58:05 | mriedem | 2. do it the new way if qemu>=2.10 AND libvirt>=3.9.0 | |
| 02:58:09 | mriedem | else | |
| 02:58:12 | mriedem | 3. shart your pants | |
| 02:58:31 | mriedem | and only 3 is testable in our current CI env :( | |
| 02:58:31 | melwitt | sigh, so this has some history | |
| 03:01:14 | clarkb | didnt nova implement its own lock wrapper for that? | |
| 03:01:29 | mriedem | for what? | |
| 03:01:33 | openstackgerrit | Eli Qiao proposed openstack/nova master: Api-guide: Add Block Device Mapping https://review.openstack.org/522084 | |
| 03:03:28 | clarkb | mriedem: for the lack of working locks | |
| 03:03:40 | clarkb | there wa sa while wrapper script thing iirc | |
| 03:04:42 | bjhuangr | mriedem: hi Matt, do you have a chance to review https://review.openstack.org/#/c/523387/ ? Thanks in advance | |
| 03:05:45 | mriedem | bjhuangr: not right now sorry | |
| 03:05:51 | mriedem | clarkb: sounds like something different | |
| 03:06:04 | mriedem | melwitt: looking at https://github.com/libvirt/libvirt/commit/28907b0043fbf71085a798372ab9c816ba043b93 it actually looks like that went into libvirt 3.10 | |
| 03:06:26 | mriedem | and it's unclear to me if we'd actually have to put something in the disk config xml or if libvirt just handles that for us | |
| 03:09:13 | melwitt | oh, hm. I wasn't sure about that either | |
| 03:10:20 | melwitt | looking at this, it looks like libvirt handles it. but not sure | |
| 03:10:32 | mriedem | yeah that's what i'm wondering | |
| 03:10:41 | mriedem | based on https://bugzilla.redhat.com/show_bug.cgi?id=1378242#c14 | |
| 03:10:42 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 03:11:14 | mriedem | i think peter is saying, you need qemu 2.10 with the new lock stuff and the fixes mentioned in comment 10 to libvirt | |
| 03:11:32 | mriedem | so if qemu>=2.10, libvirt must be >=3.10 (i think) | |
| 03:11:52 | mriedem | says it was fixed in libvirt-3.9.0-3.el7 but maybe that's an rpm package version with the patch backported? | |
| 03:12:02 | mriedem | because https://github.com/libvirt/libvirt/commit/28907b0043fbf71085a798372ab9c816ba043b93 says 3.10 | |
| 03:12:03 | melwitt | that's how I interpret it too | |
| 03:12:26 | mriedem | suck | |
| 03:12:50 | mriedem | i wonder if i could swindle jamespage and the gang to backport that to libvirt 3.6.0 :) | |
| 03:13:06 | melwitt | yeah, I'm honestly surprised this is all so recent | |
| 03:13:41 | melwitt | with all of the multiattach talk that's been happening for years, I never thought there could be a blocker at the qemu and libvirt level | |
| 03:14:38 | mriedem | sweet revenge | |
| 03:15:02 | melwitt | and the "fixed-in-version 3.9" with the commit in 3.10 doesn't make sense to me either | |
| 03:15:07 | mriedem | did you notice the guy complaining in comment 14 is using oracle RAC? | |
| 03:15:17 | mriedem | which is like the #1 use case for multiattach and why oracle wants it in nova so badly | |
| 03:15:41 | mriedem | comment 13 i should say | |
| 03:15:53 | mriedem | stvnoyes: fyi as this is all going to impact multiattach support in nova in queens ^ | |
| 03:16:08 | melwitt | what a mess | |
| 03:16:10 | mriedem | stvnoyes: if you're looking for something to do, we need some testing for the multiattach patches using qemu 2.10 and libvirt 3.1.0 | |
| 03:16:12 | mriedem | *3.10 | |
| 03:19:06 | mriedem | oh ha | |
| 03:19:07 | mriedem | lyarwood: https://bugzilla.redhat.com/show_bug.cgi?id=1415250 | |
| 03:19:09 | openstack | bugzilla.redhat.com bug 1415250 in openstack-nova "QEMU image file locking (RHOS)" [High,Post] - Assigned to lyarwood | |
| 03:19:10 | mriedem | melwitt: ^ | |
| 03:20:15 | melwitt | oh hello friends | |
| 03:21:33 | mriedem | left a comment because i like to comment in bz's | |
| 03:21:37 | mriedem | feels like i'm part of the team | |
| 03:21:50 | mriedem | ooo cc'ed to markmc@redhat.com | |
| 03:22:16 | mriedem | and eglynn@redhat.com, dansmith@redhat.com, | |
| 03:22:17 | mriedem | heh | |
| 03:23:45 | melwitt | :) | |
| 03:43:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments https://review.openstack.org/267587 | |
| 03:43:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [api] Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 03:44:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments https://review.openstack.org/267587 | |
| 03:44:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [api] Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 05:39:05 | takashin | ||
| 05:39:12 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Adds view builders for keypairs controller https://review.openstack.org/347289 | |
| 05:41:30 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Adds view builders for keypairs controller https://review.openstack.org/347289 | |
| 07:03:50 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Use neutron port_list when filtering instance by ip https://review.openstack.org/525505 | |
| 07:20:22 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795 | |
| 07:42:43 | takashin | git log -1 | |
| 08:14:01 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add cross cell sort support for get_migrations https://review.openstack.org/517273 | |
| 08:14:01 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406 | |
| 08:25:51 | hrw | morning | |
| 09:51:57 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: [WIP] Fix 500 in test_resize_server_negative_invalid_state https://review.openstack.org/531117 | |
| 09:55:18 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Use volume shared_targets to lock during attach/detach https://review.openstack.org/529695 | |
| 09:55:19 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: [libvirt] Allow multiple volume attachments https://review.openstack.org/267587 | |
| 09:55:19 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: [api] Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 09:59:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: zuul: Move legacy jobs to project https://review.openstack.org/514309 | |
| 10:00:39 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: [WIP] Fix 500 in test_resize_server_negative_invalid_state https://review.openstack.org/531117 | |
| 10:01:23 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add index(instance_uuid, updated_at) on instance_actions table https://review.openstack.org/530429 | |
| 10:06:54 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: [api] Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 10:21:21 | mdbooth | Is today finally going to be the day when https://review.openstack.org/#/c/529037/ doesn't fail spuriously in the gate? | |
| 10:21:42 | mdbooth | This is the fourth attempt, though, each attempt taking a full working day. | |
| 10:21:58 | mdbooth | It's not looking good for the plucky patch. | |
| 10:32:19 | kashyap | ildikov: Hi, do you have some CI Gate log of this -- https://bugzilla.redhat.com/show_bug.cgi?id=1415250#c13 | |
| 10:32:20 | openstack | bugzilla.redhat.com bug 1415250 in openstack-nova "QEMU image file locking (RHOS)" [High,Post] - Assigned to lyarwood | |
| 10:32:40 | kashyap | ildikov: So I could fish through the debug logs to find relevant bits | |
| 10:33:51 | kashyap | ildikov: Actually, disregard me, it's here: https://review.openstack.org/#/c/267587/ | |
| 10:34:09 | ildikov | kashyap: I got the logs from Matt with the following comment yeterday: "<mriedem> i can't link to these n-cpu logs because of how infra has changed where the logs live now, but that's the issue" | |
| 10:34:39 | kashyap | Ah, I didn't even realize openstack-infra has changed the log location | |
| 10:35:16 | ildikov | kashyap: no worries, Matt added a few extra bits to that patch yesterday | |
| 10:35:23 | kashyap | Yeah, just reading the review through | |