| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-25 | |||
| 14:49:55 | stephenfin | Roamer`, mriedem: That's done now. There's some cleanup/simplification that can be done, but that's a separate effort | |
| 14:50:40 | mdbooth | mriedem: Critically we're not regressing. | |
| 14:50:58 | jaypipes | mriedem: depends. if you've an NFV user, of *course* you need to know all the low-level details of everything going on. ;) | |
| 14:51:02 | Roamer` | stephenfin, thanks a lot! And, yes, good catch on the exception... | |
| 14:52:02 | mriedem | jaypipes: yeah that's true | |
| 14:52:24 | mdbooth | mriedem: The way I see it is this implementation is much simpler and more robust for Nova, as I mentioned on the ML. | |
| 14:52:32 | jaypipes | efried: is the recheck hell due to timeouts in zuul jobs or is it due to a particular bug in a dependency (or nova etc) | |
| 14:52:32 | mriedem | jaypipes: well, except for the part where you just tell nova to put the instance on this host and this numa node and this set of cpus, totally bypassing the scheduler | |
| 14:52:45 | jaypipes | mriedem: right. NFV. | |
| 14:53:01 | efried | jaypipes The latest and greatest is the log server went read-only. We're supposed to not recheck anything until they've sent another broadcast saying it's fixed. | |
| 14:53:02 | mriedem | jaypipes: there was a big regression on tuesday that caused most dsvm jobs to fail. the fix for that merged yesterday. | |
| 14:53:06 | mdbooth | We can add complexity to work around a temporary limitation in the underlying library, but given that the feature in question never worked anyway I think it would be better just to note that it doesn't work. | |
| 14:53:09 | jaypipes | mriedem: that's pretty much what I've been talking about at this $work thing for 2 days. | |
| 14:53:12 | mriedem | however, still hitting job timeouts and post failures | |
| 14:53:26 | jaypipes | k | |
| 14:53:45 | jaypipes | efried: ack | |
| 14:53:59 | mdbooth | mriedem: Specifically, I don't want to add yet anothing config option for this: it should be transparent to the operator. | |
| 14:54:01 | efried | jaypipes I also noticed (not sure if there's actual cause->effect here) that rechecking jobs *later* in The Big Series caused the bottom patch to stall in the gate. | |
| 14:54:11 | jaypipes | hmm. | |
| 14:54:22 | efried | jaypipes So I think whenever stuff opens up again, I'm going to try just checking the bottom one. | |
| 14:54:23 | jaypipes | not sure I've personally seen that yet. | |
| 14:55:04 | efried | yeah, it was weird. The bottom one was in the gate and blue worms creeping along, looking good. Then I rechecked some above it. And then the bottom one stopped having blue worms (but stayed in the gate queue). | |
| 14:55:18 | mriedem | mdbooth: i'm not asking for a config option | |
| 14:55:52 | efried | ...where it has been for ~11.5h now | |
| 14:56:47 | openstackgerrit | Ameed Ashour proposed openstack/nova master: detaching volumes when deleted VM is in error state https://review.openstack.org/528069 | |
| 14:56:48 | openstackgerrit | Ameed Ashour proposed openstack/nova master: detach instance volumes when VM creation fails https://review.openstack.org/528385 | |
| 14:57:44 | mdbooth | mriedem: It's not a regression though, right? Specifically because it was always broken. There is nobody currently using this feature who will have their workflow broken by this change. | |
| 14:58:28 | mriedem | mdbooth: if we're not going to put some kind of conditional fallback code in place, then i don't think the release note should mention it really - it's confusing and alarming if you didn't realize this iddn't work until recently; and on top of that, we likely need an api-ref update to cinder to mention that retype/migration of an attached encrypted volume won't work on the compute side because of some bug | |
| 14:59:05 | mriedem | s/of some bug/it's not yet supported/ | |
| 14:59:45 | mdbooth | mriedem: Well we discovered that swap volume was broken in review, about a month ago. Lee opened a bug about it. | |
| 15:00:12 | ameeda | please review my code for this two gerrit https://review.openstack.org/528069 and https://review.openstack.org/528385 , thanks in advance :) | |
| 15:00:23 | mriedem | mdbooth: https://bugs.launchpad.net/nova/+bug/1739593 | |
| 15:00:25 | openstack | Launchpad bug 1739593 in OpenStack Security Advisory "Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack" [Undecided,Incomplete] | |
| 15:00:29 | mdbooth | https://bugs.launchpad.net/nova/+bug/1739593 | |
| 15:00:31 | mdbooth | Yeah, that's the one | |
| 15:00:37 | mriedem | ok so that says fix released | |
| 15:00:45 | mdbooth | Yep, last Monday | |
| 15:01:04 | mriedem | so if i'm a new user to openstack and trying to swap an encrypted volume with libvirt, and it doesn't work, i might look for a bug and find this and it says it's fixed, | |
| 15:01:12 | dansmith | bauzas: are you around today? | |
| 15:01:14 | mriedem | but now my operator is running new enough libvirt/qemu for the native stuff, and it doesn't work | |
| 15:01:29 | mriedem | which is a new bug/regression technically | |
| 15:01:34 | mriedem | mdbooth: so what do you tell that user? | |
| 15:01:47 | mriedem | "it never worked, until it did, and then it didn't again" | |
| 15:02:24 | mdbooth | I think we're splitting hairs here, as nobody at all will have used it in the intervening period, and it will be fixed upstream at some point anyway. | |
| 15:02:49 | mdbooth | And as I said, this does simplify Nova. | |
| 15:02:51 | mriedem | mdbooth: "fixed upstream at some point" in libvirt/qemu could be ages before someone actually picks up the versions of those packages that have those fixes, that's not using RHOSP | |
| 15:03:28 | mriedem | mdbooth: if there is a bz for libvirt/qemu, then that's the known issue, | |
| 15:03:39 | mriedem | and i think we should have something in the docs about this is a known limitation | |
| 15:03:43 | mdbooth | mriedem: It's linked in a code comment | |
| 15:03:48 | mriedem | jesus christ | |
| 15:03:53 | mriedem | end users shouldn't have to read code | |
| 15:04:10 | mriedem | nevermind, i'll push the patch | |
| 15:04:37 | mdbooth | mriedem: Thanks. | |
| 15:04:40 | mriedem | what's the bz? | |
| 15:04:57 | mdbooth | Sec... | |
| 15:05:53 | mdbooth | mriedem: Incidentally, swap volume is also broken for multi attach. | |
| 15:06:10 | mdbooth | It's generally a difficult feature. | |
| 15:07:07 | openstack | bugzilla.redhat.com bug 760547 in libvirt "[RFE] specifying the entire image chain as a qemu drive (blockdev-add) (libvirt)" [Medium,Assigned] - Assigned to pkrempa | |
| 15:07:07 | mdbooth | https://bugzilla.redhat.com/show_bug.cgi?id=760547 | |
| 15:07:12 | mriedem | how is swap volume broken for multiattach? | |
| 15:07:39 | mdbooth | mriedem: Do we not disable swap volume on multiattached volumes? | |
| 15:07:45 | kashyap | The above bugzilla in turn depends on libvirt wiring up the new QMP command 'blockdev-add' | |
| 15:07:53 | kashyap | (New from QEMU 2.9 onwards.) | |
| 15:08:18 | mdbooth | mriedem: So, swap volume does a block rebase in Nova, which is local to a single instance. | |
| 15:08:38 | mdbooth | It makes a bunch of assumptions about things not changing under it, which are broken by multi attach. | |
| 15:08:57 | mdbooth | So if you do a volume migration on a multi-attach volume, that would cause data corruption. | |
| 15:10:05 | mdbooth | Incidentally, I had some ideas about how you might make it work but they're crazy complex. Simplest just to disable it, which is what I assumed we'd done. | |
| 15:10:40 | kashyap | mriedem: Swapping an ecrypted volume won't be solved soon -- as it requires libvirt doing some surgery as noted above. | |
| 15:11:00 | kashyap | Nova _swap_volume() method uses blockRebase() API. And libvirt's current notion of encryption key is persistent only to the active layer. | |
| 15:11:16 | kashyap | Where 'active layer' means, given A.raw <-- B.qcow2 (live) | |
| 15:11:28 | mriedem | kashyap: that's why i said, "meh, it'll eventually be fixed upstream so who cares" isn't a good response to a user | |
| 15:11:29 | kashyap | B is the "active layer", where A is the backing file of B. | |
| 15:11:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Bumping functional test job timeouts https://review.openstack.org/537933 | |
| 15:12:02 | gibi | mriedem: ^^ | |
| 15:12:17 | mriedem | mdbooth: we don't currently block swap volume with a multiattach volume | |
| 15:12:21 | mriedem | on the nova side | |
| 15:13:12 | mriedem | i believe on the cinder side, you can't retype a multiattach volume that is attached to an instance | |
| 15:13:12 | mdbooth | mriedem: Cool, so we just prevent it? I didn't look specifically last time I reviewed that patch. | |
| 15:13:14 | mriedem | jgriffith: right ^ ? | |
| 15:13:27 | mriedem | mdbooth: i said we don't bloc kit | |
| 15:13:29 | mriedem | meaning we allow it | |
| 15:14:06 | mdbooth | mriedem: Ah... | |
| 15:14:08 | mriedem | mdbooth: if the volume's secondary attachments are only read-only, it shouldn't be an issue right? | |
| 15:14:36 | mdbooth | mriedem: Well that's going to get a bit weird, but it won't cause data corruption. | |
| 15:15:13 | mdbooth | Although presumably they're also going to have to swap volume, because the old one goes away, right? | |
| 15:15:58 | mriedem | not parsing that | |
| 15:16:05 | mriedem | see L50 here for what i'm talking about https://etherpad.openstack.org/p/nova-ptg-rocky | |
| 15:18:07 | mdbooth | The way swap volume works is we create a local COW for the volume, which means we're no longer making changes to it. We (Nova) then copies all the data from the source to the dest. During this copy, all writes are written to the COW, which is local to a compute. When the copy is complete, we commit all the changes in the COW to the new copy, then flip to the new copy. | |
| 15:18:14 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408 | |
| 15:18:16 | mriedem | https://specs.openstack.org/openstack/cinder-specs/specs/queens/enable-multiattach.html | |
| 15:18:21 | mriedem | "Again NOTE we will NOT allow retype of multiattachment setting for an in-use volume." | |
| 15:18:48 | mdbooth | mriedem: Right. If I'm understanding you correctly we missed that? | |
| 15:19:01 | mriedem | it's supposed to be handled on the cinder side | |
| 15:19:11 | mriedem | i don't know if that includes volume migration | |
| 15:19:16 | mriedem | where it's not a retype | |
| 15:19:21 | mriedem | jgriffith: smcginnis: ildikov: ^ | |
| 15:19:23 | jgriffith | mriedem: it's set in the policy | |
| 15:19:45 | mriedem | jgriffith: i thought we said you simply couldn't retype a multiattach in-use volume, regardless of policy | |
| 15:19:48 | jgriffith | mriedem: You can if policy permits, default policy being admin/owner | |
| 15:20:06 | jgriffith | mriedem: Oh | |