| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-25 | |||
| 14:43:57 | SpazZz | If any cores got some cycles today for this would be much appreciated. +1'd by two citrix folks, need mah +2 love now :) https://review.openstack.org/#/c/533168/ | |
| 14:44:01 | mdbooth | sean-k-mooney: If that's the same as 'tox -e pep8 -- -HEAD' it will also include uncommitted changes in the working directory. | |
| 14:44:19 | SpazZz | Ignore the IBM.. it'll go away eventually :P | |
| 14:44:20 | ameeda | sean-k-mooney: thank you, I appreciate all your help | |
| 14:44:57 | mdbooth | efried: Nice... | |
| 14:45:10 | sean-k-mooney | mdbooth: oh am yes it is. i sometiems have line lenght issue in my commit message so i prefer to commit first out of habbit | |
| 14:45:12 | mriedem | lyarwood: mdbooth: regarding https://review.openstack.org/#/c/523958/ - since it knowingly regresses swap volume with a luks-encrypted volume, is anyone working on a nova patch to workaround that and fallback to the old non-native routine? | |
| 14:45:30 | mdbooth | mriedem: I responded to that on the ML | |
| 14:45:35 | mriedem | lyarwood: mdbooth: because it's getting pretty late to just merge this and put something in the release notes about oopsies | |
| 14:45:51 | mdbooth | I don't really regard it as a regression, tbh, as it was entirely broken until this Monday | |
| 14:46:10 | mdbooth | And there's already an upstream bug tracking it in libvirt | |
| 14:46:17 | jaypipes | mriedem, efried: good morning fellas. I have a few hours before I need to go to the airport. so I'll try to focus on patch reviews during this time. do we have a priority list? | |
| 14:46:27 | jaypipes | dansmith: you too.. | |
| 14:46:54 | mdbooth | mriedem: We'd be adding in workaround code for something which needs an underlying fix. | |
| 14:47:13 | mriedem | mdbooth: then we need something more discoverable than a small note in a release note, | |
| 14:47:30 | mriedem | e.g. maybe a docs update to cinder saying that you can't migrate/retype an encrypted volume for now | |
| 14:47:41 | efried | jaypipes The update_provider_tree series is ready up to update_from_provider_tree. There's some possibly-controversial stuff towards the top. Not sure how much chance of getting approved before FF, but at least worth skimming. | |
| 14:48:01 | mdbooth | mriedem: Well you never could... | |
| 14:48:02 | jaypipes | mriedem, efried: I'm also continuing my patch series on the alloc candidates and n-r-p. been slow going because of $work the last two days | |
| 14:48:19 | mdbooth | mriedem: It's only a disappointment to anybody running master who updated on Monday | |
| 14:48:31 | efried | Heh. $work meaning work you get a paycheck for? :) | |
| 14:48:31 | jaypipes | efried: what's the latest status on alex_xu's patches around traits in alloc candidates? | |
| 14:48:45 | jaypipes | efried: yeah, that :) | |
| 14:48:51 | mriedem | mdbooth: ok - my point is, i don't know if no one else in the world knows you never could | |
| 14:48:55 | efried | jaypipes I believe we're +W and in recheck hell, but that could just be the bottom patch. | |
| 14:48:57 | efried | looking... | |
| 14:48:57 | mriedem | so, docs would be a good start | |
| 14:49:15 | mriedem | because users of openstack do'nt know, or shouldn't need to know, low level details about how the code works | |
| 14:49:18 | jaypipes | efried: the REST API change one? | |
| 14:49:19 | alex_xu | jaypipes: yes, all of those patches get approved...then recheck and recheck... | |
| 14:49:34 | jaypipes | alex_xu: gotcha, k, thx for the update | |
| 14:49:55 | stephenfin | Roamer`, mriedem: That's done now. There's some cleanup/simplification that can be done, but that's a separate effort | |
| 14:50:40 | mdbooth | mriedem: Critically we're not regressing. | |
| 14:50:58 | jaypipes | mriedem: depends. if you've an NFV user, of *course* you need to know all the low-level details of everything going on. ;) | |
| 14:51:02 | Roamer` | stephenfin, thanks a lot! And, yes, good catch on the exception... | |
| 14:52:02 | mriedem | jaypipes: yeah that's true | |
| 14:52:24 | mdbooth | mriedem: The way I see it is this implementation is much simpler and more robust for Nova, as I mentioned on the ML. | |
| 14:52:32 | jaypipes | efried: is the recheck hell due to timeouts in zuul jobs or is it due to a particular bug in a dependency (or nova etc) | |
| 14:52:32 | mriedem | jaypipes: well, except for the part where you just tell nova to put the instance on this host and this numa node and this set of cpus, totally bypassing the scheduler | |
| 14:52:45 | jaypipes | mriedem: right. NFV. | |
| 14:53:01 | efried | jaypipes The latest and greatest is the log server went read-only. We're supposed to not recheck anything until they've sent another broadcast saying it's fixed. | |
| 14:53:02 | mriedem | jaypipes: there was a big regression on tuesday that caused most dsvm jobs to fail. the fix for that merged yesterday. | |
| 14:53:06 | mdbooth | We can add complexity to work around a temporary limitation in the underlying library, but given that the feature in question never worked anyway I think it would be better just to note that it doesn't work. | |
| 14:53:09 | jaypipes | mriedem: that's pretty much what I've been talking about at this $work thing for 2 days. | |
| 14:53:12 | mriedem | however, still hitting job timeouts and post failures | |
| 14:53:26 | jaypipes | k | |
| 14:53:45 | jaypipes | efried: ack | |
| 14:53:59 | mdbooth | mriedem: Specifically, I don't want to add yet anothing config option for this: it should be transparent to the operator. | |
| 14:54:01 | efried | jaypipes I also noticed (not sure if there's actual cause->effect here) that rechecking jobs *later* in The Big Series caused the bottom patch to stall in the gate. | |
| 14:54:11 | jaypipes | hmm. | |
| 14:54:22 | efried | jaypipes So I think whenever stuff opens up again, I'm going to try just checking the bottom one. | |
| 14:54:23 | jaypipes | not sure I've personally seen that yet. | |
| 14:55:04 | efried | yeah, it was weird. The bottom one was in the gate and blue worms creeping along, looking good. Then I rechecked some above it. And then the bottom one stopped having blue worms (but stayed in the gate queue). | |
| 14:55:18 | mriedem | mdbooth: i'm not asking for a config option | |
| 14:55:52 | efried | ...where it has been for ~11.5h now | |
| 14:56:47 | openstackgerrit | Ameed Ashour proposed openstack/nova master: detaching volumes when deleted VM is in error state https://review.openstack.org/528069 | |
| 14:56:48 | openstackgerrit | Ameed Ashour proposed openstack/nova master: detach instance volumes when VM creation fails https://review.openstack.org/528385 | |
| 14:57:44 | mdbooth | mriedem: It's not a regression though, right? Specifically because it was always broken. There is nobody currently using this feature who will have their workflow broken by this change. | |
| 14:58:28 | mriedem | mdbooth: if we're not going to put some kind of conditional fallback code in place, then i don't think the release note should mention it really - it's confusing and alarming if you didn't realize this iddn't work until recently; and on top of that, we likely need an api-ref update to cinder to mention that retype/migration of an attached encrypted volume won't work on the compute side because of some bug | |
| 14:59:05 | mriedem | s/of some bug/it's not yet supported/ | |
| 14:59:45 | mdbooth | mriedem: Well we discovered that swap volume was broken in review, about a month ago. Lee opened a bug about it. | |
| 15:00:12 | ameeda | please review my code for this two gerrit https://review.openstack.org/528069 and https://review.openstack.org/528385 , thanks in advance :) | |
| 15:00:23 | mriedem | mdbooth: https://bugs.launchpad.net/nova/+bug/1739593 | |
| 15:00:25 | openstack | Launchpad bug 1739593 in OpenStack Security Advisory "Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack" [Undecided,Incomplete] | |
| 15:00:29 | mdbooth | https://bugs.launchpad.net/nova/+bug/1739593 | |
| 15:00:31 | mdbooth | Yeah, that's the one | |
| 15:00:37 | mriedem | ok so that says fix released | |
| 15:00:45 | mdbooth | Yep, last Monday | |
| 15:01:04 | mriedem | so if i'm a new user to openstack and trying to swap an encrypted volume with libvirt, and it doesn't work, i might look for a bug and find this and it says it's fixed, | |
| 15:01:12 | dansmith | bauzas: are you around today? | |
| 15:01:14 | mriedem | but now my operator is running new enough libvirt/qemu for the native stuff, and it doesn't work | |
| 15:01:29 | mriedem | which is a new bug/regression technically | |
| 15:01:34 | mriedem | mdbooth: so what do you tell that user? | |
| 15:01:47 | mriedem | "it never worked, until it did, and then it didn't again" | |
| 15:02:24 | mdbooth | I think we're splitting hairs here, as nobody at all will have used it in the intervening period, and it will be fixed upstream at some point anyway. | |
| 15:02:49 | mdbooth | And as I said, this does simplify Nova. | |
| 15:02:51 | mriedem | mdbooth: "fixed upstream at some point" in libvirt/qemu could be ages before someone actually picks up the versions of those packages that have those fixes, that's not using RHOSP | |
| 15:03:28 | mriedem | mdbooth: if there is a bz for libvirt/qemu, then that's the known issue, | |
| 15:03:39 | mriedem | and i think we should have something in the docs about this is a known limitation | |
| 15:03:43 | mdbooth | mriedem: It's linked in a code comment | |
| 15:03:48 | mriedem | jesus christ | |
| 15:03:53 | mriedem | end users shouldn't have to read code | |
| 15:04:10 | mriedem | nevermind, i'll push the patch | |
| 15:04:37 | mdbooth | mriedem: Thanks. | |
| 15:04:40 | mriedem | what's the bz? | |
| 15:04:57 | mdbooth | Sec... | |
| 15:05:53 | mdbooth | mriedem: Incidentally, swap volume is also broken for multi attach. | |
| 15:06:10 | mdbooth | It's generally a difficult feature. | |
| 15:07:07 | openstack | bugzilla.redhat.com bug 760547 in libvirt "[RFE] specifying the entire image chain as a qemu drive (blockdev-add) (libvirt)" [Medium,Assigned] - Assigned to pkrempa | |
| 15:07:07 | mdbooth | https://bugzilla.redhat.com/show_bug.cgi?id=760547 | |
| 15:07:12 | mriedem | how is swap volume broken for multiattach? | |
| 15:07:39 | mdbooth | mriedem: Do we not disable swap volume on multiattached volumes? | |
| 15:07:45 | kashyap | The above bugzilla in turn depends on libvirt wiring up the new QMP command 'blockdev-add' | |
| 15:07:53 | kashyap | (New from QEMU 2.9 onwards.) | |
| 15:08:18 | mdbooth | mriedem: So, swap volume does a block rebase in Nova, which is local to a single instance. | |
| 15:08:38 | mdbooth | It makes a bunch of assumptions about things not changing under it, which are broken by multi attach. | |
| 15:08:57 | mdbooth | So if you do a volume migration on a multi-attach volume, that would cause data corruption. | |
| 15:10:05 | mdbooth | Incidentally, I had some ideas about how you might make it work but they're crazy complex. Simplest just to disable it, which is what I assumed we'd done. | |
| 15:10:40 | kashyap | mriedem: Swapping an ecrypted volume won't be solved soon -- as it requires libvirt doing some surgery as noted above. | |
| 15:11:00 | kashyap | Nova _swap_volume() method uses blockRebase() API. And libvirt's current notion of encryption key is persistent only to the active layer. | |