| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-06 | |||
| 14:07:15 | mriedem | artom: who is doing that? | |
| 14:07:29 | artom | mriedem, see mdbooth's reply :) | |
| 14:07:31 | mriedem | swap volume in nova is an admin-only api | |
| 14:07:42 | artom | We also got a downstream but about it | |
| 14:07:42 | mdbooth | mriedem: We really do have customers who call swap volume directly. | |
| 14:07:48 | mdbooth | More than one. | |
| 14:08:03 | mriedem | why? | |
| 14:08:29 | kashyap | Presumably to move storage from one type (local) to another (NFS)? | |
| 14:08:33 | mdbooth | I still haven't nailed it down, but I *think* it's a workaround related to BFV. | |
| 14:08:41 | artom | mriedem, I mean, I would agree, it's a weird thing to do, but we can't honestly put an API out there and get defensive when people use it in unexpected ways | |
| 14:08:42 | mriedem | kashyap: that would be a retype | |
| 14:08:47 | mriedem | which you should initiate through cinder | |
| 14:08:54 | kashyap | Ah, damned if I ever remember the names | |
| 14:09:13 | mdbooth | mriedem: But regardless, even if we think it's dumb, it's a public api and they're using it. | |
| 14:10:04 | mriedem | awesome | |
| 14:10:16 | mdbooth | Indeed | |
| 14:10:18 | mriedem | i made it 70 minutes today before hating my life | |
| 14:10:25 | mdbooth | \o/ | |
| 14:10:26 | kashyap | This morning we chatted here about mdbooth's e-mail (which I haven't pulled down yet). At least in context of what libvirt APi does by default | |
| 14:10:38 | jaypipes | mdbooth: my annoyance is that this is a libvirt/QEMU-specific behaviour/implementation that is leaking out of the Compute API. | |
| 14:10:59 | kashyap | jaypipes: By _default_ libvirt doesn't copy shared disks | |
| 14:11:02 | mdbooth | jaypipes: That was the thrust of my reply. It's not libvirt/qemu specific at all. | |
| 14:11:09 | jaypipes | mdbooth: BTW, where exactly does "swap volume" imply or denote that there is a copy operation going on? :( | |
| 14:11:09 | kashyap | But if you _ask_ for it, it will migrate it | |
| 14:11:16 | mriedem | mdbooth: the only virt driver that implements this is libvirt | |
| 14:11:21 | mdbooth | jaypipes: It could be implemented by any hypervisor, and they'd have to do the copy. | |
| 14:11:22 | jaypipes | mdbooth: yes, it is. | |
| 14:11:32 | mriedem | same with guest-assisted volume snapshot | |
| 14:11:42 | mdbooth | jaypipes: I replied in depth on the ML, anyway. | |
| 14:11:50 | jaypipes | yes, I read your reply. | |
| 14:11:57 | mriedem | i have no idea how non-libvirt drivers handle volume retype/live migration either | |
| 14:12:07 | jaypipes | mriedem: they don't. wisely. | |
| 14:12:10 | mdbooth | But Cinder can't do this, regardless of which hypervisor is attached. | |
| 14:12:27 | jaypipes | mdbooth: it's the "live" part that Cinder can't do, yes? | |
| 14:12:28 | mriedem | i must be missing something but i don't see any reply to jaypipes from mdbooth in the ML | |
| 14:12:38 | mdbooth | jaypipes: No, that's just an optimisation. | |
| 14:12:49 | mdbooth | jaypipes: If anything is attached, Cinder needs to ask Nova for help. | |
| 14:12:58 | mdbooth | Nova could just pause, copy, unpause | |
| 14:12:59 | kashyap | jaypipes: To your "swap volume" denoting copy operation, you wouldn't know it, unless you look at the Nova _swap_volume() method | |
| 14:13:01 | jaypipes | mdbooth: then I don't understand what is different between the pause VM, volume migrate, start VM. | |
| 14:13:10 | mriedem | i wonder when k8s is going to implement swap volume for cinder to use | |
| 14:13:14 | mriedem | that will be awesome | |
| 14:13:23 | kashyap | jaypipes: And then, map it to libvirt's blockRebase() API, which is called by _swap_volume(). | |
| 14:13:25 | mdbooth | jaypipes: How would you prevent unpause during the volume migrate? | |
| 14:13:29 | jaypipes | kashyap: no... unless you look at the LIBVIRT driver _swap_volume() method. | |
| 14:13:33 | jaypipes | kashyap: which is my point. | |
| 14:13:40 | kashyap | jaypipes: That's what I said :-) | |
| 14:13:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 14:14:00 | jaypipes | kashyap: no, you said "the Nova _swap_volume() method". | |
| 14:14:21 | kashyap | jaypipes: Yes, you're right - Nova's libvirt driver. | |
| 14:14:43 | jaypipes | mdbooth: it's an administrative action that involves multiple orchestrated pieces. | |
| 14:15:40 | mdbooth | jaypipes: Note that we're redefining swap_volume here, which is a bit out of scope. My concern relates to multiattach. | |
| 14:15:45 | jaypipes | mdbooth: and just because libvirt/QEMU has some magic foo to do all that orchestration in a (sometimes) live fashion, we've made the Compute API look like the libvirt/QEMU API and embedded expectations of full coordination/orchestration into our API. and that's wrong IMHO. | |
| 14:16:17 | mdbooth | However, there are 2 parts to it. | |
| 14:16:41 | MultipleCrashes | Please have a look at this review and leave a feedback : https://review.openstack.org/#/c/570370/ | |
| 14:16:45 | jaypipes | mdbooth: yes, I understand your concern about multiattach and artom's suggestion of just barfing back a 400 Bad Request when any of the source or dest volumes in the swap operation are multiattach is a perfectly reasonable "solution" to that. | |
| 14:16:49 | mdbooth | If we removed swap volume and implemented this with pause/resume, we would still need an additional api in nova to 'lock' the pause. | |
| 14:17:08 | jaypipes | mdbooth: I'm just annoyed at everything, sorry. | |
| 14:17:09 | mdbooth | Otherwise, orchestration or not, a concurrent operation could unpause it. | |
| 14:17:26 | artom | jaypipes, would you like a lawn and some kids to yell at? ;) | |
| 14:17:35 | jaypipes | mdbooth: precisely. which is why it belongs in an orchestration layer, not the virt driver layer :) | |
| 14:17:38 | mdbooth | Also, customers who don't like 6 hours of downtime would be grumpy. | |
| 14:17:48 | jaypipes | artom: :) | |
| 14:18:08 | mdbooth | jaypipes: But you *can't* orchestrate an *inability* for somebody else to do something without a new api. | |
| 14:18:28 | mdbooth | Your orchestrator can pause, migrate, unpause | |
| 14:18:29 | mriedem | mdbooth: so your use case in the ML reply to jay was that people are doing retype to get from old shitty storage to new shiny storage, | |
| 14:18:36 | mriedem | but that's still initiated through cinder's retype API | |
| 14:18:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 14:18:44 | mriedem | what's the use case for using the swap volume API directly w/o cinder? | |
| 14:18:47 | mdbooth | But it can't prevent my user, frustrated that his data has been unavailable for 6 hours, from unpausing | |
| 14:19:19 | mdbooth | mriedem: No idea. I don't think it matters here, though. | |
| 14:20:16 | mriedem | i think it matters to understand how people are using or abusing the APIs | |
| 14:20:31 | mriedem | os-server-external-events is a public REST API which i could call directly with admin creds, i don't know why i would, but i can | |
| 14:20:33 | mriedem | doesn't mean i should | |
| 14:20:38 | jaypipes | mriedem: ++ | |
| 14:20:41 | mdbooth | mriedem: Agreed. I don't think it's specific to this case, though. | |
| 14:21:01 | mdbooth | mriedem: It doesn't impact the multiattach thing. | |
| 14:21:03 | artom | mriedem, I think the difference here is that, calling os-server-external-events doesn't potentially corrupt data | |
| 14:21:25 | mriedem | you guys are missing the point | |
| 14:21:27 | jaypipes | mdbooth: BTW, I'm not insinuating that you personally have anything to do with this swap_volume() code. Just pointing out I don't think this particular API call belongs in the compute API. | |
| 14:21:28 | artom | For instance, if we discovered that calling it with a certain event caused a VM to go down (suspend your disbelief, work with me) | |
| 14:21:43 | artom | We'd fix that, regardless of how silly the "use case" (or lack thereof) was | |
| 14:21:55 | bhagyashri_s | efried, jaypipes, bauzas: Hi, Addressed review comments on https://review.openstack.org/#/c/560459 and replied to few comments. Request to review the same. Thank you in advanced :) | |
| 14:22:17 | mdbooth | jaypipes: Don't get me wrong, I've wasted many, many curse words on swap_volume. However, I don't see how you can implement this without something like it. | |
| 14:22:54 | mriedem | mdbooth: have you opened a nova bug for this yet? | |
| 14:24:12 | mdbooth | mriedem: No, but I will. I was kinda hoping somebody, either Nova or Cinder, would pipe up with a simple fix. | |
| 14:24:28 | mriedem | mdbooth: i can but wanted a but quick | |
| 14:25:08 | artom | mdbooth, I mean, I did? | |
| 14:25:11 | artom | Return a 400? | |
| 14:25:19 | mdbooth | Oh, I think you're saying that the direct call use case is relevant here because it's all that's stopping us from implementing this in cinder? | |
| 14:25:24 | artom | jaypipes seems on board with it | |
| 14:25:43 | mdbooth | In which case, yeah. | |
| 14:26:03 | mriedem | mdbooth: yes | |
| 14:26:10 | mriedem | i would prefer to just put the blocker in cinder | |
| 14:26:21 | mriedem | reasons are in my ML reply | |
| 14:26:44 | mriedem | 1. source of truth so less racy and 2. no rollbacks for retype or live migration for volumes | |
| 14:26:47 | artom | mriedem, so Cinder would barf if we "fix" it *just* in Nova, right? | |
| 14:26:51 | openstackgerrit | Jan Gutter proposed openstack/nova master: Add support for vrouter HW offloads https://review.openstack.org/572082 | |
| 14:26:55 | mriedem | artom: yes, but way late | |
| 14:27:00 | smcginnis | I haven't been following the discussion at all, but if we want to block something in cinder, make sure it makes sense for use cases outside of nova. | |