Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-01
10:20:31 kashyap Do they have requisite knowledge to work on it? Or should I reassign the bug back to 'nodbody'?
10:22:42 gmann cdent: yea, api meeting. me too to skip.
10:27:20 johnthetubaguy kashyap: we are training him up, he is struggling a bit on it though
10:27:53 kashyap johnthetubaguy: Me nods. That bug is quite complicated. Maybe a lower-hanging fruit might be better?
10:27:57 johnthetubaguy kashyap: he is testing out the fix in an environment at the moment I believe
10:28:05 kashyap johnthetubaguy: I say that because, right now QEMU migration devs are asking: What is Nova's stance on postcopy
10:28:13 kashyap So, I lost complete memory of it
10:28:19 kashyap Maybe mdbooth can shed some light there?
10:28:33 johnthetubaguy kashyap: support is in, works with majority of neutron back ends, but not all, AFAIK
10:29:06 johnthetubaguy we discussed it alot at the summit, and siva_krishnan is implementing what we agreed would be the "quick fix", calling neutron earlier in the process
10:29:41 mdbooth johnthetubaguy: IIRC the problem with that change wasn't so much working out what needed to be done, as finding somebody with enough arcane neutron knowledge to know which backends allowed it to be done.
10:29:49 johnthetubaguy would love to trigger of QEMU events to get more exact timing of when the post copy occurs, but the basic structure is going in the above change
10:30:23 johnthetubaguy mdbooth: I am confused then, neutron folks agreed this should be OK at the summit, what changed from that?
10:30:32 mdbooth johnthetubaguy: The qemu events things isn't required at all in this instance, because the change happens synchronously on the source.
10:30:41 kashyap johnthetubaguy: I have asked David Gilbert of QEMU to join here. He can ask questions to his heart's content, and he goes out of his way to explain
10:30:44 davidgiluk kashyap: nod
10:30:54 kashyap davidgiluk: Ah, you are here :-)
10:30:55 mdbooth i.e. there's no need to respond to an event from a call which you just made in code you're still executing
10:31:00 johnthetubaguy mdbooth: I keep being told its not syncronus
10:31:13 mdbooth What's not synchronous about it?
10:31:19 johnthetubaguy you just say, please do post copy at end of next iterration, I believe
10:31:30 johnthetubaguy so when we call post copy, it doesn't immediatly happen, I was told
10:31:31 kashyap davidgiluk: So, this is what johnthetubaguy was saying, when I asked him about "What is Nova's stance on postcopy": < johnthetubaguy> support is in, works with majority of neutron back ends, but not all, AFAIK
10:32:13 johnthetubaguy there can be a small network outage with some backends, during the post copy process (which ironically speeds up the live-migrate, of course) doing work to avoid that
10:32:24 mdbooth johnthetubaguy: In _live_migration_monitor we do:
10:32:24 openstack Launchpad bug 1605016 in OpenStack Compute (nova) "Post copy live migration interrupts network connectivity" [High,In progress] https://launchpad.net/bugs/1605016 - Assigned to Sivasathurappan Radhakrishnan (siva-radhakrishnan)
10:32:24 davidgiluk kashyap: Yeh so my understanding was it was waiting on a fix to lp 1605016 for the others, with https://review.openstack.org/#/c/413021/9 being a proposed fix
10:32:32 mdbooth libvirt_migrate.trigger_postcopy_switch(guest,
10:32:32 mdbooth previous_data_remaining, migration.status)):
10:32:32 mdbooth info.memory_iteration, info.data_remaining,
10:32:32 mdbooth libvirt_migrate.should_switch_to_postcopy(
10:32:32 mdbooth if (is_post_copy_enabled and
10:32:33 mdbooth instance,
10:32:35 mdbooth migration)
10:32:51 johnthetubaguy mdbooth: its the libvirt call that I am told is async
10:33:07 johnthetubaguy mdbooth: it just request post copy at some point in the future, and returns right away, before actually triggering it
10:33:12 mdbooth johnthetubaguy: Well we might switch a few milliseconds early
10:33:16 johnthetubaguy mdbooth: pkoniszewski has the details on that
10:33:19 mdbooth Rather than a few milliseconds late
10:33:39 johnthetubaguy mdbooth: I was told more like seconds, but yeah, I was betting on milliseconds myself
10:33:46 davidgiluk the postcopy transition happens at the end of a pass, and libvirt gets involved before starting the destination
10:34:03 johnthetubaguy mdbooth: either way, I vote for the approach siva_krishnan has in the works already, just need to iron things out
10:34:12 johnthetubaguy davidgiluk: sorry, whats waiting on the fix/
10:34:23 johnthetubaguy s/ / / ? /
10:34:32 davidgiluk johnthetubaguy: The support for other networking configs
10:35:05 johnthetubaguy davidgiluk: that bug fix should help reduce the network outage some networking configs currently see, yes
10:35:43 davidgiluk johnthetubaguy: < johnthetubaguy> would love to trigger of QEMU events to get more exact timing of when the post copy occurs, but the basic structure is going in the above change
10:35:48 davidgiluk johnthetubaguy: We already emit the events
10:36:20 johnthetubaguy davidgiluk: agreed its omitted, we first need all the structures in place to call neutron at the right time, the current patch
10:36:34 johnthetubaguy oops emitted
10:36:43 davidgiluk generate :-)
10:37:32 mdbooth Well, this merged: https://review.openstack.org/#/c/402318/ :(
10:37:49 mdbooth The libvirt driver becomes slightly less fathomable for no appreciable gain.
10:38:11 davidgiluk johnthetubaguy: you should probably loop in jdenemar to check what it looks like for you from ontop of libvirt - but you should have all the events you need to know when postcopy kicks in
10:38:35 kashyap mdbooth: Propose a revert?
10:38:47 johnthetubaguy davidgiluk: thats the plan for the follow up fix
10:38:54 mdbooth kashyap: I proposed not merging it, which didn't work.
10:39:01 kashyap mdbooth: From glancing at your comments on the review
10:39:06 davidgiluk johnthetubaguy: OK, great; if you do need anything more from qemu, just ask
10:39:09 johnthetubaguy mdbooth: sorry, got pulled into meetings after that one, so I didn't get chance to delay it
10:39:10 kashyap mdbooth: No one has bothered to respond with a counter-argument
10:39:17 johnthetubaguy mdbooth: I can fast revert it if you want
10:39:36 kashyap Then, you have by all means, a right to revert it, and restart the discussion, and make whoever proposed it to respond
10:39:48 mdbooth johnthetubaguy: If you wouldn't mind. I'm assuming, btw, that there are no real users being impacted by this.
10:40:40 johnthetubaguy mdbooth: no idea, the SUSE folks proposed this, and I believe they support users on Xen + Libivrt + Nova, so I was assuming yes
10:41:15 mdbooth It's been broken forever though, right?
10:41:36 johnthetubaguy mdbooth: sure, I assumed they were trying to change that for one of their users that are complaining
10:42:09 mdbooth I just don't want to add more sticky tape without investigating fixing the root cause.
10:42:25 mdbooth I don't see any evidence anybody looked for the root cause.
10:43:17 johnthetubaguy mdbooth: I added my assumption of the cause in the review, well the root cause is Xen being very picky about the device names
10:43:55 mdbooth We need to either be picky earlier, or translate it earlier.
10:44:18 johnthetubaguy mdbooth: the problem is, you don't know if its Xen until it hits the compute node
10:44:19 mdbooth Sticking a band-aid at the lowest possible level is just going to be a source of bugs
10:44:42 mdbooth Yeah, that occurred to me. No immediate answer to that as I haven't given it any thought.
10:44:53 mdbooth Seems that nobody else has either, though.
10:45:46 mdbooth Might be something to be resolved in driver.spawn, and attach/detach volume?
10:46:03 mdbooth That way it's resolved at the time it's created
10:47:55 mdbooth Do those names have any semantic meaning at the api level beyond 'the Nth disk' anyway? I don't think we support multiple controller types, do we?
10:48:24 johnthetubaguy we support multiple hypervisors in a single cloud, at least in theory
10:48:38 mdbooth If ^^^ is a valid assumption, ComputeManager could rewrite it to a number, and the driver could change it back to whatever it wanted.
10:48:58 johnthetubaguy I don't think we even respect the number any more, but it could
10:49:01 mdbooth johnthetubaguy: It's in a single instance. i.e. could we have both sda and vda in the same instance?
10:49:36 johnthetubaguy sda would always fail to attach, all disks need to be vd* I believe, or xvd*
10:50:17 mdbooth Right, so computemanager would see it got 'sda' from the api, translate that to '0', and the driver would change it to 'sda', 'vda', or 'xvda' as required.
10:51:08 johnthetubaguy seems like a good refactor to do on top of the bug fix
10:51:54 mdbooth johnthetubaguy: The bug fix makes the existing code worse, though, and I see no evidence that anybody looked for a root cause.
10:52:47 mdbooth Given that it's obviously not urgent, I don't see why we would choose to take more tech debt for that.
10:53:28 mdbooth Anyway, it's not that important in itself.
10:53:44 mdbooth Just death by 1,000 cuts.
10:53:55 johnthetubaguy it breaks a pretty fundamental API feature, that seemed fairly important to be from a parity point of view, but I duno really.
10:56:54 mdbooth johnthetubaguy: Yeah, I know. But it's been broken a long time and I'd have preferred to see an attempt to fix it without adding tech debt first.
10:57:34 johnthetubaguy mdbooth: that my preference too, but its a balancing act
10:58:11 johnthetubaguy I was very boarderline on -1 +2 myself
10:58:14 mdbooth If it's important to somebody, they'll fix it. You even suggested where to look, and I agreed.
10:58:48 bauzas lyarwood: nice catch, +2d https://review.openstack.org/#/c/427267/2
10:59:13 sfinucan bauzas: Could you take a look at https://review.openstack.org/#/c/392643/, please? :)
10:59:14 bauzas johnthetubaguy: in case you wanna fix a nasty bug with an easy peasy fix ^
10:59:35 bauzas sfinucan: roger
10:59:39 mdbooth johnthetubaguy: Now ^^^ is a nasty bug :)
10:59:56 mdbooth Ah, not that one.

Earlier   Later