| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-17 | |||
| 11:22:50 | sean-k-mooney | :P | |
| 11:23:04 | artom | aspiers, time to introduce you to this classic: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 11:23:10 | aspiers | the word "evacuate" implies the VM is still there, but in these cases it is not | |
| 11:23:11 | sean-k-mooney | yep ^ | |
| 11:23:17 | aspiers | artom: I read that years ago ;-) | |
| 11:23:27 | sean-k-mooney | aspiers: it is for things with ephemeral storage | |
| 11:23:34 | sean-k-mooney | * non ephmeral storage | |
| 11:23:50 | sean-k-mooney | aspiers: its basically just a cold migration for that case | |
| 11:24:22 | aspiers | the implementation is like cold migration, but semantically it is not | |
| 11:24:25 | sean-k-mooney | its when you use the default image backend e.g. qcow2 and its not on shared stoage that we acrully recreate it from glance and loose data | |
| 11:24:29 | cdent | sean-k-mooney: so in that case either the person at bug https://bugs.launchpad.net/nova/+bug/1829479 is on an old version (as I thought we had fixed migration-like allocations bugs) or we've got remaining migration-like allocations bugs | |
| 11:24:30 | openstack | Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,New] | |
| 11:25:22 | sean-k-mooney | cdent: we proably have a remaing bug | |
| 11:26:07 | aspiers | artom: here's another "classic" for you ;-) https://youtu.be/lddtWUP_IKQ?t=751 | |
| 11:26:12 | sean-k-mooney | cdent: im not sure if we actullly do the whole create a migration contex use it for allocation and swap at the end | |
| 11:26:43 | artom | aspiers, oh, hah | |
| 11:26:50 | aspiers | sean-k-mooney, cdent: that video will help you understand where I'm coming from too | |
| 11:27:04 | artom | aspiers, this is like the time I naively asked about your experience with Gerrit, isn't it :P | |
| 11:27:12 | aspiers | artom: haha, maybe a bit ;-) | |
| 11:27:26 | aspiers | I've been working on compute HA in nova for about 4-5 years | |
| 11:27:29 | sean-k-mooney | cdent: or it could be that we delete the old allocation on the source node and that does not run in the evacuate case | |
| 11:27:50 | aspiers | artom: but don't feel bad, there's no good reason why you should have known that ;) | |
| 11:28:01 | cdent | that's what the bug report is saying, yes | |
| 11:28:03 | sean-k-mooney | aspiers: that was a poor life choice :P | |
| 11:28:15 | aspiers | sean-k-mooney: LOL, yeah probably ;-) | |
| 11:28:20 | artom | I see to have a lot of feet in my mouth, maybe wash them out with aforementioned coffee | |
| 11:28:25 | artom | *seem | |
| 11:28:35 | aspiers | sean-k-mooney: I mean, not *exclusively* compute HA ... I've done a lot of other stuff too ;) | |
| 11:29:17 | sean-k-mooney | aspiers: so for https://review.opendev.org/#/c/643578/3 back to the ptg session your prefere is openstack server resuerect | |
| 11:29:28 | aspiers | sean-k-mooney: yes exactly | |
| 11:30:26 | aspiers | "evacuate" and "migrate" both imply that everything is currently running (mostly) fine | |
| 11:30:43 | artom | Well | |
| 11:30:48 | aspiers | "resurrect" implies (correctly) "oh shit, something already blew up, so let's clean up after it" | |
| 11:30:55 | artom | The compute service/control plane has to be down | |
| 11:31:07 | artom | But the data plane? IIUC the instance can be running, no? | |
| 11:31:12 | sean-k-mooney | aspiers: ya that why i called the new command recreate but i can update it to resurect | |
| 11:31:33 | sean-k-mooney | i spell recreate more consitenly however but i can live with that | |
| 11:31:36 | aspiers | artom: which scenario are you talking about exactly? there are a few here | |
| 11:31:45 | sean-k-mooney | artom: we dont | |
| 11:31:45 | artom | aspiers, evacuate | |
| 11:32:04 | sean-k-mooney | artom: but we require that you have marked the compute node as disabled before we allow it | |
| 11:32:12 | aspiers | artom: in this context you're gonna have to use more than one word to describe the scenario sorry ;-) | |
| 11:32:21 | aspiers | the whole problem is the ambiguity | |
| 11:32:23 | sean-k-mooney | artom: and we require the operator to confim that it is dead before the evac | |
| 11:32:32 | aspiers | sean-k-mooney: actually not always | |
| 11:32:44 | sean-k-mooney | aspiers: that is the api contract | |
| 11:32:45 | aspiers | sean-k-mooney: OOB fencing can *ensure* that the VM is dead | |
| 11:32:53 | sean-k-mooney | sure | |
| 11:33:04 | sean-k-mooney | but the api contract is only call this if it is dead | |
| 11:33:06 | aspiers | sean-k-mooney: i.e. it can be automated. but yes we're on the same page I think | |
| 11:33:10 | aspiers | exactly | |
| 11:33:58 | aspiers | artom: there are multiple failure scenarios here to consider | |
| 11:34:03 | sean-k-mooney | if you violate that contract and you get 2 instance trying to read/write to the same non multi atach volume some how we just say "have fun" and walk away smiling | |
| 11:34:38 | openstackgerrit | Miguel Ángel Herranz Trillo proposed openstack/nova master: Fix type error on call to mount device https://review.opendev.org/659780 | |
| 11:34:38 | aspiers | sean-k-mooney: exactly, that is the main message of 3 talks I have given on this | |
| 11:35:02 | aspiers | in Austin, Boston, and OpenStack Day Israel | |
| 11:35:31 | aspiers | e.g. https://youtu.be/lddtWUP_IKQ?t=366 | |
| 11:35:34 | artom | aspiers, evacuate the nova client command | |
| 11:35:43 | aspiers | (because exploding kittens is fun) | |
| 11:36:04 | artom | And my thought was with respect to resurrect, that the *instance* might not be dead, even if it's unmanageable because the host it's on *is* dead | |
| 11:36:13 | sean-k-mooney | ill convert https://review.opendev.org/#/c/643578/3 to resurect and reupload it proably on monday | |
| 11:36:19 | artom | So "resurrect" might not be the best word | |
| 11:36:34 | aspiers | sean-k-mooney: ++ | |
| 11:36:49 | aspiers | artom: as sean-k-mooney said, the API contract is that the instance *must* be dead before calling the API | |
| 11:36:49 | sean-k-mooney | artom: well its like if your heart stoped | |
| 11:37:05 | artom | aspiers, ah, didn't know that, ok then | |
| 11:37:07 | sean-k-mooney | you were technically dead for a bit and this operation is like a defibulator | |
| 11:37:45 | aspiers | artom: that's why the force_down API was introduced ages ago - to get nova to realise quicker that it's dead, rather than waiting for its own RPC timeout | |
| 11:38:09 | sean-k-mooney | actully not just for that usecase | |
| 11:38:30 | aspiers | OK, that was one use case at least | |
| 11:38:49 | sean-k-mooney | there were case where for example the root disk and the disk where the vms images were seperate | |
| 11:39:19 | sean-k-mooney | so the agent and libvirt could be fine but when the backing disk died for the image and they crashed you wanted to be able to mark the host as down | |
| 11:39:28 | sean-k-mooney | or similar for network issues | |
| 11:39:53 | aspiers | right | |
| 11:40:11 | sean-k-mooney | aspiers: were you working with emma foley on the collectd/doctor/vitriage demoes? | |
| 11:40:23 | aspiers | artom: https://youtu.be/lddtWUP_IKQ?t=648 covers some of the failure modes, but you and sean-k-mooney are both right that there are other failure modes based on different network failures | |
| 11:41:02 | aspiers | sean-k-mooney: nope not heard of that, but with my self-healing SIG hat on, I'd really like to learn more about it! | |
| 11:41:06 | aspiers | sean-k-mooney: got any URLs? | |
| 11:41:31 | sean-k-mooney | there have been at least 3 summit presentation on it | |
| 11:41:47 | sean-k-mooney | the effort was driven via opnfv | |
| 11:41:54 | aspiers | damn, thought I'd been to all the vitrage presentations | |
| 11:43:36 | aspiers | so I'm just looking through my old decks and found this https://aspiers.github.io/openstack-day-israel-2017-compute-ha/#/comparison | |
| 11:43:45 | aspiers | had forgotten about the "needs improvements in nova" bit | |
| 11:43:55 | aspiers | trying to remember the details of that | |
| 11:44:11 | aspiers | I think it was around the reliability of the evacuate API | |
| 11:44:49 | cdent | that's some can of worms I opened | |
| 11:45:04 | aspiers | bad cdent! | |
| 11:45:11 | aspiers | keep quiet in future ;-) | |
| 11:45:18 | cdent | unpossible! | |
| 11:45:26 | aspiers | j/k, glad to be reminded about this stuff | |
| 11:45:52 | cdent | I hope that somehow this leads to that bug getting more attention. people get squeamish about orphaned providers | |
| 11:46:06 | cdent | especially because we have that damn uniq constraint on rp names | |
| 11:46:24 | cdent | which is the source of a ton of support requests | |
| 11:46:47 | cdent | "I was messing around and one of my compute nodes got rejiggered and it says can't create a resource provider" | |
| 11:46:56 | cdent | turns out the old one with the same name still exists | |
| 11:47:54 | aspiers | cdent: sorry to hijack your discussion with a big tangent | |
| 11:48:36 | aspiers | sean-k-mooney, artom: I think this 4-year-old blog post is still very relevant http://blog.clusterlabs.org/blog/2015/living-intersection-pets-cattle | |
| 11:48:44 | cdent | I tend to think the reason tangents happen is because there is insufficient shared understanding and that the tangents are necessary to get it | |
| 11:48:48 | aspiers | yeah | |
| 11:49:07 | cdent | but there is so little shared understanding in nova that is sometimes seems like we are on a tangent all the time | |