| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-17 | |||
| 08:49:17 | aspiers | Drove me crazy | |
| 08:56:00 | jangutter | aspiers: The absolute _best_ thing about git is when you learn about rebasing. And the nice thing about gerrit is it fills in the gaps (tracking iterations of the same "patch queue"). | |
| 08:56:41 | aspiers | jangutter: agreed :) git has lots of "best" things though, e.g. I remember when I learnt about the reflog it blew my mind | |
| 08:56:58 | aspiers | ditto git-rerere | |
| 08:57:10 | aspiers | so many cool tricks | |
| 08:58:05 | jangutter | aspiers: who knew that a content-addressable file system accidentally solved the "version control" issue :-p | |
| 08:58:27 | aspiers | :D | |
| 10:15:57 | kashyap | aspiers: Yeah, I see what you mean. I didn't have the misfortune to use CVS, but I did start with SVN, though | |
| 10:16:08 | aspiers | I started with RCS :-o | |
| 10:34:59 | kashyap | Hope you've fully recovered | |
| 10:35:26 | aspiers | LOL | |
| 10:35:43 | aspiers | Actually RCS can be surprisingly effective for particular use cases | |
| 10:35:54 | aspiers | but yeah, it's very limited | |
| 10:36:35 | aspiers | I actually talked a bit about the evolutionary history of version control systems in this interview http://episodes.gitminutes.com/2015/03/gitminutes-32-adam-spiers-on-git-deps.html | |
| 10:37:27 | aspiers | according to the index that starts around 07:08 | |
| 10:52:55 | cdent | What are the options for a VM after it is evacuated? I ask in relation to this new bug: https://bugs.launchpad.net/nova/+bug/1829479 | |
| 10:52:57 | openstack | Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,New] | |
| 11:20:17 | sean-k-mooney | cdent: in what sense | |
| 11:20:47 | sean-k-mooney | cdent: if it is evacuated sucessfully its as if ti was migrated | |
| 11:21:10 | sean-k-mooney | cdent: assuming it was on shared storage its state was not even lost | |
| 11:21:25 | sean-k-mooney | cdent: althogh if it was on epheraml stoage it wasd effectivly a new instance | |
| 11:21:32 | cdent | sean-k-mooney: So evacuate means "recreate this vm somewhere else"? | |
| 11:21:39 | sean-k-mooney | cdent: yes | |
| 11:21:53 | sean-k-mooney | but keep the same ips and volumes | |
| 11:22:00 | cdent | So shouldn't that mean that when the evacuate finishes an allocation is made which replaces the old allocations? | |
| 11:22:06 | artom | Rebuild on a different host, basically | |
| 11:22:16 | artom | So yeah, allocations need to move | |
| 11:22:19 | sean-k-mooney | so if its boot from volume or you hapeend to have the rbd image backend you keep you root disk | |
| 11:22:33 | aspiers | PLEASE can we use the word "resurrect" instead of evacuate wherever possible | |
| 11:22:42 | sean-k-mooney | aspiers: no | |
| 11:22:50 | sean-k-mooney | :P | |
| 11:23:04 | artom | aspiers, time to introduce you to this classic: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 11:23:10 | aspiers | the word "evacuate" implies the VM is still there, but in these cases it is not | |
| 11:23:11 | sean-k-mooney | yep ^ | |
| 11:23:17 | aspiers | artom: I read that years ago ;-) | |
| 11:23:27 | sean-k-mooney | aspiers: it is for things with ephemeral storage | |
| 11:23:34 | sean-k-mooney | * non ephmeral storage | |
| 11:23:50 | sean-k-mooney | aspiers: its basically just a cold migration for that case | |
| 11:24:22 | aspiers | the implementation is like cold migration, but semantically it is not | |
| 11:24:25 | sean-k-mooney | its when you use the default image backend e.g. qcow2 and its not on shared stoage that we acrully recreate it from glance and loose data | |
| 11:24:29 | cdent | sean-k-mooney: so in that case either the person at bug https://bugs.launchpad.net/nova/+bug/1829479 is on an old version (as I thought we had fixed migration-like allocations bugs) or we've got remaining migration-like allocations bugs | |
| 11:24:30 | openstack | Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,New] | |
| 11:25:22 | sean-k-mooney | cdent: we proably have a remaing bug | |
| 11:26:07 | aspiers | artom: here's another "classic" for you ;-) https://youtu.be/lddtWUP_IKQ?t=751 | |
| 11:26:12 | sean-k-mooney | cdent: im not sure if we actullly do the whole create a migration contex use it for allocation and swap at the end | |
| 11:26:43 | artom | aspiers, oh, hah | |
| 11:26:50 | aspiers | sean-k-mooney, cdent: that video will help you understand where I'm coming from too | |
| 11:27:04 | artom | aspiers, this is like the time I naively asked about your experience with Gerrit, isn't it :P | |
| 11:27:12 | aspiers | artom: haha, maybe a bit ;-) | |
| 11:27:26 | aspiers | I've been working on compute HA in nova for about 4-5 years | |
| 11:27:29 | sean-k-mooney | cdent: or it could be that we delete the old allocation on the source node and that does not run in the evacuate case | |
| 11:27:50 | aspiers | artom: but don't feel bad, there's no good reason why you should have known that ;) | |
| 11:28:01 | cdent | that's what the bug report is saying, yes | |
| 11:28:03 | sean-k-mooney | aspiers: that was a poor life choice :P | |
| 11:28:15 | aspiers | sean-k-mooney: LOL, yeah probably ;-) | |
| 11:28:20 | artom | I see to have a lot of feet in my mouth, maybe wash them out with aforementioned coffee | |
| 11:28:25 | artom | *seem | |
| 11:28:35 | aspiers | sean-k-mooney: I mean, not *exclusively* compute HA ... I've done a lot of other stuff too ;) | |
| 11:29:17 | sean-k-mooney | aspiers: so for https://review.opendev.org/#/c/643578/3 back to the ptg session your prefere is openstack server resuerect | |
| 11:29:28 | aspiers | sean-k-mooney: yes exactly | |
| 11:30:26 | aspiers | "evacuate" and "migrate" both imply that everything is currently running (mostly) fine | |
| 11:30:43 | artom | Well | |
| 11:30:48 | aspiers | "resurrect" implies (correctly) "oh shit, something already blew up, so let's clean up after it" | |
| 11:30:55 | artom | The compute service/control plane has to be down | |
| 11:31:07 | artom | But the data plane? IIUC the instance can be running, no? | |
| 11:31:12 | sean-k-mooney | aspiers: ya that why i called the new command recreate but i can update it to resurect | |
| 11:31:33 | sean-k-mooney | i spell recreate more consitenly however but i can live with that | |
| 11:31:36 | aspiers | artom: which scenario are you talking about exactly? there are a few here | |
| 11:31:45 | artom | aspiers, evacuate | |
| 11:31:45 | sean-k-mooney | artom: we dont | |
| 11:32:04 | sean-k-mooney | artom: but we require that you have marked the compute node as disabled before we allow it | |
| 11:32:12 | aspiers | artom: in this context you're gonna have to use more than one word to describe the scenario sorry ;-) | |
| 11:32:21 | aspiers | the whole problem is the ambiguity | |
| 11:32:23 | sean-k-mooney | artom: and we require the operator to confim that it is dead before the evac | |
| 11:32:32 | aspiers | sean-k-mooney: actually not always | |
| 11:32:44 | sean-k-mooney | aspiers: that is the api contract | |
| 11:32:45 | aspiers | sean-k-mooney: OOB fencing can *ensure* that the VM is dead | |
| 11:32:53 | sean-k-mooney | sure | |
| 11:33:04 | sean-k-mooney | but the api contract is only call this if it is dead | |
| 11:33:06 | aspiers | sean-k-mooney: i.e. it can be automated. but yes we're on the same page I think | |
| 11:33:10 | aspiers | exactly | |
| 11:33:58 | aspiers | artom: there are multiple failure scenarios here to consider | |
| 11:34:03 | sean-k-mooney | if you violate that contract and you get 2 instance trying to read/write to the same non multi atach volume some how we just say "have fun" and walk away smiling | |
| 11:34:38 | aspiers | sean-k-mooney: exactly, that is the main message of 3 talks I have given on this | |
| 11:34:38 | openstackgerrit | Miguel Ángel Herranz Trillo proposed openstack/nova master: Fix type error on call to mount device https://review.opendev.org/659780 | |
| 11:35:02 | aspiers | in Austin, Boston, and OpenStack Day Israel | |
| 11:35:31 | aspiers | e.g. https://youtu.be/lddtWUP_IKQ?t=366 | |
| 11:35:34 | artom | aspiers, evacuate the nova client command | |
| 11:35:43 | aspiers | (because exploding kittens is fun) | |
| 11:36:04 | artom | And my thought was with respect to resurrect, that the *instance* might not be dead, even if it's unmanageable because the host it's on *is* dead | |
| 11:36:13 | sean-k-mooney | ill convert https://review.opendev.org/#/c/643578/3 to resurect and reupload it proably on monday | |
| 11:36:19 | artom | So "resurrect" might not be the best word | |
| 11:36:34 | aspiers | sean-k-mooney: ++ | |
| 11:36:49 | sean-k-mooney | artom: well its like if your heart stoped | |
| 11:36:49 | aspiers | artom: as sean-k-mooney said, the API contract is that the instance *must* be dead before calling the API | |
| 11:37:05 | artom | aspiers, ah, didn't know that, ok then | |
| 11:37:07 | sean-k-mooney | you were technically dead for a bit and this operation is like a defibulator | |
| 11:37:45 | aspiers | artom: that's why the force_down API was introduced ages ago - to get nova to realise quicker that it's dead, rather than waiting for its own RPC timeout | |
| 11:38:09 | sean-k-mooney | actully not just for that usecase | |
| 11:38:30 | aspiers | OK, that was one use case at least | |