| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-17 | |||
| 08:45:07 | kashyap | aspiers: ACKed your change, FWIW | |
| 08:45:26 | aspiers | kashyap: yep, thanks! | |
| 08:45:56 | kashyap | aspiers: I think your change is ready to merge. Let's check with efried (or stephenfin) when they're about. | |
| 08:46:14 | kashyap | aspiers: Hope it won't be delayed, so that you can maintain the momentum. | |
| 08:49:04 | aspiers | Well it's already been delayed a few months ;) but thankfully git makes this much less of an issue. I remember the awful CVS days where the only place for all your pending unmerged work was mushed up together in the work tree, and you had to untangle each new commit-to-be from that | |
| 08:49:17 | aspiers | Drove me crazy | |
| 08:56:00 | jangutter | aspiers: The absolute _best_ thing about git is when you learn about rebasing. And the nice thing about gerrit is it fills in the gaps (tracking iterations of the same "patch queue"). | |
| 08:56:41 | aspiers | jangutter: agreed :) git has lots of "best" things though, e.g. I remember when I learnt about the reflog it blew my mind | |
| 08:56:58 | aspiers | ditto git-rerere | |
| 08:57:10 | aspiers | so many cool tricks | |
| 08:58:05 | jangutter | aspiers: who knew that a content-addressable file system accidentally solved the "version control" issue :-p | |
| 08:58:27 | aspiers | :D | |
| 10:15:57 | kashyap | aspiers: Yeah, I see what you mean. I didn't have the misfortune to use CVS, but I did start with SVN, though | |
| 10:16:08 | aspiers | I started with RCS :-o | |
| 10:34:59 | kashyap | Hope you've fully recovered | |
| 10:35:26 | aspiers | LOL | |
| 10:35:43 | aspiers | Actually RCS can be surprisingly effective for particular use cases | |
| 10:35:54 | aspiers | but yeah, it's very limited | |
| 10:36:35 | aspiers | I actually talked a bit about the evolutionary history of version control systems in this interview http://episodes.gitminutes.com/2015/03/gitminutes-32-adam-spiers-on-git-deps.html | |
| 10:37:27 | aspiers | according to the index that starts around 07:08 | |
| 10:52:55 | cdent | What are the options for a VM after it is evacuated? I ask in relation to this new bug: https://bugs.launchpad.net/nova/+bug/1829479 | |
| 10:52:57 | openstack | Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,New] | |
| 11:20:17 | sean-k-mooney | cdent: in what sense | |
| 11:20:47 | sean-k-mooney | cdent: if it is evacuated sucessfully its as if ti was migrated | |
| 11:21:10 | sean-k-mooney | cdent: assuming it was on shared storage its state was not even lost | |
| 11:21:25 | sean-k-mooney | cdent: althogh if it was on epheraml stoage it wasd effectivly a new instance | |
| 11:21:32 | cdent | sean-k-mooney: So evacuate means "recreate this vm somewhere else"? | |
| 11:21:39 | sean-k-mooney | cdent: yes | |
| 11:21:53 | sean-k-mooney | but keep the same ips and volumes | |
| 11:22:00 | cdent | So shouldn't that mean that when the evacuate finishes an allocation is made which replaces the old allocations? | |
| 11:22:06 | artom | Rebuild on a different host, basically | |
| 11:22:16 | artom | So yeah, allocations need to move | |
| 11:22:19 | sean-k-mooney | so if its boot from volume or you hapeend to have the rbd image backend you keep you root disk | |
| 11:22:33 | aspiers | PLEASE can we use the word "resurrect" instead of evacuate wherever possible | |
| 11:22:42 | sean-k-mooney | aspiers: no | |
| 11:22:50 | sean-k-mooney | :P | |
| 11:23:04 | artom | aspiers, time to introduce you to this classic: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 11:23:10 | aspiers | the word "evacuate" implies the VM is still there, but in these cases it is not | |
| 11:23:11 | sean-k-mooney | yep ^ | |
| 11:23:17 | aspiers | artom: I read that years ago ;-) | |
| 11:23:27 | sean-k-mooney | aspiers: it is for things with ephemeral storage | |
| 11:23:34 | sean-k-mooney | * non ephmeral storage | |
| 11:23:50 | sean-k-mooney | aspiers: its basically just a cold migration for that case | |
| 11:24:22 | aspiers | the implementation is like cold migration, but semantically it is not | |
| 11:24:25 | sean-k-mooney | its when you use the default image backend e.g. qcow2 and its not on shared stoage that we acrully recreate it from glance and loose data | |
| 11:24:29 | cdent | sean-k-mooney: so in that case either the person at bug https://bugs.launchpad.net/nova/+bug/1829479 is on an old version (as I thought we had fixed migration-like allocations bugs) or we've got remaining migration-like allocations bugs | |
| 11:24:30 | openstack | Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,New] | |
| 11:25:22 | sean-k-mooney | cdent: we proably have a remaing bug | |
| 11:26:07 | aspiers | artom: here's another "classic" for you ;-) https://youtu.be/lddtWUP_IKQ?t=751 | |
| 11:26:12 | sean-k-mooney | cdent: im not sure if we actullly do the whole create a migration contex use it for allocation and swap at the end | |
| 11:26:43 | artom | aspiers, oh, hah | |
| 11:26:50 | aspiers | sean-k-mooney, cdent: that video will help you understand where I'm coming from too | |
| 11:27:04 | artom | aspiers, this is like the time I naively asked about your experience with Gerrit, isn't it :P | |
| 11:27:12 | aspiers | artom: haha, maybe a bit ;-) | |
| 11:27:26 | aspiers | I've been working on compute HA in nova for about 4-5 years | |
| 11:27:29 | sean-k-mooney | cdent: or it could be that we delete the old allocation on the source node and that does not run in the evacuate case | |
| 11:27:50 | aspiers | artom: but don't feel bad, there's no good reason why you should have known that ;) | |
| 11:28:01 | cdent | that's what the bug report is saying, yes | |
| 11:28:03 | sean-k-mooney | aspiers: that was a poor life choice :P | |
| 11:28:15 | aspiers | sean-k-mooney: LOL, yeah probably ;-) | |
| 11:28:20 | artom | I see to have a lot of feet in my mouth, maybe wash them out with aforementioned coffee | |
| 11:28:25 | artom | *seem | |
| 11:28:35 | aspiers | sean-k-mooney: I mean, not *exclusively* compute HA ... I've done a lot of other stuff too ;) | |
| 11:29:17 | sean-k-mooney | aspiers: so for https://review.opendev.org/#/c/643578/3 back to the ptg session your prefere is openstack server resuerect | |
| 11:29:28 | aspiers | sean-k-mooney: yes exactly | |
| 11:30:26 | aspiers | "evacuate" and "migrate" both imply that everything is currently running (mostly) fine | |
| 11:30:43 | artom | Well | |
| 11:30:48 | aspiers | "resurrect" implies (correctly) "oh shit, something already blew up, so let's clean up after it" | |
| 11:30:55 | artom | The compute service/control plane has to be down | |
| 11:31:07 | artom | But the data plane? IIUC the instance can be running, no? | |
| 11:31:12 | sean-k-mooney | aspiers: ya that why i called the new command recreate but i can update it to resurect | |
| 11:31:33 | sean-k-mooney | i spell recreate more consitenly however but i can live with that | |
| 11:31:36 | aspiers | artom: which scenario are you talking about exactly? there are a few here | |
| 11:31:45 | sean-k-mooney | artom: we dont | |
| 11:31:45 | artom | aspiers, evacuate | |
| 11:32:04 | sean-k-mooney | artom: but we require that you have marked the compute node as disabled before we allow it | |
| 11:32:12 | aspiers | artom: in this context you're gonna have to use more than one word to describe the scenario sorry ;-) | |
| 11:32:21 | aspiers | the whole problem is the ambiguity | |
| 11:32:23 | sean-k-mooney | artom: and we require the operator to confim that it is dead before the evac | |
| 11:32:32 | aspiers | sean-k-mooney: actually not always | |
| 11:32:44 | sean-k-mooney | aspiers: that is the api contract | |
| 11:32:45 | aspiers | sean-k-mooney: OOB fencing can *ensure* that the VM is dead | |
| 11:32:53 | sean-k-mooney | sure | |
| 11:33:04 | sean-k-mooney | but the api contract is only call this if it is dead | |
| 11:33:06 | aspiers | sean-k-mooney: i.e. it can be automated. but yes we're on the same page I think | |
| 11:33:10 | aspiers | exactly | |
| 11:33:58 | aspiers | artom: there are multiple failure scenarios here to consider | |
| 11:34:03 | sean-k-mooney | if you violate that contract and you get 2 instance trying to read/write to the same non multi atach volume some how we just say "have fun" and walk away smiling | |
| 11:34:38 | openstackgerrit | Miguel Ángel Herranz Trillo proposed openstack/nova master: Fix type error on call to mount device https://review.opendev.org/659780 | |
| 11:34:38 | aspiers | sean-k-mooney: exactly, that is the main message of 3 talks I have given on this | |
| 11:35:02 | aspiers | in Austin, Boston, and OpenStack Day Israel | |
| 11:35:31 | aspiers | e.g. https://youtu.be/lddtWUP_IKQ?t=366 | |
| 11:35:34 | artom | aspiers, evacuate the nova client command | |
| 11:35:43 | aspiers | (because exploding kittens is fun) | |
| 11:36:04 | artom | And my thought was with respect to resurrect, that the *instance* might not be dead, even if it's unmanageable because the host it's on *is* dead | |
| 11:36:13 | sean-k-mooney | ill convert https://review.opendev.org/#/c/643578/3 to resurect and reupload it proably on monday | |
| 11:36:19 | artom | So "resurrect" might not be the best word | |
| 11:36:34 | aspiers | sean-k-mooney: ++ | |
| 11:36:49 | aspiers | artom: as sean-k-mooney said, the API contract is that the instance *must* be dead before calling the API | |
| 11:36:49 | sean-k-mooney | artom: well its like if your heart stoped | |