| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-17 | |||
| 11:33:06 | aspiers | sean-k-mooney: i.e. it can be automated. but yes we're on the same page I think | |
| 11:33:10 | aspiers | exactly | |
| 11:33:58 | aspiers | artom: there are multiple failure scenarios here to consider | |
| 11:34:03 | sean-k-mooney | if you violate that contract and you get 2 instance trying to read/write to the same non multi atach volume some how we just say "have fun" and walk away smiling | |
| 11:34:38 | aspiers | sean-k-mooney: exactly, that is the main message of 3 talks I have given on this | |
| 11:34:38 | openstackgerrit | Miguel Ángel Herranz Trillo proposed openstack/nova master: Fix type error on call to mount device https://review.opendev.org/659780 | |
| 11:35:02 | aspiers | in Austin, Boston, and OpenStack Day Israel | |
| 11:35:31 | aspiers | e.g. https://youtu.be/lddtWUP_IKQ?t=366 | |
| 11:35:34 | artom | aspiers, evacuate the nova client command | |
| 11:35:43 | aspiers | (because exploding kittens is fun) | |
| 11:36:04 | artom | And my thought was with respect to resurrect, that the *instance* might not be dead, even if it's unmanageable because the host it's on *is* dead | |
| 11:36:13 | sean-k-mooney | ill convert https://review.opendev.org/#/c/643578/3 to resurect and reupload it proably on monday | |
| 11:36:19 | artom | So "resurrect" might not be the best word | |
| 11:36:34 | aspiers | sean-k-mooney: ++ | |
| 11:36:49 | sean-k-mooney | artom: well its like if your heart stoped | |
| 11:36:49 | aspiers | artom: as sean-k-mooney said, the API contract is that the instance *must* be dead before calling the API | |
| 11:37:05 | artom | aspiers, ah, didn't know that, ok then | |
| 11:37:07 | sean-k-mooney | you were technically dead for a bit and this operation is like a defibulator | |
| 11:37:45 | aspiers | artom: that's why the force_down API was introduced ages ago - to get nova to realise quicker that it's dead, rather than waiting for its own RPC timeout | |
| 11:38:09 | sean-k-mooney | actully not just for that usecase | |
| 11:38:30 | aspiers | OK, that was one use case at least | |
| 11:38:49 | sean-k-mooney | there were case where for example the root disk and the disk where the vms images were seperate | |
| 11:39:19 | sean-k-mooney | so the agent and libvirt could be fine but when the backing disk died for the image and they crashed you wanted to be able to mark the host as down | |
| 11:39:28 | sean-k-mooney | or similar for network issues | |
| 11:39:53 | aspiers | right | |
| 11:40:11 | sean-k-mooney | aspiers: were you working with emma foley on the collectd/doctor/vitriage demoes? | |
| 11:40:23 | aspiers | artom: https://youtu.be/lddtWUP_IKQ?t=648 covers some of the failure modes, but you and sean-k-mooney are both right that there are other failure modes based on different network failures | |
| 11:41:02 | aspiers | sean-k-mooney: nope not heard of that, but with my self-healing SIG hat on, I'd really like to learn more about it! | |
| 11:41:06 | aspiers | sean-k-mooney: got any URLs? | |
| 11:41:31 | sean-k-mooney | there have been at least 3 summit presentation on it | |
| 11:41:47 | sean-k-mooney | the effort was driven via opnfv | |
| 11:41:54 | aspiers | damn, thought I'd been to all the vitrage presentations | |
| 11:43:36 | aspiers | so I'm just looking through my old decks and found this https://aspiers.github.io/openstack-day-israel-2017-compute-ha/#/comparison | |
| 11:43:45 | aspiers | had forgotten about the "needs improvements in nova" bit | |
| 11:43:55 | aspiers | trying to remember the details of that | |
| 11:44:11 | aspiers | I think it was around the reliability of the evacuate API | |
| 11:44:49 | cdent | that's some can of worms I opened | |
| 11:45:04 | aspiers | bad cdent! | |
| 11:45:11 | aspiers | keep quiet in future ;-) | |
| 11:45:18 | cdent | unpossible! | |
| 11:45:26 | aspiers | j/k, glad to be reminded about this stuff | |
| 11:45:52 | cdent | I hope that somehow this leads to that bug getting more attention. people get squeamish about orphaned providers | |
| 11:46:06 | cdent | especially because we have that damn uniq constraint on rp names | |
| 11:46:24 | cdent | which is the source of a ton of support requests | |
| 11:46:47 | cdent | "I was messing around and one of my compute nodes got rejiggered and it says can't create a resource provider" | |
| 11:46:56 | cdent | turns out the old one with the same name still exists | |
| 11:47:54 | aspiers | cdent: sorry to hijack your discussion with a big tangent | |
| 11:48:36 | aspiers | sean-k-mooney, artom: I think this 4-year-old blog post is still very relevant http://blog.clusterlabs.org/blog/2015/living-intersection-pets-cattle | |
| 11:48:44 | cdent | I tend to think the reason tangents happen is because there is insufficient shared understanding and that the tangents are necessary to get it | |
| 11:48:48 | aspiers | yeah | |
| 11:49:07 | cdent | but there is so little shared understanding in nova that is sometimes seems like we are on a tangent all the time | |
| 11:49:18 | aspiers | haha true | |
| 11:49:22 | cdent | tangents on tangets on tangents | |
| 11:49:43 | aspiers | beekhof and I did a lot of work a few years back considering all the failure cases of nova compute HA | |
| 11:49:59 | aspiers | he was definitely the thought-leader in this space, but he escaped to greener pastures | |
| 11:50:18 | sean-k-mooney | isnt a tangent to a tangent called a normal | |
| 11:50:26 | sean-k-mooney | nova embrases this :P | |
| 11:50:48 | aspiers | if a tangent is a 90 degree turn, then 4 tangents get you back in the original direction, right? | |
| 11:51:45 | sean-k-mooney | aspiers: for what its worth a lot of the presentation i was refering to were framed in the context fo service assurance not self healing or instance ha | |
| 11:52:04 | aspiers | sean-k-mooney: got it | |
| 11:52:26 | cdent | at the limit, N tangents is a circle, no matter the angle of turn | |
| 11:52:40 | cdent | or something | |
| 11:52:44 | aspiers | :D | |
| 11:52:53 | aspiers | https://aspiers.github.io/openstack-day-israel-2017-compute-ha/#/nova-host-alerter is the future converged architecture beekhof and I agreed on, in case anyone is interested | |
| 11:53:21 | sean-k-mooney | cdent: are you suggesting we would go around in circles on things, im shocked :) | |
| 11:53:37 | aspiers | coupled with http://blog.clusterlabs.org/blog/2015/reliable-notifications and masakari adding a layer of retries on top of nova's somewhat lossy workflow, the hope was that this would be more robust | |
| 11:53:44 | aspiers | s/more/sufficiently/ | |
| 11:54:01 | sean-k-mooney | anyway i better file my expense report for the ptg or finance will get annoyed at me | |
| 11:54:24 | aspiers | sean-k-mooney: IIRC one of the concerns was that if scheduler or conductor crashed while handling evacuate API, it would get lost forever - is that still true? | |
| 11:54:38 | cdent | sean-k-mooney: hush | |
| 11:54:48 | sean-k-mooney | aspiers: ya maybe | |
| 11:55:03 | sean-k-mooney | it woudl depend on when it failed i guess | |
| 11:55:19 | sean-k-mooney | if we have not deleted it form the souce node yet then you should be fine | |
| 11:55:22 | aspiers | I think the idea was maybe that masakari could reinvoke the API in those circumstances | |
| 11:55:50 | sean-k-mooney | ya i think if the db says its still on the source node then you can evac again | |
| 11:55:57 | aspiers | but is there even a way to query the API for the state of an evacuate workflow? | |
| 11:56:15 | sean-k-mooney | but depending on the failure mode the vm could be paused/running on another node at that point | |
| 11:56:19 | aspiers | I vaguely remember that being discussed at past summits | |
| 11:56:30 | aspiers | yeah, the devil is in the details I guess | |
| 11:57:20 | aspiers | I don't think there's ever been enough corporate sponsorship from RH or SUSE to spend time really cleaning up all those corner cases | |
| 11:57:21 | sean-k-mooney | if your contol plane if failing you shoudl really stop what your doing and fix that first then go back to fixing your broken compute nodes | |
| 11:57:36 | aspiers | oh sure, that's a totally different kettle of fish | |
| 11:58:27 | aspiers | compute HA is impossible without a resilient control plane | |
| 11:58:49 | aspiers | same for most workflows | |
| 12:01:01 | aspiers | sean-k-mooney: mriedem makes a good point on https://review.opendev.org/#/c/643578/3//COMMIT_MSG that resurrect is an admin-only command, so "openstack server resurrect" could mislead non-admin users into thinking they can call it | |
| 12:01:42 | aspiers | OTOH I don't agree with his statement "evacuate moves the server to another host (actually it rebuilds it on another host)" | |
| 12:02:04 | sean-k-mooney | aspiers: migrate is an admin command too | |
| 12:02:13 | sean-k-mooney | and that is openstack server migrate | |
| 12:02:42 | aspiers | since a) you can't "move" a server which has already died, and b) it only rebuilds if it was using ephemeral storage | |
| 12:02:44 | sean-k-mooney | the only non admin migration is resize | |
| 12:03:01 | aspiers | sean-k-mooney: right, that's why he was suggesting nesting under "openstack server migrate" | |
| 12:03:07 | sean-k-mooney | no | |
| 12:03:17 | sean-k-mooney | migrate implies it will preserve you data | |
| 12:03:20 | aspiers | but I really don't like that cos resurrect is not a form of migration | |
| 12:03:27 | sean-k-mooney | so that would be worse then evac | |
| 12:03:41 | aspiers | yes, I think we are saying the same thing in different ways :) | |
| 12:04:18 | sean-k-mooney | what im saying is migrate is not an option because migrate implies we move data and we dont in the most common form for evac | |
| 12:04:29 | sean-k-mooney | most deployment have an ephermeal root disk | |
| 12:04:36 | aspiers | yes, that's one good reason to avoid migrate | |
| 12:04:50 | aspiers | and it's not the only one | |