Posted	Nick	Remark
#openstack-nova - 2019-05-17
11:27:12	aspiers	artom: haha, maybe a bit ;-)
11:27:26	aspiers	I've been working on compute HA in nova for about 4-5 years
11:27:29	sean-k-mooney	cdent: or it could be that we delete the old allocation on the source node and that does not run in the evacuate case
11:27:50	aspiers	artom: but don't feel bad, there's no good reason why you should have known that ;)
11:28:01	cdent	that's what the bug report is saying, yes
11:28:03	sean-k-mooney	aspiers: that was a poor life choice :P
11:28:15	aspiers	sean-k-mooney: LOL, yeah probably ;-)
11:28:20	artom	I see to have a lot of feet in my mouth, maybe wash them out with aforementioned coffee
11:28:25	artom	*seem
11:28:35	aspiers	sean-k-mooney: I mean, not *exclusively* compute HA ... I've done a lot of other stuff too ;)
11:29:17	sean-k-mooney	aspiers: so for https://review.opendev.org/#/c/643578/3 back to the ptg session your prefere is openstack server resuerect
11:29:28	aspiers	sean-k-mooney: yes exactly
11:30:26	aspiers	"evacuate" and "migrate" both imply that everything is currently running (mostly) fine
11:30:43	artom	Well
11:30:48	aspiers	"resurrect" implies (correctly) "oh shit, something already blew up, so let's clean up after it"
11:30:55	artom	The compute service/control plane has to be down
11:31:07	artom	But the data plane? IIUC the instance can be running, no?
11:31:12	sean-k-mooney	aspiers: ya that why i called the new command recreate but i can update it to resurect
11:31:33	sean-k-mooney	i spell recreate more consitenly however but i can live with that
11:31:36	aspiers	artom: which scenario are you talking about exactly? there are a few here
11:31:45	artom	aspiers, evacuate
11:31:45	sean-k-mooney	artom: we dont
11:32:04	sean-k-mooney	artom: but we require that you have marked the compute node as disabled before we allow it
11:32:12	aspiers	artom: in this context you're gonna have to use more than one word to describe the scenario sorry ;-)
11:32:21	aspiers	the whole problem is the ambiguity
11:32:23	sean-k-mooney	artom: and we require the operator to confim that it is dead before the evac
11:32:32	aspiers	sean-k-mooney: actually not always
11:32:44	sean-k-mooney	aspiers: that is the api contract
11:32:45	aspiers	sean-k-mooney: OOB fencing can *ensure* that the VM is dead
11:32:53	sean-k-mooney	sure
11:33:04	sean-k-mooney	but the api contract is only call this if it is dead
11:33:06	aspiers	sean-k-mooney: i.e. it can be automated. but yes we're on the same page I think
11:33:10	aspiers	exactly
11:33:58	aspiers	artom: there are multiple failure scenarios here to consider
11:34:03	sean-k-mooney	if you violate that contract and you get 2 instance trying to read/write to the same non multi atach volume some how we just say "have fun" and walk away smiling
11:34:38	aspiers	sean-k-mooney: exactly, that is the main message of 3 talks I have given on this
11:34:38	openstackgerrit	Miguel Ángel Herranz Trillo proposed openstack/nova master: Fix type error on call to mount device https://review.opendev.org/659780
11:35:02	aspiers	in Austin, Boston, and OpenStack Day Israel
11:35:31	aspiers	e.g. https://youtu.be/lddtWUP_IKQ?t=366
11:35:34	artom	aspiers, evacuate the nova client command
11:35:43	aspiers	(because exploding kittens is fun)
11:36:04	artom	And my thought was with respect to resurrect, that the *instance* might not be dead, even if it's unmanageable because the host it's on *is* dead
11:36:13	sean-k-mooney	ill convert https://review.opendev.org/#/c/643578/3 to resurect and reupload it proably on monday
11:36:19	artom	So "resurrect" might not be the best word
11:36:34	aspiers	sean-k-mooney: ++
11:36:49	sean-k-mooney	artom: well its like if your heart stoped
11:36:49	aspiers	artom: as sean-k-mooney said, the API contract is that the instance *must* be dead before calling the API
11:37:05	artom	aspiers, ah, didn't know that, ok then
11:37:07	sean-k-mooney	you were technically dead for a bit and this operation is like a defibulator
11:37:45	aspiers	artom: that's why the force_down API was introduced ages ago - to get nova to realise quicker that it's dead, rather than waiting for its own RPC timeout
11:38:09	sean-k-mooney	actully not just for that usecase
11:38:30	aspiers	OK, that was one use case at least
11:38:49	sean-k-mooney	there were case where for example the root disk and the disk where the vms images were seperate
11:39:19	sean-k-mooney	so the agent and libvirt could be fine but when the backing disk died for the image and they crashed you wanted to be able to mark the host as down
11:39:28	sean-k-mooney	or similar for network issues
11:39:53	aspiers	right
11:40:11	sean-k-mooney	aspiers: were you working with emma foley on the collectd/doctor/vitriage demoes?
11:40:23	aspiers	artom: https://youtu.be/lddtWUP_IKQ?t=648 covers some of the failure modes, but you and sean-k-mooney are both right that there are other failure modes based on different network failures
11:41:02	aspiers	sean-k-mooney: nope not heard of that, but with my self-healing SIG hat on, I'd really like to learn more about it!
11:41:06	aspiers	sean-k-mooney: got any URLs?
11:41:31	sean-k-mooney	there have been at least 3 summit presentation on it
11:41:47	sean-k-mooney	the effort was driven via opnfv
11:41:54	aspiers	damn, thought I'd been to all the vitrage presentations
11:43:36	aspiers	so I'm just looking through my old decks and found this https://aspiers.github.io/openstack-day-israel-2017-compute-ha/#/comparison
11:43:45	aspiers	had forgotten about the "needs improvements in nova" bit
11:43:55	aspiers	trying to remember the details of that
11:44:11	aspiers	I think it was around the reliability of the evacuate API
11:44:49	cdent	that's some can of worms I opened
11:45:04	aspiers	bad cdent!
11:45:11	aspiers	keep quiet in future ;-)
11:45:18	cdent	unpossible!
11:45:26	aspiers	j/k, glad to be reminded about this stuff
11:45:52	cdent	I hope that somehow this leads to that bug getting more attention. people get squeamish about orphaned providers
11:46:06	cdent	especially because we have that damn uniq constraint on rp names
11:46:24	cdent	which is the source of a ton of support requests
11:46:47	cdent	"I was messing around and one of my compute nodes got rejiggered and it says can't create a resource provider"
11:46:56	cdent	turns out the old one with the same name still exists
11:47:54	aspiers	cdent: sorry to hijack your discussion with a big tangent
11:48:36	aspiers	sean-k-mooney, artom: I think this 4-year-old blog post is still very relevant http://blog.clusterlabs.org/blog/2015/living-intersection-pets-cattle
11:48:44	cdent	I tend to think the reason tangents happen is because there is insufficient shared understanding and that the tangents are necessary to get it
11:48:48	aspiers	yeah
11:49:07	cdent	but there is so little shared understanding in nova that is sometimes seems like we are on a tangent all the time
11:49:18	aspiers	haha true
11:49:22	cdent	tangents on tangets on tangents
11:49:43	aspiers	beekhof and I did a lot of work a few years back considering all the failure cases of nova compute HA
11:49:59	aspiers	he was definitely the thought-leader in this space, but he escaped to greener pastures
11:50:18	sean-k-mooney	isnt a tangent to a tangent called a normal
11:50:26	sean-k-mooney	nova embrases this :P
11:50:48	aspiers	if a tangent is a 90 degree turn, then 4 tangents get you back in the original direction, right?
11:51:45	sean-k-mooney	aspiers: for what its worth a lot of the presentation i was refering to were framed in the context fo service assurance not self healing or instance ha
11:52:04	aspiers	sean-k-mooney: got it
11:52:26	cdent	at the limit, N tangents is a circle, no matter the angle of turn
11:52:40	cdent	or something
11:52:44	aspiers	:D
11:52:53	aspiers	https://aspiers.github.io/openstack-day-israel-2017-compute-ha/#/nova-host-alerter is the future converged architecture beekhof and I agreed on, in case anyone is interested
11:53:21	sean-k-mooney	cdent: are you suggesting we would go around in circles on things, im shocked :)
11:53:37	aspiers	coupled with http://blog.clusterlabs.org/blog/2015/reliable-notifications and masakari adding a layer of retries on top of nova's somewhat lossy workflow, the hope was that this would be more robust
11:53:44	aspiers	s/more/sufficiently/
11:54:01	sean-k-mooney	anyway i better file my expense report for the ptg or finance will get annoyed at me
11:54:24	aspiers	sean-k-mooney: IIRC one of the concerns was that if scheduler or conductor crashed while handling evacuate API, it would get lost forever - is that still true?

Earlier   Later