| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-17 | |||
| 11:45:52 | cdent | I hope that somehow this leads to that bug getting more attention. people get squeamish about orphaned providers | |
| 11:46:06 | cdent | especially because we have that damn uniq constraint on rp names | |
| 11:46:24 | cdent | which is the source of a ton of support requests | |
| 11:46:47 | cdent | "I was messing around and one of my compute nodes got rejiggered and it says can't create a resource provider" | |
| 11:46:56 | cdent | turns out the old one with the same name still exists | |
| 11:47:54 | aspiers | cdent: sorry to hijack your discussion with a big tangent | |
| 11:48:36 | aspiers | sean-k-mooney, artom: I think this 4-year-old blog post is still very relevant http://blog.clusterlabs.org/blog/2015/living-intersection-pets-cattle | |
| 11:48:44 | cdent | I tend to think the reason tangents happen is because there is insufficient shared understanding and that the tangents are necessary to get it | |
| 11:48:48 | aspiers | yeah | |
| 11:49:07 | cdent | but there is so little shared understanding in nova that is sometimes seems like we are on a tangent all the time | |
| 11:49:18 | aspiers | haha true | |
| 11:49:22 | cdent | tangents on tangets on tangents | |
| 11:49:43 | aspiers | beekhof and I did a lot of work a few years back considering all the failure cases of nova compute HA | |
| 11:49:59 | aspiers | he was definitely the thought-leader in this space, but he escaped to greener pastures | |
| 11:50:18 | sean-k-mooney | isnt a tangent to a tangent called a normal | |
| 11:50:26 | sean-k-mooney | nova embrases this :P | |
| 11:50:48 | aspiers | if a tangent is a 90 degree turn, then 4 tangents get you back in the original direction, right? | |
| 11:51:45 | sean-k-mooney | aspiers: for what its worth a lot of the presentation i was refering to were framed in the context fo service assurance not self healing or instance ha | |
| 11:52:04 | aspiers | sean-k-mooney: got it | |
| 11:52:26 | cdent | at the limit, N tangents is a circle, no matter the angle of turn | |
| 11:52:40 | cdent | or something | |
| 11:52:44 | aspiers | :D | |
| 11:52:53 | aspiers | https://aspiers.github.io/openstack-day-israel-2017-compute-ha/#/nova-host-alerter is the future converged architecture beekhof and I agreed on, in case anyone is interested | |
| 11:53:21 | sean-k-mooney | cdent: are you suggesting we would go around in circles on things, im shocked :) | |
| 11:53:37 | aspiers | coupled with http://blog.clusterlabs.org/blog/2015/reliable-notifications and masakari adding a layer of retries on top of nova's somewhat lossy workflow, the hope was that this would be more robust | |
| 11:53:44 | aspiers | s/more/sufficiently/ | |
| 11:54:01 | sean-k-mooney | anyway i better file my expense report for the ptg or finance will get annoyed at me | |
| 11:54:24 | aspiers | sean-k-mooney: IIRC one of the concerns was that if scheduler or conductor crashed while handling evacuate API, it would get lost forever - is that still true? | |
| 11:54:38 | cdent | sean-k-mooney: hush | |
| 11:54:48 | sean-k-mooney | aspiers: ya maybe | |
| 11:55:03 | sean-k-mooney | it woudl depend on when it failed i guess | |
| 11:55:19 | sean-k-mooney | if we have not deleted it form the souce node yet then you should be fine | |
| 11:55:22 | aspiers | I think the idea was maybe that masakari could reinvoke the API in those circumstances | |
| 11:55:50 | sean-k-mooney | ya i think if the db says its still on the source node then you can evac again | |
| 11:55:57 | aspiers | but is there even a way to query the API for the state of an evacuate workflow? | |
| 11:56:15 | sean-k-mooney | but depending on the failure mode the vm could be paused/running on another node at that point | |
| 11:56:19 | aspiers | I vaguely remember that being discussed at past summits | |
| 11:56:30 | aspiers | yeah, the devil is in the details I guess | |
| 11:57:20 | aspiers | I don't think there's ever been enough corporate sponsorship from RH or SUSE to spend time really cleaning up all those corner cases | |
| 11:57:21 | sean-k-mooney | if your contol plane if failing you shoudl really stop what your doing and fix that first then go back to fixing your broken compute nodes | |
| 11:57:36 | aspiers | oh sure, that's a totally different kettle of fish | |
| 11:58:27 | aspiers | compute HA is impossible without a resilient control plane | |
| 11:58:49 | aspiers | same for most workflows | |
| 12:01:01 | aspiers | sean-k-mooney: mriedem makes a good point on https://review.opendev.org/#/c/643578/3//COMMIT_MSG that resurrect is an admin-only command, so "openstack server resurrect" could mislead non-admin users into thinking they can call it | |
| 12:01:42 | aspiers | OTOH I don't agree with his statement "evacuate moves the server to another host (actually it rebuilds it on another host)" | |
| 12:02:04 | sean-k-mooney | aspiers: migrate is an admin command too | |
| 12:02:13 | sean-k-mooney | and that is openstack server migrate | |
| 12:02:42 | aspiers | since a) you can't "move" a server which has already died, and b) it only rebuilds if it was using ephemeral storage | |
| 12:02:44 | sean-k-mooney | the only non admin migration is resize | |
| 12:03:01 | aspiers | sean-k-mooney: right, that's why he was suggesting nesting under "openstack server migrate" | |
| 12:03:07 | sean-k-mooney | no | |
| 12:03:17 | sean-k-mooney | migrate implies it will preserve you data | |
| 12:03:20 | aspiers | but I really don't like that cos resurrect is not a form of migration | |
| 12:03:27 | sean-k-mooney | so that would be worse then evac | |
| 12:03:41 | aspiers | yes, I think we are saying the same thing in different ways :) | |
| 12:04:18 | sean-k-mooney | what im saying is migrate is not an option because migrate implies we move data and we dont in the most common form for evac | |
| 12:04:29 | sean-k-mooney | most deployment have an ephermeal root disk | |
| 12:04:36 | aspiers | yes, that's one good reason to avoid migrate | |
| 12:04:50 | aspiers | and it's not the only one | |
| 12:05:44 | aspiers | at a higher level of abstraction, what you are saying could be reframed as "migrate" implies moving something from A to B without significantly changing it | |
| 12:05:51 | sean-k-mooney | we also discussed having two commands one that guarnetees it preserves your data and one tha that always destroys it | |
| 12:06:04 | aspiers | but you start with something dead on A, and end with something alive on B, that is a big change in state | |
| 12:06:12 | sean-k-mooney | e.g. recreate woudl always destoy any instance and recreate it loasing data | |
| 12:06:15 | aspiers | therefore does not deserve to be called a form of migration | |
| 12:07:02 | artom | Data destruction/preservation also depends on where said data is | |
| 12:07:10 | sean-k-mooney | openstack server migrate --evacuate would guarentte that your data was moved or would not try to do dit | |
| 12:07:10 | aspiers | sean-k-mooney: yeah that might be a good approach | |
| 12:07:36 | aspiers | I mean, having two different commands | |
| 12:07:48 | aspiers | but I don't like migrate --evacuate, for reasons just explained | |
| 12:08:04 | aspiers | I don't really like migrate --resurrect either | |
| 12:08:35 | aspiers | although it bothers me less than migrate --evacuate | |
| 12:09:40 | aspiers | "openstack server recreate" for the ephemeral case and "openstack server resurrect" for the shared storage case feels OK to me | |
| 12:09:50 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: Allow passing negative values for the locked search_opt in server list https://review.opendev.org/659783 | |
| 12:10:08 | aspiers | but I can see the concern around non-admin users expecting to be able to use it | |
| 12:10:49 | sean-k-mooney | aspiers: well wel could allow openstack server recreate to be non admin | |
| 12:11:40 | sean-k-mooney | its basically a rebuild but optionally to another host | |
| 12:12:20 | aspiers | sean-k-mooney: in what scenarios would non-admin users use that? not if the server is already running right? cos it reuses the old name | |
| 12:13:21 | sean-k-mooney | aspiers: i was implying that openstack server recreate would kill the server if it was running and rescudle it | |
| 12:13:28 | aspiers | oh | |
| 12:13:53 | sean-k-mooney | they would use it if the image or flavor had been updated and they wanted to create the sever with the updated version | |
| 12:14:07 | sean-k-mooney | but keep the same ports and data volumes | |
| 12:14:16 | aspiers | yeah I guess that could be useful | |
| 12:14:45 | aspiers | or if the instance got screwed up somehow, e.g. accidental rm -rf ;-) | |
| 12:15:18 | sean-k-mooney | yep if you instacne are vnfs it would be a neat way to do a software update | |
| 12:15:27 | aspiers | yep | |
| 12:15:27 | sean-k-mooney | kindof like in k8s land | |
| 12:15:45 | sean-k-mooney | e.g. just kill the pod and recreate with the latest version of it | |
| 12:17:54 | aspiers | right | |
| 12:18:21 | aspiers | so are you thinking to update https://review.opendev.org/#/c/643578 to split into two separate commands recreate/resurrect? | |
| 12:18:52 | aspiers | that would work for me, not sure what mriedem will think though | |
| 12:19:12 | aspiers | with both I'd prefer to avoid mentioning the word "migrate" anywhere | |
| 12:19:45 | aspiers | since "migrate" implies avoiding any significant change in state (except the location) | |
| 12:26:28 | aspiers | sean-k-mooney: BTW which timezone are you in? | |
| 12:26:52 | aspiers | I'm guessing somewhere in Ireland? | |
| 12:27:43 | sean-k-mooney | yes | |
| 12:27:57 | sean-k-mooney | im shannon on the west coast of ireland | |
| 12:27:59 | aspiers | nice | |
| 12:28:16 | sean-k-mooney | what timezone i am awake in is not alway the same as where im located | |
| 12:28:22 | aspiers | true for me too ;) | |
| 12:28:35 | aspiers | for some reason I still expect all nova devs to be in north america | |