Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-14
15:21:00 mriedem sure, we have --force on delete_cell already
15:21:04 dansmith and delete_cell has --force already which means something else though
15:21:15 dansmith it means "delete all the things, yeah", but this would be "and also, don't delete some stuff"
15:21:21 mriedem ha
15:21:24 dansmith right?
15:21:40 mriedem it's just confusing ux
15:21:56 dansmith but really, they're two meanings of force
15:22:03 dansmith what we have now is really --recursive
15:23:20 stephenfin dansmith: This NUMA aware vSwitch spec is basically turning into a book. Be prepared!
15:23:41 dansmith regardless, we need that thing and some appropriate flaggage to control intent
15:23:48 dansmith stephenfin: noted
15:24:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_get_by_name db api method https://review.openstack.org/544379
15:24:25 dansmith belmoreira: tssurya: can one of you open a bug for this so I can point a patch at it?
15:24:48 tssurya dansmith : sure
15:24:59 belmoreira dansmith yes we can do it, we can also work on it if you prefer
15:25:06 dansmith or that ;)
15:25:26 mriedem dansmith: in the scenario you said, "someone with placement down during an upgrade pruning a host wants to know that they're about to orphan some placement records" - if i'm removing a compute host (node really here), are you just saying that you want to *not* delete the compute node if placement is down, right?
15:25:44 dansmith mriedem: right
15:25:54 mriedem default behavior would be, we try to remove from placement and if that fails, we fail, unless instructed to ignore it
15:25:56 dansmith mriedem: unless they say "I know what I'm doing" or something
15:26:01 dansmith mriedem: yes
15:26:18 mriedem that still seems like --force should cover that case to me
15:26:34 dansmith but they may want --force to mean "delete all the hosts too",
15:26:46 dansmith but then they would be opting into "and orphan all the data in placement if it's down"
15:27:21 dansmith so let's say you're deleting a cell while you're down for an upgrade, and that cell has 10 hosts
15:27:22 mriedem well, --force is a destructive option
15:27:51 dansmith it is but it's not a "and corrupt my data"
15:27:52 dansmith so you're down and deleting a cell with ten hosts
15:27:54 dansmith you run delete_cell --force, to delete all ten hosts and the cell record
15:27:57 dansmith if placement is down,
15:28:04 dansmith you now just orphaned the placement records for those ten hosts,
15:28:22 dansmith so when you come back up, ten providers in placement are no longer present, but will be returned by placement forever,
15:28:29 dansmith and unless you know what the uuids were of those records you deleted,
15:28:33 dansmith you can't really clean that up
15:28:52 openstackgerrit Merged openstack/nova-specs master: Update Queens NRP spec to reflect reality https://review.openstack.org/544086
15:29:10 dansmith our current --force is poorly named.. it should be --recursive
15:29:23 dansmith because --recursive means "and delete everything you have to to delete the cell"
15:29:51 dansmith cdent: do you not get what I'm saying here?
15:30:11 cdent No, I totally get it. I'm not kidding. It is an astute observation.
15:30:26 dansmith okay
15:30:33 mriedem i get the scenario, i just, idk
15:30:50 cdent one way around the problem is "don't have placement down"
15:30:59 cdent which is probably a good rule of thumb anyway
15:31:05 dansmith mriedem: in belmoreira's case, he would run --force to delete all the hosts, and then he would be in exactly the same position that started this conersation
15:31:24 dansmith cdent: aye, it'd be better to not allow skipping the placement cleanup at all
15:31:33 mriedem dansmith: did belmoreira say placement was down?
15:31:49 dansmith mriedem: no, but he'd want to use delete_cell --force to delete the cell and all the hosts,
15:32:14 dansmith but if placement was down when doing that (because maintenance), then we'd orphan all the records that he's currently saying we orphan, which is a problem
15:32:35 dansmith i.e. you can't opt into the recursiveness without choosing the "and also corrupt the data if you need to"
15:33:20 mriedem so we could just say, you can't do that if placement is down, but sure i agree people will want a backdoor
15:33:22 mriedem to bypass that
15:33:51 dansmith right, which is why I'm saying we need another flag
15:33:52 dansmith since we already burned --force to mean --recursive, we need something else
15:34:09 mriedem --ignore-placement?
15:34:19 mriedem --fudge-it?
15:34:32 dansmith yeah, I said --no-placement above, but exactly.. something like that
15:35:41 mriedem apparently --force doesn't also remove the instances/mappings
15:35:49 tssurya mriedem : I have a patch for that
15:36:00 tssurya in gerrit
15:36:17 dansmith yeah and I think I'm +2 on it already right?
15:36:18 mriedem force deleting instances from nova-manage is pretty scary
15:36:37 mriedem you're likely orphaning ports, volumes, etc
15:36:38 tssurya mriedem : no --force does not delete instances
15:36:47 mriedem just the mappings?
15:36:53 tssurya mriedem : it only removes/cleans mappings for deleted instances
15:36:54 tssurya yes
15:36:57 tssurya just mappings
15:36:58 dansmith ah, nm was thinking of the other one
15:37:08 tssurya we under no circumstance delete a cell if there are instances
15:37:10 mriedem i'm saying delete_cell --force will fail today if there are instance mappings in the cell
15:37:50 mriedem ok this patch https://review.openstack.org/#/c/540073/
15:38:08 tssurya mriedem : yes , my after merging my patch, delete_cell will fail if there any living instances
15:39:25 mriedem ok i see
15:39:47 jaypipes stephenfin: https://review.openstack.org/#/c/538217/3/specs/rocky/approved/integrate-mypy-type-checking.rst <-- well, sheeet, let's just rewrite Nova in Golang (or Rust). :P
15:40:49 stephenfin jaypipes: If you can find me a tool to autoconvert one language to another, I'm all for it ;)
15:40:59 mriedem vish suggested java after joining oracle
15:41:04 mriedem i know java, not go or rust
15:41:27 jaypipes stephenfin: :)
15:41:34 belmoreira a concern that I also have with placement is that it doesn't know anything about cells, that is now an official think in nova
15:41:39 mnaser i've been looking through the nova codebase throughout this morning -- we're running into an issue where nova fails to boot an instance "Invalid volume: Volume status must be available to reserve, but the status is attaching. (HTTP 400)"
15:41:46 jaypipes I'd be up for a rewrite in C++.
15:41:49 mriedem belmoreira: placement shouldn't know about cells
15:41:52 belmoreira we map projects to cells. Meaning that to handle a "small" VM the placement will return "all" nodes and then is the scheduler filtering to find the right cell
15:41:57 stephenfin jaypipes: I do suggest checking out the sample patches I've thrown up through. An afternoons work and it really is minimally invasive
15:42:04 jaypipes stephenfin: will do :)
15:42:04 belmoreira is there a way to get a more filtered answer from placement?
15:42:09 stephenfin jaypipes: and whatever you do, _don't_ listen to mdbooth! :D
15:42:09 mnaser it seems like *somehow*, the cinder volume is in attaching state, and nova tries to reserve it, and so it fails after
15:42:15 jaypipes heh
15:42:22 mriedem mnaser: this is pike 16.0.4?
15:42:25 dansmith belmoreira: aggregates
15:42:33 dansmith belmoreira: that's how you get a more filtered answer
15:42:35 mdbooth stephenfin: Generally good advice.
15:42:47 belmoreira dansmith placement aggregates?
15:42:50 stephenfin mdbooth: https://review.openstack.org/#/c/538217/3/specs/rocky/approved/integrate-mypy-type-checking.rst
15:42:51 dansmith belmoreira: aye
15:43:00 mdbooth stephenfin: Haha
15:43:04 mnaser mriedem: i believe the control plane is stable/pike, i think computes might be a bit older i'm not sure if they're 16.0.4
15:43:06 jaypipes dansmith: aggregate filtering isn't supported yet in allocation candidates...
15:43:14 jaypipes dansmith: or at least, I don't think it is..

Earlier   Later