Posted	Nick	Remark
#openstack-nova - 2018-02-14
14:59:34	dansmith	mriedem: ack
14:59:55	dansmith	I'm around for it, but I don't really have anything specific other than my reviews
15:07:29	belmoreira	have some placement related questions. dansmith, mriedem do you have some time?
15:07:49	dansmith	belmoreira: I can try
15:08:03	belmoreira	thanks dansmith
15:08:14	belmoreira	we are running local placements in ocata. Trying to consolidate them and get some performance numbers.
15:08:39	belmoreira	But, when nodes are retired (removed from the cloud) they continue to exist in placement. Are they eventually removed? not seeing how
15:08:49	dansmith	no they're not
15:09:29	belmoreira	and they are included in the candidates?
15:09:44	mriedem	hmm, they probably are,
15:09:51	dansmith	they would be from placement yeah, but scheduler would filter them out
15:09:56	mriedem	similarly, we don't have a direct hook to remove entries from the compute_nodes table
15:10:08	belmoreira	we are seeing that placement continue to include them
15:10:10	dansmith	this isn't really any different from our compute nodes befoe placement
15:10:31	mriedem	belmoreira: when you retire a node, do you manually remove it's entry from the nova.compute_nodes table?
15:10:54	belmoreira	the difference is that the scheduling is now global
15:11:29	belmoreira	removing few hundred nodes could impact the performance of the entire cloud
15:11:45	mriedem	what did you do when you removed them before this?
15:11:57	belmoreira	mriedem currently we remove entire cells, so we don't have this problem
15:12:12	mriedem	ok so you drop the entire nova db
15:12:14	mriedem	for that cell
15:12:19	belmoreira	correct
15:12:32	dansmith	hmm, and compute nodes don't get mirrored up?
15:12:42	dansmith	I can't remember how that works, but maybe they don't
15:12:54	mriedem	we've talked before about adding a nova-manage CLI to remove a compute_nodes entry
15:12:59	belmoreira	dansmith what do you mean?
15:13:11	mriedem	the host_mappings table would be pointing at hosts that no longer exist too
15:13:13	dansmith	belmoreira: well, in cellsv1 a bunch of stuff gets mirrored up to the top db
15:13:28	dansmith	belmoreira: I would have expected compute_nodes would too, but I don't really know I guess
15:13:34	dansmith	it doesn't matter though
15:13:39	belmoreira	dansmith compute nodes are only in the child cells
15:13:46	dansmith	going forward, pruning will have to happen when you remove stuff
15:14:01	dansmith	ack
15:14:14	belmoreira	is there any work already done in that?
15:14:16	dansmith	so one way to do that, I think,
15:14:29	mriedem	we likely could use a nova-manage CLI that removes the compute_nodes table entry in the cell, the host_mappings entry in the api db, and the resource_providers entry in placement
15:14:37	dansmith	would be to go through host_mappings, find any mappings that don't have a cell, remove them and tell placement about it
15:14:53	dansmith	mriedem: yeah, but if you remove a cell first, you'll need a way to clean up the mess too
15:15:07	tssurya	mriedem : yes that would be a useful command
15:15:18	mriedem	can we delete a cell mapping that has host mappings?
15:15:24	mriedem	don't we have a fkey?
15:15:33	dansmith	idk
15:15:50	mriedem	nullable=False)
15:15:50	mriedem	cell_id = Column(Integer, ForeignKey('cell_mappings.id'),
15:16:06	mriedem	and nullable=False
15:16:14	mriedem	yeah, so we have to remove host mappings if we remove a cell mapping
15:16:17	dansmith	but,
15:16:29	dansmith	you can delete a cell with delete_cell and it will nuke all the host mappings
15:16:33	dansmith	so then you're really toast
15:16:41	mriedem	if you use --force yeah
15:17:27	dansmith	so we have delete_host
15:17:58	dansmith	which unmaps the compute node, and deletes the hostmapping
15:18:06	dansmith	so in both of those I guess we need to update placement
15:18:23	mriedem	i'm not sure why delete_host requires a --cell_uuid arg,
15:18:26	mriedem	but that's unrelated
15:18:37	dansmith	although nova-manage doesn't do any rpc or calls to other services right now, AFAIK, so we kinda need a --no-placement flag to each of those in case you're operating in an island
15:19:13	mriedem	well,
15:19:29	mriedem	nova-manage could use the scheduler report client and handle an http error response if it can't talk to placement
15:19:43	dansmith	yes, but you dont' want to proceed then,
15:19:48	dansmith	unless you tell it to
15:20:00	dansmith	because if we can't talk to placement, we can't proceed without corrupting the linkage (as we are now)
15:20:07	dansmith	so you need to fail in that case, unless instructed to ignore
15:20:36	dansmith	someone with placement down during an upgrade pruning a host wants to know that they're about to orphan some placement records
15:20:38	mriedem	could just throw another --force on delete_host
15:20:48	dansmith	we need it for both though
15:20:54	dansmith	delete_cell and delete_host
15:21:00	mriedem	sure, we have --force on delete_cell already
15:21:04	dansmith	and delete_cell has --force already which means something else though
15:21:15	dansmith	it means "delete all the things, yeah", but this would be "and also, don't delete some stuff"
15:21:21	mriedem	ha
15:21:24	dansmith	right?
15:21:40	mriedem	it's just confusing ux
15:21:56	dansmith	but really, they're two meanings of force
15:22:03	dansmith	what we have now is really --recursive
15:23:20	stephenfin	dansmith: This NUMA aware vSwitch spec is basically turning into a book. Be prepared!
15:23:41	dansmith	regardless, we need that thing and some appropriate flaggage to control intent
15:23:48	dansmith	stephenfin: noted
15:24:00	openstackgerrit	Matt Riedemann proposed openstack/nova master: Remove old flavor_get_by_name db api method https://review.openstack.org/544379
15:24:25	dansmith	belmoreira: tssurya: can one of you open a bug for this so I can point a patch at it?
15:24:48	tssurya	dansmith : sure
15:24:59	belmoreira	dansmith yes we can do it, we can also work on it if you prefer
15:25:06	dansmith	or that ;)
15:25:26	mriedem	dansmith: in the scenario you said, "someone with placement down during an upgrade pruning a host wants to know that they're about to orphan some placement records" - if i'm removing a compute host (node really here), are you just saying that you want to *not* delete the compute node if placement is down, right?
15:25:44	dansmith	mriedem: right
15:25:54	mriedem	default behavior would be, we try to remove from placement and if that fails, we fail, unless instructed to ignore it
15:25:56	dansmith	mriedem: unless they say "I know what I'm doing" or something
15:26:01	dansmith	mriedem: yes
15:26:18	mriedem	that still seems like --force should cover that case to me
15:26:34	dansmith	but they may want --force to mean "delete all the hosts too",
15:26:46	dansmith	but then they would be opting into "and orphan all the data in placement if it's down"
15:27:21	dansmith	so let's say you're deleting a cell while you're down for an upgrade, and that cell has 10 hosts
15:27:22	mriedem	well, --force is a destructive option
15:27:51	dansmith	it is but it's not a "and corrupt my data"
15:27:52	dansmith	so you're down and deleting a cell with ten hosts
15:27:54	dansmith	you run delete_cell --force, to delete all ten hosts and the cell record
15:27:57	dansmith	if placement is down,
15:28:04	dansmith	you now just orphaned the placement records for those ten hosts,
15:28:22	dansmith	so when you come back up, ten providers in placement are no longer present, but will be returned by placement forever,
15:28:29	dansmith	and unless you know what the uuids were of those records you deleted,
15:28:33	dansmith	you can't really clean that up

Earlier   Later