Posted	Nick	Remark
#openstack-nova - 2018-06-18
22:30:54	jaypipes	jroll: k, no prob. was just curious if we should make a specific change.
22:32:49	jaypipes	efried: I don't see *end users* ever doing `openstack acc create --type GPU <specs>`. Maybe some admin... maybe. though that said, I don't really know what <specs> would be.
22:33:11	jaypipes	efried: end users (at least that I'm aware of) don't have the ability to create "accelerators".
22:33:12	efried	jaypipes: same thing as whatever it would be in the flavor, more or less.
22:33:31	jaypipes	efried: I guess I don't see why we need a separate service for that, but ok.
22:33:33	efried	Okay, if this is all driven by the flavor, so be it.
22:33:58	efried	jaypipes: If we don't, we don't. I've never been a huge fan of the `port create` pre-step.
22:34:34	efried	I guess the difference is that every port needs to have specific *unique* config info (IP addr).
22:34:46	efried	...which doesn't make sense to be in a flavor.
22:36:25	jaypipes	efried: we're talking about different things entirely.
22:37:42	jaypipes	efried: I'll just leave this conversation like this... I don't see why Cyborg isn't the generic device manager...
22:37:56	efried	I fully hope it will be
22:38:03	jaypipes	in other words, I don't see the point of having `openstack acc $ANYTHING`.
22:38:20	efried	I'm on board with that, if dansmith is.
22:38:46	jaypipes	there should be a YAML inventory file describing devices (that can't for some reason be auto-discovered on a node) and that's that...
22:39:23	efried	The way the conversation with Sundar started was that his specs talk about os-acc parallelling os-vif, but then doesn't actually do that.
22:39:50	jaypipes	note that I didn't say that os-acc shouldn't exist.
22:40:40	efried	I wanted os-acc to have Acc* objects (corresponding to VIF*) and Acc*Profile (=~ VIF*PortProfile); and to define plug/unplug methods which call to plugins registered/loaded stevedore-wise.
22:40:49	efried	they didn't have any of that in their specs.
22:41:48	jaypipes	efried: but I don't see the point of a Cyborg REST API. sorry if that offends anyone. I just really only see the point of a utility library (os-acc) that can be called to configure/init a particular device slot.
22:42:20	efried	...where os-acc has a plugin mechanism that in turn lets it call out to platform-specific code to do plugging etc.
22:42:31	jaypipes	sure, that's fine with me.
22:42:55	efried	I imagine the REST API may be useful for things like programming FPGAs or twiddling details in their database.
22:43:14	efried	but Nova isn't going to touch it, as you say, because all Nova needs to interact with is os-acc.
22:43:46	efried	now it's possible that os-acc will talk to the cyborg API.
22:43:48	efried	isn't it?
22:43:50	jaypipes	efried: nobody is going to be "prgramming FPGAs" via a REST API. :)
22:44:25	jaypipes	efried: sure, I guess I'm fine with this sort of thing being in the Cyborg API: https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/deployables.py
22:44:46	jaypipes	efried: what I *don't* think should be in the Cyborg API is this: https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/accelerators.py
22:44:55	jaypipes	efried: because all that is is inventory and resource providers information.
22:45:05	efried	jaypipes: Just meaning in the same sense that we have a REST API to create a server by identifying the UUID of an image to load up; cyborg would have an API to identify the UUID of a bitstream image to program onto the thing.
22:45:06	jaypipes	efried: and that belongs in placement, IMHO.
22:45:08	efried	is that off base?
22:45:37	jaypipes	efried: no. https://github.com/openstack/cyborg/blob/master/cyborg/api/controllers/v1/accelerators.py <-- that is the equivalent of compute_nodes in Nova, not instances table.
22:46:28	efried	I was still talking about "programming FPGAs via a REST API".
22:46:34	jaypipes	oh.
22:47:05	jaypipes	efried: sure, that's fine. that's the /deployables REST API endpoint in Cyborg.
22:52:28	efried	jaypipes: The other thing dansmith talked about was this need for a "bind" step. This is after allocate but before plug, and (I guess) lets cyborg know the actual RP we're grabbing. I think the purpose is so that cyborg can do their on-the-fly inventory/trait adjustment or perhaps final checking stuff? Not sure why this couldn't happen at the front of plug() before the plugin is invoked.
22:53:35	openstackgerrit	Matt Riedemann proposed openstack/nova master: Wait for network-vif-plugged before starting live migration https://review.openstack.org/558001
22:53:36	dansmith	efried: the point of bind would be to get the inventory adjustment step and start programming the thing as early as we know where it's going to be
22:53:36	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add check if neutron "binding-extended" extension is available https://review.openstack.org/523548
22:53:37	mriedem	watch out
22:53:37	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add "bind_ports_to_host" neutron API method https://review.openstack.org/523604
22:53:38	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add "delete_port_binding" network API method https://review.openstack.org/552170
22:53:39	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add "activate_port_binding" neutron API method https://review.openstack.org/555947
22:53:40	openstackgerrit	Matt Riedemann proposed openstack/nova master: Delete port bindings in setup_networks_on_host if teardown=True https://review.openstack.org/556333
22:53:41	openstackgerrit	Matt Riedemann proposed openstack/nova master: Implement migrate_instance_start method for neutron https://review.openstack.org/556334
22:53:42	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add VIFMigrateData object for live migration https://review.openstack.org/515423
22:53:43	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add VIFMigrateData.get_dest_vif https://review.openstack.org/566931
22:53:44	openstackgerrit	Matt Riedemann proposed openstack/nova master: libvirt: factor out pre_live_migration plug_vifs call https://review.openstack.org/566932
22:53:45	openstackgerrit	Matt Riedemann proposed openstack/nova master: libvirt: use dest host port bindings during pre_live_migration https://review.openstack.org/566933
22:53:46	openstackgerrit	Matt Riedemann proposed openstack/nova master: libvirt: use dest host vif migrate details for live migration https://review.openstack.org/551370
22:53:48	openstackgerrit	Matt Riedemann proposed openstack/nova master: Annotate flows and handle PortBindingDeletionFailed in ComputeManager https://review.openstack.org/551371
22:53:49	openstackgerrit	Matt Riedemann proposed openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870
22:53:50	openstackgerrit	Matt Riedemann proposed openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537
22:53:58	mriedem	gibi: thanks for the reviews ^
22:54:10	efried	dansmith: I don't think we have the actual "VF" at that point, though.
22:54:26	efried	dansmith: Any case, is there a reason that couldn't be done at the front of plug() before the plugin's plug() is invoked?
22:54:36	dansmith	efried: we have the host, and cyborg could decide which "VF" it's going to give away
22:54:47	efried	cyborg doesn't decide that - the plugin does.
22:55:10	dansmith	efried: no, but it just serializes things that are parallelizeable, and makes us wait later to actually let cyborg adjust its inventory if it needs to
22:55:25	dansmith	which reduces the window where we might've promised two things that can't actually co-exist
22:56:03	efried	dansmith: And, since you're here now, do you agree that there's no `openstack accelerator create` thing to get an attachment handle - because the accelerator details are going to be in the flavor?
22:56:13	dansmith	I really wish I wasn't here, I'm supposed to be EOD
22:56:25	efried	Then just agree, and you can go free.
22:57:09	dansmith	I dunno, I thought that if we don't allow them to create via cyborg (like creating a port via neutron) we're not going to be able to represent all the extra shit they want, and/or have to pass it all through our API
22:58:03	efried	If by "pass it all through" you mean it piggybacks on the flavor and the allocation record in the RequestSpec, then... yeah.
22:58:10	dansmith	no,
22:58:24	dansmith	I mean if we don't let them do anything other than set stuff in the flavor,
22:58:59	dansmith	but that doesn't provide enough detail to say things like "I want SSLv3 offload accelerator vs. I want generic crypto accelerator" or something
22:59:15	dansmith	if it's only ever going to be "I want N of $foo" then flavor is enough,
22:59:18	dansmith	but it's never that
22:59:31	efried	heh
22:59:35	dansmith	like neutron, it was just network,
22:59:48	dansmith	but then it became mac, and qos, and subnet, and ip and....
22:59:53	efried	well, I guess we can do it one way and then add the other way later. That's never a problem.
23:03:17	eandersson	Is there a way to do live migration one-by-one in Mitaka? e.g. using some external script or similar?
23:04:05	eandersson	with NUMA support
23:04:25	efried	eandersson: I imagine you could do some kind of
23:04:25	efried	for uuid in `nova list --params`; do nova migrate $uuid blah; done
23:04:25	efried	But I suspect that's not what you're asking...
23:04:47	eandersson	Ah yea, sorry left out an important detail about NUMA :p
23:05:44	eandersson	Trying to figure out if there is a one-off script we could use to manually fix the NUMA calculation after live migrating VMs
23:06:48	efried	eandersson: Ask stephenfin in about 12h
23:07:08	efried	sean-k-mooney[m] may also be able to help, but he's also UTC.
23:07:20	efried	cfriesen possibly?
23:08:27	eandersson	Thanks efried
23:08:30	cfriesen	eandersson: one issue is that you may not actually succeed in live migrating
23:09:12	cfriesen	eandersson: since in the general case you need to recalculate any dedicated CPUs, guest-to-host NUMA mapping, etc
23:11:47	cfriesen	If you could arrange things such that you have an empty compute node that is identical to the one you want to migrate off of, that would simplify things since they could all keep the same resources.
23:12:50	cfriesen	eandersson: ^ (I suppose it would also work if dest compute node was larger than the source.)
23:20:02	eandersson	cfriesen, yea we were thinking of disabling a compute node, and then live migrating to it
23:20:19	eandersson	to avoid raceconditions etc
23:23:08	eandersson	most of these vms are probably gonna take up a single numa zone, with two zones per host
23:30:09	openstackgerrit	Eric Fried proposed openstack/os-vif master: doc: Fix arg specs and object types in docs https://review.openstack.org/575151
23:32:52	openstackgerrit	Matt Riedemann proposed openstack/nova stable/pike: Fixed auto-convergence option name in doc https://review.openstack.org/576282
23:34:52	cfriesen	eandersson: if it's disabled then nova won't let you live migrate to it
23:36:08	cfriesen	eandersson: but as long as you can guarantee that the resources being used on the source host are available on the dest, then it should be basically okay. Once the resource audit has run on the dest (within a minute by default) then you should be basically sane.
23:46:29	mriedem	for all you ceph nerds, this doesn't make a ton of sense https://bugs.launchpad.net/nova/+bug/1777123
23:46:30	openstack	Launchpad bug 1777123 in OpenStack Compute (nova) "Nova 17.04 fails to create ephemeral storage with rbd driver " [Undecided,New]

Earlier   Later