Posted	Nick	Remark
#openstack-nova - 2019-12-11
22:13:17	dansmith	so cyborg can look for that,
22:13:21	efried	those are simply flags on that virt driver dict
22:13:25	dansmith	and we throw that into the scheduling request also if we're doing accel stuff?
22:13:49	efried	Well, I don't think we need it in the sched request, if cyborg is using it to decide whether to present the inventory in the first place.
22:14:09	efried	but if you think it should be there for form's sake, I wouldn't object.
22:14:11	dansmith	again, that puts the onus on cyborg to be responsible, which I don't like
22:14:19	efried	where's the trust??
22:14:29	Sundar	So, the use case for all this is that, the operator added accelerator to a compute node but did not upgrade n-cpu on that node?
22:14:31	dansmith	right, thank you :)
22:14:35	dansmith	efried: let me check my butt, hang on
22:14:57	efried	That's not spelled "onus" dansmith
22:15:06	dansmith	hehe
22:15:08	efried	Sundar: yes, exactly.
22:15:37	efried	Sundar: we're trying to make nova account for poorly-behaved a) operators, b) service code from other services.
22:15:57	efried	and not-work in a predictable rather than unpredictable (and potentially unrecoverable) way.
22:16:11	dansmith	just to be clear,
22:16:33	dansmith	I definitely think that the rpc method change is the right thing to do, instead of just coding in new assumptions on both sides
22:16:57	dansmith	I understand the hesitation to that, so I won't hold strictly to it, despite it being the safer option, IMHO
22:17:28	efried	dansmith: if we did that, we would have to have a post-placement filter to remove hosts based on RPC version, right?
22:17:35	efried	perhaps we already have that.
22:17:47	Sundar	What's wrong with the option where Nova places a CAN_DO_CYBORG trait on some compute node RPs, and factors that into Placement query?
22:18:14	efried	Sundar: nothing's wrong with that, and we should do it, but we're talking about doing another thing as well.
22:18:22	dansmith	efried: no, you'd still want to pick a capable host during scheduling, and scheduling based on rpc version is not a thing
22:18:39	efried	dansmith: so what's the RPC version for again? Just passing the arq uuids?
22:19:37	dansmith	efried: yes, and to crystalize our request to the compute node so that (a) the RPC layer can tell if we're (for whatever reason) asking the compute node to do something it can't handle, and so that compute doesn't bake the assumption of the instance having a magical key in its flavor into meaning that it should call to cyborg to do all this stuff
22:20:13	dansmith	efried: because sometime in the future, the mere presence of that flavor key will imply a slightly different thing, and we will have an upgrade concern to deal with because we have a bunch of old computes that will still be doing the assume-it thing
22:20:41	dansmith	I made reference to this in a previous discussion about all of this
22:22:16	efried	This is getting into details I probably don't care about rn, but do we get (a) for free (does the call bounce if the compute is too old) or do we have to do an explicit check?
22:22:37	dansmith	summarized the bits you do care about here: https://review.opendev.org/#/c/631244/51/nova/conductor/manager.py
22:23:30	efried	ack
22:23:55	dansmith	efried: the rpc layer knows what the minimum supported version is across the cluster, so when it is converting the request, it can explode if doing so would drop an arqs parameter
22:24:14	dansmith	efried: it gets you the circuit breaker, not necessarily the "graceful" part -- that should still be scheduler-based
22:24:31	efried	cool
22:26:28	dansmith	Sundar: while you're here, what's the plan for getting some small example of this running with an actual pci device in the intel pci ci system (or something) ?
22:27:53	Sundar	dansmith: We have the server and the FPGA in a lab, and one person is working on that. We expect it by Jan, if not this year.
22:28:52	Sundar	FWIW, I check each patch set with real FPGAs, with different device profiles, and also the fake driver.
22:29:49	efried	We should have a 3rd party CI called "Intel Cyborg Sundar Manual CI"
22:30:08	dansmith	Sundar: that would be good except that I totally don't trust you as much as a computer
22:30:10	dansmith	:)
22:30:10	Sundar	We already have that hehe
22:30:35	dansmith	Sundar: so one server and one fpga and one person.. what is the plan? to have that be a manual "recheck use fpgas" command or something?
22:30:41	Sundar	dansmith: Too bad, I am more reliable than other past Intel CIs ;)
22:30:56	dansmith	Sundar: that's true, although doesn't really change my feelings about it :)
22:31:11	efried	dansmith: experimental queue?
22:31:33	dansmith	efried: that's what I meant.. some not-on-everything per-request method
22:31:37	efried	or severely restrict files
22:31:53	efried	but yeah
22:32:03	dansmith	per-request would be better I think, if it's severely constrained
23:09:22	openstackgerrit	Matt Riedemann proposed openstack/nova master: DNM: debug cross-cell resize https://review.opendev.org/698304
23:17:18	openstackgerrit	Merged openstack/nova master: Add resource provider allocation unset example to troubleshooting doc https://review.opendev.org/696582
23:17:25	openstackgerrit	Merged openstack/nova master: nova-net: Convert remaining API tests to use neutron https://review.opendev.org/696509
23:38:17	pmatulis	how do i map the hypervisor long ID (server show) to the hypervisor hostname (hypervisor list/show)?
23:40:05	dansmith	pmatulis: if you server-show as admin, you should see the unobfuscated hostid , IIRC
23:41:56	dansmith	pmatulis: as OS-EXT-SRV-ATTR:hypervisor_hostname
#openstack-nova - 2019-12-12
00:01:04	pmatulis	dansmith, yeah, i'm going to change a nova policy to make that available to non-admins
00:16:24	efried	dansmith: here's a weird one: If I shelve my instance, and then my compute goes down, would a subsequent offload "succeed"?
00:16:24	efried	The virt driver's destroy() would have to happen when the compute came back up later, just like we reap instances that are deleted when their compute is down.
00:16:24	efried	But do we do that, or do we just fail the offload and ERROR the instance?
00:28:52	dansmith	efried: offload should succeed when it comes back up, but would fail while the compute is down with just an rpc failure type thing
00:29:10	efried	okay. thanks.
00:29:22	dansmith	O dpm
00:29:24	dansmith	wow.
00:29:31	dansmith	I don't think there's a local delete sort of thing for offload
00:29:51	dansmith	yeah, there's not
00:30:00	efried	okay.
00:30:21	efried	dansmith: typos courtesy of "shift right"?
00:30:32	dansmith	yep
00:30:52	dansmith	efried: also, shelve_offload() is a cast, so no error result or moving instance to ERROR if it fails due to a down compute
00:30:52	efried	I use dvorak
00:31:03	efried	so if I shift right, I just look like I'm having a stroke.
00:31:16	dansmith	heh
00:31:31	efried	dansmith: that was my next question: offload kinda happens "later" sometimes, right?
00:31:46	efried	so... does it still have the context (auth etc) of the user who did the original shelve?
00:32:00	dansmith	it can be requested via the API, or via periodic after a timeout yeah
00:32:11	dansmith	efried: no it could be the operator via the API, or no context via periodic
00:32:32	efried	ugh
00:32:53	dansmith	in fact, I can't think of any reason a regular user would ever manually offload
00:33:09	efried	make space?
00:33:26	efried	but anyway, it's another monkey wrench for me.
00:33:27	dansmith	shelve is what causes them to start saving money, offload is the cache purge for the operator so something else can go there, but shouldn't really impact the user, other than to make unshelve take longer
00:33:41	dansmith	right, the user doesn't care to make space, they're already saving the money
00:33:51	efried	oh, yeah, I guess the space is reclaimed at shelve time.
00:33:58	dansmith	the operator might offload a shelved instance to get the space back before the timer
00:34:09	dansmith	well, not really,
00:34:22	efried	shitballs, I need to go back and rewrite this rewrite.
00:34:33	dansmith	the user doesn't get any space (quota) back, but they do get charged less (theoretically)
00:34:43	dansmith	the operator doesn't get space (i.e. room for another instance on that host) until the offload
00:35:01	efried	oh
00:35:08	efried	allocations etc only go away at offload?
00:35:27	dansmith	I would think.. the virt driver doesn't know a shelve has happened until the offload
00:35:30	efried	so yeah, that's what I meant by "make space". But again yeah, that's something the operator cares about, not the user.
00:35:35	dansmith	right
00:35:50	efried	the virt driver just gets told to snapshot(). I guess they could figure it out from a task state?
00:35:56	efried	but yeah, they don't.
00:36:13	dansmith	well, it would be super problematic if they did,
00:36:37	efried	Well, on that note...
00:36:37	efried	what's the task state during the virt driver's destroy() during the offload?
00:36:43	dansmith	because the point of the shelved state is that it is still ready to be powered back on, which means that like POWERED_OFF state, it still takes the same amount of resources as ACTIVE
00:36:51	efried	(I'm going to have to find out anyway, but if you know off the top...)
00:37:18	dansmith	I dunno, but why? the virt driver shouldn't be implying anything from task_state during an op like that

Earlier   Later