| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-11 | |||
| 13:14:04 | bauzas | because, and I told you that, if you have a physical device that can support X and Y types, once you create *one* single mediated device of X, then you litterally have 0 available for Y | |
| 13:14:14 | bauzas | exactly | |
| 13:14:34 | bauzas | (at least for the nvidia driver) | |
| 13:14:50 | mdbooth | Right, so the only advantage to cleanup is that you could potentially repurpose it later to be Y if you removed all X without requiring a restart. | |
| 13:15:03 | bauzas | mdbooth: http://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf section 1.3.2 "homogeneous vGPUs" | |
| 13:15:11 | mdbooth | That's in the realms of optimisation, though. Very much not important. | |
| 13:15:37 | bauzas | mdbooth: right, as an operator, you really need to consider which type to use, and not try to mix them up in between a single node | |
| 13:16:07 | mdbooth | The major issues were the races, the inconsistent XML generation, and that as a maintainer of the libvirt driver, in general I don't want _get_guest_xml doing even more surprising things. | |
| 13:16:14 | bauzas | I'd be in favor to limit as the maximum the fact to mix up different types across a single compute, even if you have multiple GPUs | |
| 13:16:43 | bauzas | mdbooth: the races are a concern, but I'd be in favor of adding the locks in a follow-up if you don't mind | |
| 13:16:45 | mdbooth | So if you create one of these instances, stop it, then start it, you've lost your vgpu. | |
| 13:16:54 | bauzas | mdbooth: that code is already pretty heavy | |
| 13:17:24 | bauzas | mdbooth: that's part of the extra testing I'm doing on my box | |
| 13:17:48 | mdbooth | bauzas: I think we can safely say it won't work without testing though, right? | |
| 13:17:53 | bauzas | I'm just tackling the last bits of privsep, but I was considering a shelve | |
| 13:17:57 | mdbooth | Because we don't pass allocations to power_on. | |
| 13:18:13 | bauzas | mdbooth: we could if necessary | |
| 13:18:27 | mdbooth | It would also be nice to block live migrations, don't know if we do that alrady. | |
| 13:18:50 | bauzas | mdbooth: well, live migrating an instance having a VGPU attached is like "hold my beer" | |
| 13:18:57 | mdbooth | Similarly, if you rescue one, when you unrescue it you'll have lost your vgpu | |
| 13:19:16 | bauzas | mdbooth: okay, here is what I propose | |
| 13:19:28 | mdbooth | bauzas: Hehe, yeah. I wouldn't expect that to work, tbh. | |
| 13:19:46 | bauzas | mdbooth: step #1 amend the relnote and mention the caveats you noticed | |
| 13:19:54 | mdbooth | Can we split the mdev allocation code into a function which does: | |
| 13:19:57 | bauzas | mdbooth: step #2 do a follow-up on the race | |
| 13:20:26 | bauzas | mdbooth: step #3 and beyond : patch each caller of get_guest_config to pass the allocation | |
| 13:20:27 | mdbooth | _grab_mdevs(allocations) -> list of mdev uuids | |
| 13:20:36 | mdbooth | Then pass list of mdev uuids to _get_guest_xml | |
| 13:21:14 | mdbooth | It would also give us the opportunity to do explicitly different things to generate that list in different contexts. | |
| 13:21:31 | mdbooth | e.g. at startup, our mdevs may exist already or they may not. | |
| 13:22:03 | bauzas | right, that's another caveat, mdevs aren't persistent | |
| 13:22:09 | mdbooth | If we just restarted Nova, they will exist. If we rebooted the host, they won't. | |
| 13:22:28 | bauzas | but I was considering to use init_host to just lookup all the guest XMLs and recreate the mdevs | |
| 13:22:29 | mdbooth | We don't want to be trying to handle weird situations like that in the middle of _get_guest_xml | |
| 13:22:47 | bauzas | (if of course your node kernel doesn't have that uevent race) | |
| 13:23:33 | mdbooth | So my structural concern is with my 'maintainer of the libvirt driver' hat on. It isn't specifically related to mdevs. | |
| 13:23:54 | mdbooth | I'd prefer that we didn't do host configuration in _get_guest_xml. | |
| 13:24:05 | mdbooth | s/do/do more/ | |
| 13:25:21 | bauzas | mdbooth: btw. issue confirmed : if you stop/start your instance, then goodbye my dev | |
| 13:25:50 | bauzas | ok, I'm gonna tackle one issue at time | |
| 13:26:15 | mdbooth | bauzas: We can get allocations back, I think. I'm pretty sure we can do this without having to persist it anywhere, although that was my first thought. | |
| 13:26:25 | bauzas | mdbooth: I'll just spin another revision modifying the privsep call, could you please post your comments back on the next rev ? | |
| 13:26:40 | mdbooth | bauzas: Sure thing. | |
| 13:26:43 | bauzas | mdbooth: this is more a scheduling problem | |
| 13:26:56 | bauzas | mdbooth: because if we stop the guest, then we loose the mediated device | |
| 13:27:09 | mdbooth | Oh, true. | |
| 13:27:10 | bauzas | mdbooth: I wonder if we keep the mdev attachment when we restart | |
| 13:27:32 | bauzas | mdbooth: that needs to be tested, because I ask to lookup all libvirt guests, including those stopped | |
| 13:27:36 | mdbooth | I think the domain sticks around of you just stop it. | |
| 13:27:47 | mdbooth | I think you're good there. | |
| 13:27:50 | bauzas | so, in theory, we still get the association even if the domain is destroyed | |
| 13:27:57 | bauzas | because it's not undefined | |
| 13:28:13 | bauzas | so I think we're good | |
| 13:28:18 | mdbooth | I'm also assuming, btw, that it doesn't matter *which* vgpu we get. | |
| 13:28:24 | bauzas | we could just need to pass the allocations to power_on | |
| 13:28:35 | mdbooth | So stop/start, get a new vgpu, we don't care. | |
| 13:28:41 | bauzas | mdbooth: right, all VGPUs are equally equal :p | |
| 13:28:50 | bauzas | based on the same type, of course :) | |
| 13:28:54 | bauzas | it's cloud, brah | |
| 13:29:05 | mdbooth | Hehe. | |
| 13:31:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: Send bytes to sockets https://review.openstack.org/531834 | |
| 13:31:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce the VeNCrypt RFB authentication scheme https://review.openstack.org/345398 | |
| 13:31:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce framework for RFB authentication https://review.openstack.org/345397 | |
| 13:31:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Document TLS security setup for noVNC proxy https://review.openstack.org/500544 | |
| 13:31:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: Provide an RFB security proxy implementation https://review.openstack.org/345399 | |
| 13:31:40 | stephenfin | melwitt: Addressed concerns in ^ | |
| 13:31:59 | ygl | bauzas: any idea about the scheduler behaving this way ? | |
| 13:32:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Increase notification wait timeout in functional tests https://review.openstack.org/532816 | |
| 13:33:17 | bauzas | mdbooth: https://twitter.com/sylvainbauza/status/951446715470577664 | |
| 13:33:44 | gibi | melwitt: hi!, https://bugs.launchpad.net/nova/+bug/1742688 and https://review.openstack.org/532816 It seems we are the victim of the gate slowness | |
| 13:33:45 | openstack | Launchpad bug 1742688 in OpenStack Compute (nova) "test_live_migration_actions notification sample test fails intermittently with 'notification instance.live_migration_rollback.start hasn't been received'" [Undecided,In progress] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 13:36:56 | bauzas | gibi: yup, the gate is like super-slow | |
| 13:39:23 | bauzas | mdbooth: so, tell me more about what you want | |
| 13:40:01 | bauzas | mdbooth: you said to have a magical _grab_mdevs(allocations) that would return you a list of mdev UUIDs, right? | |
| 13:40:26 | bauzas | mdbooth: so basically identical to _guest_add_vgpus() except the very last line ? | |
| 13:40:59 | mdbooth | bauzas: Sec, lemme look | |
| 13:42:38 | bauzas | mdbooth: tl;dr: I only care about the guest parameter because I eventually do a guest.add_device() | |
| 13:42:50 | bauzas | but at the very very end of that method | |
| 13:42:54 | mdbooth | bauzas: Yes. There's only like 3 lines of that function which relate to xml generation :) | |
| 13:43:24 | bauzas | and 2 of them are unrelated to the guest itself | |
| 13:43:39 | bauzas | so, then, what would be the interest of splitting that ? | |
| 13:43:49 | mdbooth | bauzas: A few | |
| 13:44:03 | mdbooth | The first as I mentioned is that it's surprising | |
| 13:44:30 | mdbooth | Somebody coming along looking for where this stuff is configured in the workflow isn't going to see _get_guest_xml() and guess that it's there. | |
| 13:44:59 | bauzas | mdbooth: that said, we do that *a lot*, right? :p | |
| 13:45:13 | mdbooth | We do a whole lot of surprising stuff, sure :) | |
| 13:45:21 | bauzas | mdbooth: I mean, that's precisely there where we attach what we want, right ? | |
| 13:45:39 | mdbooth | Second is I suspect we're going to want to have different ways to create this list. | |
| 13:45:41 | bauzas | it's not only the GPU thing, it's all the devices we create | |
| 13:45:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Resolve 'oslo.context' deprecation warnings https://review.openstack.org/532822 | |
| 13:45:52 | mdbooth | e.g. creating new mdevs vs re-using existing mdevs | |
| 13:46:16 | mdbooth | s/existing/already allocated to this instance/ | |
| 13:46:25 | bauzas | mdbooth: that is precisely something that should kept hidden from the other libvirt places | |
| 13:46:33 | bauzas | for tech debt reasons | |
| 13:46:34 | mdbooth | Where creating == create or reallocate | |
| 13:46:54 | bauzas | how the vGPUs are implemented in libvirt should be kept very isolated from the rest | |
| 13:47:16 | mdbooth | I'm talking about the code in libvirt/driver.py here | |
| 13:47:20 | bauzas | sure | |
| 13:47:30 | mdbooth | So it shouldn't affect the decisions of other drivers. | |
| 13:47:37 | bauzas | I still think we need to limit the driver knowledge about what is a vGPU | |