| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-11 | |||
| 12:35:42 | gibi | yjk: and that will be more like a new feature than a bugfix | |
| 12:36:35 | yjk | gibi: I though you had asked me to open an openstack bug. That is why opened a bug. | |
| 12:38:12 | gibi | yjk: yes, first I thought that it is a bug then I realzied that it is more like a new feature | |
| 12:38:17 | gibi | yjk: sorry for the confusion | |
| 12:38:20 | ameeda | Hello, Please help me , please check this "http://paste.openstack.org/show/642597/" that caused when I introspect the node. | |
| 12:55:17 | ygl | hi all can someone help me with nova-scheduler please ? | |
| 12:55:38 | bauzas | ygl: sure? | |
| 12:59:15 | ameeda | bauzas: can you please help me ? | |
| 13:00:04 | ygl | bauzas: recently my openstack setup took a hit. it is not launching any instances. I cant see any use ful logs indicative of the problem, even though I enabled debug in nova.conf. After a lot of investigation I found that one of the computes is not reachable by it mgmt IP | |
| 13:00:14 | ygl | bauzas: why is it that the scheduler was not lookin gto for other good computes even though they are available but looking always for that unrechable compute node ? | |
| 13:00:23 | ygl | bauzas: ununtil I removed that compute from the nova zone, it did not work | |
| 13:00:35 | ygl | bauzas: why is it so with the scheduler ? | |
| 13:02:53 | ygl | bauzas: any idea ? | |
| 13:03:13 | bauzas | ygl: not sure I understand your problem | |
| 13:04:39 | bauzas | mdbooth: thanks for your review on https://review.openstack.org/#/c/528832/ | |
| 13:05:00 | bauzas | mdbooth: there are 3 "stages" on having VGPU capabiliities with libvirt | |
| 13:05:01 | mdbooth | bauzas: Thanks for a morning of discovery :) | |
| 13:05:17 | bauzas | mdbooth: first of all, you need a vGPU capable device | |
| 13:05:20 | ygl | bauzas: because of unreachable compute by its management IP, the nova-scheduler is not allowing to launch new instances on remaining compute nodes | |
| 13:05:36 | bauzas | mdbooth: this is expressed by a "mdev capable" PCI device | |
| 13:06:19 | bauzas | ie. a PCI device having mdev "capabilities" that say "for that type, you can have X vGPUs" | |
| 13:06:39 | bauzas | but a single PCI device can support multiple profiles, ie. types | |
| 13:07:05 | bauzas | that's only when you *create* a mediated device that the other types become unavailable | |
| 13:07:16 | bauzas | creating a mediated device is stage #2 | |
| 13:07:46 | bauzas | for creating a mediated device, you can't call the libvirt API but you rather need to use sysfs directly | |
| 13:08:14 | bauzas | that will generate a mediated device that you can see in the sysfs, but no guests are allocated yet to that host | |
| 13:08:16 | ygl | bauzas: any idea ? | |
| 13:08:59 | bauzas | ygl: seriously, no, we filter out unreachable computes but the others are still checked | |
| 13:09:17 | ygl | bauzas: but in my case it was failing | |
| 13:09:19 | mdbooth | bauzas: I think the above is in line with my (very newly acquired) understanding. | |
| 13:09:23 | bauzas | mdbooth: for having a guest having VGPUs, you need a stage #3 | |
| 13:09:31 | ygl | bauzas: until I dsiabled the bad compute, it didi not work | |
| 13:09:39 | bauzas | mdbooth: which is basically use that mediated device in the guest config | |
| 13:10:24 | mdbooth | bauzas: Yep. This is the parallel I drew with host-connected volumes. | |
| 13:10:27 | bauzas | mdbooth: so, about the risk of having all the mdevs not cleaned up, it's a non-problem | |
| 13:10:48 | mdbooth | Step 0 is create the volume (in cinder) | |
| 13:10:51 | bauzas | mdbooth: because we only care of how many mediated devices are still unassigned to a guest | |
| 13:10:59 | mdbooth | Step 1 is create a host device corresponding to that volume | |
| 13:11:06 | mdbooth | Step 2 is add it to a guest. | |
| 13:11:06 | bauzas | if all the mediated devices are already created by the operator, that's not an issue | |
| 13:11:31 | bauzas | and btw. there is a kernel race that prevents libvirtd to see the newly created mediated device, see my relnote | |
| 13:11:45 | mdbooth | Yeah, I read that. That's annoying. | |
| 13:11:51 | bauzas | so, in general, it's better to just create by yourself all the mediated devices beforehand | |
| 13:12:04 | mdbooth | FWIW I wouldn't bother working round that in Nova. Just note that here be dragons and wait for the fix. | |
| 13:12:16 | mdbooth | There are enough caveats for the experimental flag already, I think. | |
| 13:12:40 | bauzas | so, see, given each mediated device is cleaned up when it's detached from the guest, I don't see any problem with reusing them in between all the instances | |
| 13:12:55 | mdbooth | Thanks for that release note, btw. Best I've read. | |
| 13:12:57 | bauzas | (wrt. your last comment) | |
| 13:13:14 | bauzas | "Looking forward, we're not cleaning up mdevs. I don't think this is a problem right now when we only support 1 type, but IIUC allocating an mdev of type X can reduce the availability for type Y, right? If that's the case, I think we should consider cleaning up. This wouldn't be an immediate priority, though.'" | |
| 13:13:42 | mdbooth | bauzas: Got it. Yeah, that would only apply if these are mix and match. i.e. You can have 6 X and 6 Y. | |
| 13:13:56 | mdbooth | But I think you're saying as soon as you have 1 X you can only have X | |
| 13:14:04 | bauzas | because, and I told you that, if you have a physical device that can support X and Y types, once you create *one* single mediated device of X, then you litterally have 0 available for Y | |
| 13:14:14 | bauzas | exactly | |
| 13:14:34 | bauzas | (at least for the nvidia driver) | |
| 13:14:50 | mdbooth | Right, so the only advantage to cleanup is that you could potentially repurpose it later to be Y if you removed all X without requiring a restart. | |
| 13:15:03 | bauzas | mdbooth: http://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf section 1.3.2 "homogeneous vGPUs" | |
| 13:15:11 | mdbooth | That's in the realms of optimisation, though. Very much not important. | |
| 13:15:37 | bauzas | mdbooth: right, as an operator, you really need to consider which type to use, and not try to mix them up in between a single node | |
| 13:16:07 | mdbooth | The major issues were the races, the inconsistent XML generation, and that as a maintainer of the libvirt driver, in general I don't want _get_guest_xml doing even more surprising things. | |
| 13:16:14 | bauzas | I'd be in favor to limit as the maximum the fact to mix up different types across a single compute, even if you have multiple GPUs | |
| 13:16:43 | bauzas | mdbooth: the races are a concern, but I'd be in favor of adding the locks in a follow-up if you don't mind | |
| 13:16:45 | mdbooth | So if you create one of these instances, stop it, then start it, you've lost your vgpu. | |
| 13:16:54 | bauzas | mdbooth: that code is already pretty heavy | |
| 13:17:24 | bauzas | mdbooth: that's part of the extra testing I'm doing on my box | |
| 13:17:48 | mdbooth | bauzas: I think we can safely say it won't work without testing though, right? | |
| 13:17:53 | bauzas | I'm just tackling the last bits of privsep, but I was considering a shelve | |
| 13:17:57 | mdbooth | Because we don't pass allocations to power_on. | |
| 13:18:13 | bauzas | mdbooth: we could if necessary | |
| 13:18:27 | mdbooth | It would also be nice to block live migrations, don't know if we do that alrady. | |
| 13:18:50 | bauzas | mdbooth: well, live migrating an instance having a VGPU attached is like "hold my beer" | |
| 13:18:57 | mdbooth | Similarly, if you rescue one, when you unrescue it you'll have lost your vgpu | |
| 13:19:16 | bauzas | mdbooth: okay, here is what I propose | |
| 13:19:28 | mdbooth | bauzas: Hehe, yeah. I wouldn't expect that to work, tbh. | |
| 13:19:46 | bauzas | mdbooth: step #1 amend the relnote and mention the caveats you noticed | |
| 13:19:54 | mdbooth | Can we split the mdev allocation code into a function which does: | |
| 13:19:57 | bauzas | mdbooth: step #2 do a follow-up on the race | |
| 13:20:26 | bauzas | mdbooth: step #3 and beyond : patch each caller of get_guest_config to pass the allocation | |
| 13:20:27 | mdbooth | _grab_mdevs(allocations) -> list of mdev uuids | |
| 13:20:36 | mdbooth | Then pass list of mdev uuids to _get_guest_xml | |
| 13:21:14 | mdbooth | It would also give us the opportunity to do explicitly different things to generate that list in different contexts. | |
| 13:21:31 | mdbooth | e.g. at startup, our mdevs may exist already or they may not. | |
| 13:22:03 | bauzas | right, that's another caveat, mdevs aren't persistent | |
| 13:22:09 | mdbooth | If we just restarted Nova, they will exist. If we rebooted the host, they won't. | |
| 13:22:28 | bauzas | but I was considering to use init_host to just lookup all the guest XMLs and recreate the mdevs | |
| 13:22:29 | mdbooth | We don't want to be trying to handle weird situations like that in the middle of _get_guest_xml | |
| 13:22:47 | bauzas | (if of course your node kernel doesn't have that uevent race) | |
| 13:23:33 | mdbooth | So my structural concern is with my 'maintainer of the libvirt driver' hat on. It isn't specifically related to mdevs. | |
| 13:23:54 | mdbooth | I'd prefer that we didn't do host configuration in _get_guest_xml. | |
| 13:24:05 | mdbooth | s/do/do more/ | |
| 13:25:21 | bauzas | mdbooth: btw. issue confirmed : if you stop/start your instance, then goodbye my dev | |
| 13:25:50 | bauzas | ok, I'm gonna tackle one issue at time | |
| 13:26:15 | mdbooth | bauzas: We can get allocations back, I think. I'm pretty sure we can do this without having to persist it anywhere, although that was my first thought. | |
| 13:26:25 | bauzas | mdbooth: I'll just spin another revision modifying the privsep call, could you please post your comments back on the next rev ? | |
| 13:26:40 | mdbooth | bauzas: Sure thing. | |
| 13:26:43 | bauzas | mdbooth: this is more a scheduling problem | |
| 13:26:56 | bauzas | mdbooth: because if we stop the guest, then we loose the mediated device | |
| 13:27:09 | mdbooth | Oh, true. | |
| 13:27:10 | bauzas | mdbooth: I wonder if we keep the mdev attachment when we restart | |
| 13:27:32 | bauzas | mdbooth: that needs to be tested, because I ask to lookup all libvirt guests, including those stopped | |
| 13:27:36 | mdbooth | I think the domain sticks around of you just stop it. | |