| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-15 | |||
| 14:47:09 | dansmith | everyone I have heard talk about this makes it sound like they tent to target a pretty specific GPU while designing their application | |
| 14:47:41 | dansmith | not that they're not speaking a generic language to it such that it could run on a different model of the same family, but that it defeats the point of hyper-optimization | |
| 14:48:25 | Kevin_Zheng | mriedem: Hi, as for your comment in https://review.openstack.org/#/c/568542 I added the loop and other logic because I was considering race conditions, like maybe when I check the existence of future obj in queue, it was there, but when I actually cancel it, it went to preparing or running so I might do also some cleanup | |
| 14:48:30 | jaypipes | dansmith: any chance you can put me in touch with an example user of these things? | |
| 14:48:36 | dansmith | it would be like saying "I want a PCI nic", and having only intel drivers in your image, but you get put on a box and handed a mellanox card | |
| 14:48:59 | dansmith | jaypipes: lol | |
| 14:49:23 | openstackgerrit | Merged openstack/nova master: __str__ methods for RequestGroup, ResourceRequest https://review.openstack.org/568353 | |
| 14:49:43 | jaypipes | dansmith: I wasn't being facetious... | |
| 14:49:55 | jaypipes | dansmith: sorry | |
| 14:50:18 | Kevin_Zheng | mriedem: the loop is because I thought if it is in “preparing” it will end up either “error” or “running” so we can cancel it when it turn to “running” or just ignore it if it turned to “error” | |
| 14:50:53 | Kevin_Zheng | mriedem: that was just my consideration, not sure it is correct | |
| 14:51:20 | dansmith | jaypipes: so you already have a cuda trait, which means you effectively already have an nvidia one, | |
| 14:51:43 | dansmith | jaypipes: what happens when I've got my image that supports m60 gpus, and then next month you add a bunch of nodes with m61s, | |
| 14:51:58 | dansmith | I start booting instances that get there and their drivers are too old to recognize and use an m61? | |
| 14:52:31 | jaypipes | dansmith: the traits indicate the version of the CUDA, OpenGL, OpenCL drivers that are supported by the vGPU type. | |
| 14:53:03 | jaypipes | dansmith: https://github.com/openstack/os-traits/blob/master/os_traits/hw/gpu/api.py | |
| 14:53:08 | dansmith | jaypipes: okay, so for cuda that might help | |
| 14:53:21 | jaypipes | dansmith: so the user would just say "hey, I wrote my app using this version of CUDA" | |
| 14:53:27 | jaypipes | dansmith: and put that trait in their image metaedata | |
| 14:53:32 | dansmith | jaypipes: what about AMD and intel? there isn't a uniform opencl api bump to indicate each one right? | |
| 14:53:44 | dansmith | and I don't have amd drivers in my instance, but you put me there | |
| 14:54:18 | jaypipes | dansmith: we wouldn't put you on an AMD instance if you asked for CUDA 7.1... | |
| 14:54:23 | dansmith | no, | |
| 14:54:41 | dansmith | I'm saying let's say I'm a non-cuda instance.. I'm using OpenCL, and I asked for v2.2 | |
| 14:54:58 | dansmith | I have intel drivers in my instance, but you put me on an AMD-only node that supports v2.2 | |
| 14:56:38 | jaypipes | dansmith: not sure? :) | |
| 14:58:20 | jaypipes | dansmith: wikipedia states this... "Programs in the OpenCL language are intended to be compiled at run-time, so that OpenCL-using applications are portable between implementations for various host devices." | |
| 14:58:35 | dansmith | um, so? | |
| 14:58:45 | dansmith | the drivers and libraries are in my instance (or not) | |
| 14:58:54 | dansmith | once again, archlinux is the best wiki on the internet: https://wiki.archlinux.org/index.php/GPGPU | |
| 14:59:02 | dansmith | see the package names you have to get installed for each type of device? | |
| 15:00:02 | jaypipes | dansmith: yeah. | |
| 15:00:25 | dansmith | those being distro packages either bundle or download proprietary binaries for that layer | |
| 15:00:26 | jaypipes | dansmith: back to hardware-defined software it seems. :( | |
| 15:00:59 | dansmith | pretty sure the nvidia ones at least are very closely-guarded and require an account with endowments in order to download them | |
| 15:02:16 | dansmith | also pretty sure most of these have pretty specific kernel versions that are required, | |
| 15:02:30 | dansmith | as they also utilize the kernel-side driver, fglrx, nvidia, etc | |
| 15:02:48 | jaypipes | dansmith: my fear is that we will get locked in to vendor-specific identifiers and behaviour in the user-facing API and when these vendors inevitably change directions, we'll be forced to change our APIs. | |
| 15:03:31 | dansmith | jaypipes: I don't think we are though, because the traits are just enums.. if nothing is exposing m60 right now, then.. nothing boots that requires it | |
| 15:04:53 | dansmith | and going back to your point of trying to shame me, | |
| 15:05:04 | jaypipes | dansmith: what does m60 give you though? I can understand a potential need for traits that represent all these vendor drivers for OpenCL and CUDA, but the model name doesn't give any more information than the resolution, display heads and API library versions, right? | |
| 15:05:18 | dansmith | I don't want vendor lock-in of course, as you know, but I don't think enumerating the things that could be out there is doing that | |
| 15:05:24 | dansmith | jaypipes: of course it does | |
| 15:05:45 | dansmith | jaypipes: max resolution and display heads means nothing to me, I need cores/shaders/memory for GPGPU | |
| 15:05:55 | dansmith | jaypipes: and of course, lineage so I know if my drivers are appropriate | |
| 15:06:14 | jaypipes | but the drivers are for a version of CUDA or OpenCL, though, right? | |
| 15:06:17 | dansmith | jaypipes: if I'm an amd instance and I have a specific version of the drivers, I may be able to support a foobar99 but not a foobar128 | |
| 15:06:25 | jaypipes | you don't have drivers for M60, drivers for M61, etc, right? | |
| 15:06:26 | dansmith | I'm not sure what the appropriate level of abstraction is exactly, | |
| 15:06:44 | dansmith | maybe m60 is too fine-grained, but I think probably not because there are cards with 1, 2, 4 m60 chips, etc | |
| 15:08:14 | openstackgerrit | Vladyslav Drok proposed openstack/nova-specs master: Allow having placement inventories with reserved value equal to total https://review.openstack.org/568613 | |
| 15:12:23 | openstackgerrit | Merged openstack/nova master: trivial: Explain how the marker works for instance-cell mapping https://review.openstack.org/567597 | |
| 15:26:46 | jaypipes | dansmith: ok, so I'll just point out one last word on this VGPU stuff... AWS' elastic GPUs don't mention any vendor at all (https://aws.amazon.com/ec2/elastic-gpus/). They only mention "supports OpenGL 4.3" and up to 8G of GPU memory. Now, I know it's just one data point, but it does seem they are trying not to get into the game of leaking out vendor information ... | |
| 15:28:39 | dansmith | jaypipes: check out this in their FAQ: Q. Do I need a driver for Elastic GPUs? | |
| 15:29:00 | dansmith | jaypipes: looks like they have network-virtualized the GPUs so they can be attached anywhere, and they consume network bandwidth | |
| 15:29:20 | dansmith | they use OpenGL as their interface, | |
| 15:29:21 | dansmith | which means you're pretty far removed from the thing as a compute device, AFAIK | |
| 15:29:35 | dansmith | they also seem to focus on "graphics acceleration", which would be in line with being OpenGL-only | |
| 15:29:46 | openstackgerrit | Vladyslav Drok proposed openstack/nova-specs master: Allow having placement inventories with reserved value equal to total https://review.openstack.org/568613 | |
| 15:30:18 | dansmith | jaypipes: ah, they say use P3 for GPGPU | |
| 15:30:39 | dansmith | jaypipes: https://aws.amazon.com/ec2/instance-types/p3/ | |
| 15:30:42 | dansmith | jaypipes: tesla v100 only | |
| 15:30:46 | dansmith | jaypipes: so, yeah, easy :) | |
| 15:32:12 | dansmith | and note they're not saying "you'll get some nvidia somethingorother" they say .. "it'll be a damned v100, alwas" | |
| 15:32:13 | jaypipes | dansmith: I see them say tesla v100. those are passthrough direct GPU devices, though, not vGPU "types", right? | |
| 15:32:35 | jaypipes | "Alternatively, you can also use the NVIDIA AMI with GPU driver and CUDA toolkit pre-installed." ... interesting. | |
| 15:32:52 | dansmith | jaypipes: yeah I dunno if it's a vGPU or them just giving you a whole card | |
| 15:33:04 | jaypipes | dansmith: it's the whole card. or 8 of them :) | |
| 15:33:32 | dansmith | doesn't have anything to do with our discussion though right? they're very specific about what you get, and thus what driver you need | |
| 15:35:08 | jaypipes | dansmith: yes and no? :) the "elastic GPUs" are very much non-vendor-specific. The P3 instances are clearly very much "here is this specific machine configuration with 8 NVIDIA GPUs", etc. | |
| 15:35:24 | jaypipes | dansmith: but I see your point. | |
| 15:35:38 | jaypipes | dansmith: we need to support both cloudy and non-cloudy things. | |
| 15:35:52 | dansmith | jaypipes: but they're not for GPGPU, they're for graphics acceleration | |
| 15:36:01 | jaypipes | dansmith: I've just been trying my best to keep the vendor-specific stuff out of the way. | |
| 15:36:05 | dansmith | and they virtualize the OpenGL interface, so you're far removed | |
| 15:36:09 | jaypipes | dansmith: yes, I understand. | |
| 15:38:02 | openstackgerrit | Vladyslav Drok proposed openstack/nova-specs master: Allow having placement inventories with reserved value equal to total https://review.openstack.org/568613 | |
| 15:38:24 | dansmith | and, they achieve the vendor neutrality of the elastic gpu thing by creating their own vendor so to speak, and provide signed drivers for supported operating systems | |
| 15:38:28 | dansmith | if they didn't have that, | |
| 15:38:45 | dansmith | they'd have to have specific details about which drivers you needed to have to support all their stuff, or provide flavors to let you choose which ones, | |
| 15:38:56 | dansmith | because drivers are required one way or the other | |
| 15:39:20 | dansmith | so unless you're going to write a driver and backend system to do that, I don't see how even the elastic gpu case alters the discussion | |
| 15:40:22 | jaypipes | dansmith: sounds like we need traits that represent the vendor software drivers, then... | |
| 15:41:53 | dansmith | or we expect that people have to do that on their own by using custom traits | |
| 15:42:11 | dansmith | which is what I said in that review, but then that generates a much less interoperable situation for everyone | |
| 15:42:33 | jaypipes | agreed.' | |
| 15:42:48 | jaypipes | dansmith: ok, thx for talking through this with me. was very useful. | |
| 15:43:01 | jaypipes | and for the record, I did not try to shame you. sorry if anything came across that way. | |
| 15:43:16 | dansmith | we also can punt the problem for a while, saying people need to use custom traits for now and see how it goes | |
| 15:45:45 | jaypipes | ack | |
| 15:46:21 | openstackgerrit | Merged openstack/nova master: placement: Fix HTTP error generation https://review.openstack.org/568567 | |
| 16:01:39 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] default to accept of application/json when */* https://review.openstack.org/568630 | |
| 16:08:27 | mriedem | stephenfin: have to talked with claudiu about whether or not the hyperv driver can support https://review.openstack.org/#/c/541290/ also? | |
| 16:09:07 | stephenfin | mriedem: I have not, but HyperV's support for all things NUMA is minimal at best | |
| 16:09:23 | stephenfin | Far as I can tell, they let you define guest topologies but not anything to do with the host topology | |
| 16:10:23 | stephenfin | So they don't do things like "restrict my instance's vCPU processes to CPUs from the same NUMA nodes as the attached PCI device". Wouldn't make sense (yet!) to do that for networks either | |
| 16:20:02 | mriedem | stephenfin: ok, as dansmith mentioned in the spec, it would be good to call out what is lacking in hyper-v to restrict them from being able to do the new magic | |
| 16:20:20 | mriedem | stephenfin: if you haven't seen yet, lots of comments on that spec, mostly concerned about the data modeling changes being proposed | |
| 16:20:26 | dansmith | yeah, sounds like it's cool, just a sentence saying "they don't do host topo" would be good I think | |
| 16:21:06 | mriedem | i'd also like to know what's going on with bauzas' numa topology modeling in placement spec | |