| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-03 | |||
| 15:29:58 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: Get inventories per vGPU type https://review.openstack.org/564215 | |
| 15:30:05 | efried | gibi: Thanks. | |
| 15:31:53 | gibi | efried: https://github.com/openstack/watcher/blob/master/watcher/decision_engine/model/notification/nova.py | |
| 15:32:24 | gibi | efried: I mriedem listed the others in the meanwhile | |
| 15:32:39 | efried | gibi, mriedem: Thanks y'all. | |
| 15:38:38 | mriedem | TheJulia: left some comments in https://bugs.launchpad.net/nova/+bug/1766661 | |
| 15:38:38 | openstack | Launchpad bug 1766661 in OpenStack Compute (nova) "'host' property is missing for Cinder volume connector when boot from volume" [Medium,Confirmed] | |
| 15:38:54 | mriedem | tl;dr nova knows how to get the host connector from the db if needed | |
| 15:39:11 | openstackgerrit | Hongbin Lu proposed openstack/nova-specs master: Choose default network on ambiguity https://review.openstack.org/520247 | |
| 15:39:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Network bandwidth resource provider https://review.openstack.org/502306 | |
| 15:42:58 | exarlos | Hi all. I had a user attempt a resize of a VM (horizon) and it has failed and reduced the VM to an error state | |
| 15:43:13 | exarlos | ocata / ubuntu / lvm storage | |
| 15:43:39 | exarlos | It mentions not being able to stat the /var/lib/nova/instances folder | |
| 15:44:09 | mriedem | https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/driver.py#L7412 | |
| 15:44:34 | mriedem | is it volume-backed or not? | |
| 15:45:17 | exarlos | mriedem: local lvm disks as the backend | |
| 15:45:26 | mriedem | then you'll eventually be failing because ^ | |
| 15:45:46 | mriedem | i think cfriesen might have a patch floating around for that | |
| 15:46:17 | mriedem | https://review.openstack.org/#/c/337334/ | |
| 15:46:21 | exarlos | mriedem: It has worked many times in the past. :-( | |
| 15:46:52 | melwitt | so, you're saying regression from newton -> ocata? | |
| 15:46:52 | mriedem | have you patched nova to make that happen? | |
| 15:48:29 | exarlos | I hadn't patched. There is a allow_resize_to_same_host option in nova.conf that permitted this to happen. | |
| 15:50:01 | mriedem | given https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/driver.py#L7412 i don't see how that's possible | |
| 15:50:46 | exarlos | Let me check local code. | |
| 15:50:59 | exarlos | Honestly though :-) | |
| 15:53:10 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 15:53:17 | bauzas | jaypipes: efried: naichuans (I know it's late for you): I have a thought about providing traits for vGPU types | |
| 15:53:45 | efried | bauzas: Listening. Though I confess I'm behind on those reviews. | |
| 15:53:55 | bauzas | jaypipes: efried: naichuans: the problem is that vGPU types differ in between virt drivers, and even can be depending on the GPU host driver (eg. nvidia vs. intel) | |
| 15:54:06 | bauzas | efried: no worries, it's unrelate | |
| 15:55:00 | efried | bauzas: The thing doing the dev discovery == the thing setting up the resource providers == the thing doing the provisioning == the virt driver. | |
| 15:55:05 | bauzas | jaypipes: efried: naichuans: so, say, I'd like to provide a flavor asking for *this* type, should we provide specific traits like https://docs.nvidia.com/grid/6.0/grid-vgpu-user-guide/index.html#supported-gpus-grid-vgpu ? | |
| 15:55:07 | efried | That's how it boils down for me. | |
| 15:55:49 | bauzas | efried: ok, so you're not concerned by the virt driver setting for example a trait like "nvidia-11" while Xen would set a different one ? | |
| 15:55:56 | efried | not at all. | |
| 15:56:03 | bauzas | if so, that's super cool, it prevents me to write a mapping table | |
| 15:56:03 | efried | well | |
| 15:56:17 | exarlos | 15.0.7 is the version of nova components i'm running. | |
| 15:56:19 | efried | bauzas: If we can standardize some traits that all virt drivers could reasonably use when talking about the same device, that's the best thing. | |
| 15:56:33 | bauzas | efried: see for example https://docs.nvidia.com/grid/6.0/grid-vgpu-user-guide/index.html#vgpu-types-tesla-m60 | |
| 15:56:37 | exarlos | Looks like the code is different, so probably I am a bit behind. | |
| 15:56:55 | efried | bauzas: IOW, the value the virt driver discovers may be different from one virt driver to the next, but as long as it can map it to the standard trait, we're good. | |
| 15:57:08 | bauzas | efried: the GPU type named "M60-0B" is named "nvidia-11" by the nvidia gpu driver | |
| 15:57:29 | bauzas | efried: so the mediated devices etc. use that "nvidia-11" name | |
| 15:57:44 | bauzas | efried: but if you use xen, it's totally named differently | |
| 15:57:55 | efried | bauzas: But is it still a 1-1 mapping? | |
| 15:58:18 | bauzas | efried: https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#how-to-discover-a-gpu-type | |
| 15:58:49 | efried | bauzas: eek. | |
| 15:58:49 | bauzas | efried: in libvirt, yes, a 1:1 | |
| 15:59:01 | efried | Let's talk through the use case real quick. | |
| 15:59:02 | exarlos | mriedem: So, whilst I look at a few things, how best do I pull this back from an error state, and set it so it's bootable again? :-/ | |
| 15:59:19 | bauzas | efried: blame the gpu driver. :p | |
| 15:59:36 | efried | bauzas: From the user pov, is it likely that he's going to put together a flavor that asks for a vgpu and then not care whether he gets a libvirt or xen instance to satisfy it? | |
| 16:00:00 | bauzas | efried: that's a flavor, right? so I don't expect at all end users requesting that | |
| 16:00:17 | efried | sorry, I never remember the right term. | |
| 16:00:21 | bauzas | efried: I rather see operators setting conf file accordingly with flavors | |
| 16:00:45 | bauzas | they could name the flavor GRID M60-0Q I'm fine | |
| 16:01:01 | efried | bauzas: So what I'm trying to figure out is whether there's such a thing as a flavor that would include a VGPU, that's possible to use to deploy instances on different hypervisors? | |
| 16:01:04 | bauzas | but the extra spec would be "requested=nvidia-11" | |
| 16:01:23 | bauzas | efried: mixing environments is highly discouraged | |
| 16:01:41 | efried | Well then I think it doesn't matter whether the traits are named the same from one hypervisor to the next. | |
| 16:01:41 | bauzas | you can do that but then split your cloud | |
| 16:01:56 | bauzas | efried: that said, there is a flaw | |
| 16:01:57 | exarlos | mriedem: and also, should I expect each "server" to have a respective folder in /var/lib/nova/instances/<uuid> ? | |
| 16:02:17 | bauzas | efried: the mdev GPU type name depends on the nvidia driver | |
| 16:02:48 | bauzas | efried: since it depends on a vendor driver, we have no guarantee that 'nvidia-11' will still correspond to a specific GPU type in the future | |
| 16:03:14 | efried | bauzas: Right, so that's what I meant by 1-1 | |
| 16:03:17 | bauzas | efried: if nvidia bumps a new driver that passes types that are not backwards compatible, then operators need to set again their flavors | |
| 16:03:40 | bauzas | that's not a public API, and I hate that | |
| 16:04:02 | bauzas | the kernel just leaks out what the VFIO device tells | |
| 16:04:03 | efried | bauzas: So wait, it's possible that 'nvidia-11' today corresponds to M60-0B and tomorrow will correspond to M99-0X ?? | |
| 16:04:10 | bauzas | efried: possibly yeah | |
| 16:04:14 | bauzas | crazy, isn't it ? | |
| 16:04:14 | efried | that's bullshit, yo. | |
| 16:04:23 | bauzas | that's a vendor API | |
| 16:04:34 | bauzas | vendors can do crazy things | |
| 16:04:44 | bauzas | including breaking changes | |
| 16:05:09 | mriedem | exarlos: without knowing where this actually failed and how it failed, it's hard to say | |
| 16:05:33 | mriedem | if the guest is still running on the source host, which you could verify via virsh (assuming libvirt), then you could probably just reset the state of the instance | |
| 16:05:35 | efried | bauzas: Well, we can only do so much. I think this mapping is something the virt driver will need to be able to keep track of. And if the stoopid vendor driver makes a breaking change, the virt driver will just have to scramble to accomodate. | |
| 16:05:49 | bauzas | efried: I think I reached the same conclusion | |
| 16:05:50 | mriedem | https://developer.openstack.org/api-ref/compute/#reset-server-state-os-resetstate-action | |
| 16:06:06 | bauzas | efried: for some reason, xen is better designed than libvirt | |
| 16:06:06 | efried | bauzas: But I think as much as we possibly can, we should create standard traits and ask the virt drivers to map to them. | |
| 16:06:15 | bauzas | efried: it already makes the translation I guess | |
| 16:06:33 | efried | bauzas: Which one is doing translation, though? | |
| 16:06:37 | bauzas | efried: because if you look the Xen GPU types, those look very identical to the nvidia doc | |
| 16:06:45 | bauzas | efried: the xen API | |
| 16:06:50 | efried | bauzas: Right, so it's libvirt that's translating. | |
| 16:06:57 | bauzas | efried: no | |
| 16:06:59 | efried | xen is just using 'em as is | |
| 16:07:01 | efried | no? | |
| 16:07:13 | bauzas | efried: libvirt is straight setting what the kernel module tells | |
| 16:07:14 | bauzas | https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt | |
| 16:07:49 | efried | oh, so xen is mapping those back to the model names in the literature | |
| 16:08:00 | exarlos | mriedem: Let me go have a read of that. | |
| 16:08:17 | exarlos | It is on the same host still. | |
| 16:08:24 | bauzas | efried: that's what I guess, yes | |
| 16:08:38 | efried | bauzas: so the point is that we need to choose one or the other, and then the docs for a given hypervisor have to tell me how to discover and determine the appropriate corresponding trait name. | |