| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-23 | |||
| 16:42:28 | mriedem | i didn't pay attention to any of this at the PTG, so if there are more details from the ptg on alternatives and such, those should be in the spec, including caveats about what happens if the conductor group for a node is updated (as i had several questions about that) | |
| 16:42:50 | mriedem | it sounds like, well that might work automagically, or it might not, shrug | |
| 16:44:06 | mriedem | and i guess we don't want anything in the hypervisors API for this because it would be too ironic specific... | |
| 16:44:26 | melwitt | pvc_: hm, looks like you're right, it says only one vGPU per instance. I don't know why that is limited https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#configure-a-flavor-controller | |
| 16:44:52 | melwitt | bauzas ^ | |
| 16:45:10 | sean-k-mooney | melwitt: its limited because libvirt cannot support multiple vgpus on a singel instance currently | |
| 16:45:21 | melwitt | sean-k-mooney: thanks | |
| 16:45:51 | pvc_ | yes but the problem is we launch instance it will use the same vgpu enabled on the nova.conf, is there no way to use the another vgpu? | |
| 16:46:06 | sean-k-mooney | pvc_: that is not how that works | |
| 16:46:21 | pvc_ | it's just for sharing sean-k-mooney? | |
| 16:46:25 | sean-k-mooney | what you are enabling in the nova.conf is the type of vgpu | |
| 16:46:55 | dansmith | mriedem: there are probably sequencing recommendations for when you change the mappings, but I think those are docs and don't need to be in the spec in great detail, IMHO. I don't think the behaviors are really any different than just spinning up a new or stopping a hashring-balanced compute today | |
| 16:46:56 | pvc_ | so if i use nvida-211 | |
| 16:47:05 | sean-k-mooney | your are not enabvling a specific mdev instnce jsut the type of mdev that will be allocated when a vm requests a vgpu | |
| 16:47:37 | pvc_ | so in terms of performance it is okay sean-k-mooney? | |
| 16:47:46 | dansmith | mriedem: if there's more that needs to be here, let's tell them what that is and let them move on | |
| 16:48:27 | sean-k-mooney | pvc_: mdevs are not shared between instances. so the perfromance of the vgpus will be determined by the mdev type you select | |
| 16:48:44 | pvc_ | so if ever i have 3 mdev types ( nvidia-1, nvidia-2, nvidia-3), i use the nvidia-1 on nova.conf. What will happen to the 2 mdev types? | |
| 16:49:02 | sean-k-mooney | higher perfroamce mdev types consume more physical resouce and allow less vgpus to be createed so its a tradeoff | |
| 16:49:32 | sean-k-mooney | if you set nvidia-1 the nova will only create mdevs of the nvida-1 type | |
| 16:50:22 | sean-k-mooney | there are stict rules about if and how mdev types can be mixed | |
| 16:50:41 | sean-k-mooney | they are vendor specific so initally we chose not to allow mixing them | |
| 16:51:02 | pvc_ | is that okay to be use for many instances? | |
| 16:51:49 | sean-k-mooney | yes we will limit you to the number of mdevs that the enabled type reporst it can allocate | |
| 16:52:03 | sean-k-mooney | there no over subsiptoin or sharing of vgpus | |
| 16:52:50 | pvc_ | so if ever i have 24 mdev types i can launch 24 instance, correct me if im wrong. | |
| 16:53:20 | sean-k-mooney | pvc_: no | |
| 16:54:33 | sean-k-mooney | an mdev type is like a nove flavor. its corresponed to a set of phyical resouces form the gpu. in the nvida case its a set of max reolutions, cuda cores, and vRAM | |
| 16:54:52 | sean-k-mooney | this is all detailed in the nvidia documentation | |
| 16:55:09 | pvc_ | thank you sean-k-mooney | |
| 16:55:28 | pvc_ | so if i have a chance to launch an instance is it okay and nova will tell me if ever i cannot allocate omre | |
| 16:55:55 | sean-k-mooney | yes you will get a no valid host error if all vpus have been consumed | |
| 16:56:43 | sean-k-mooney | nova will read the number of allication that can be made form the mdev-type specified by the nova.conf and report that to placement | |
| 16:57:09 | pvc_ | i understand now, it is okay if you only specifiy one enabled_vgpu_types on nova.conf. But by default it will consume the mdev types. Thank you :) | |
| 16:57:21 | pvc_ | But by default it will consume the all mdev types* | |
| 16:57:23 | pvc_ | am i right? | |
| 16:58:10 | sean-k-mooney | pvc_: no if you dont specyu an enabled vgpu type the no vgpus will be consumable | |
| 16:58:31 | pvc_ | i already specify one enabled ( example nvidia-160) | |
| 17:00:34 | pvc_ | thank you sean-k-mooney. :) | |
| 17:01:49 | sean-k-mooney | pvc_: you should look at https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#vgpu-types-tesla-p100 to determin how your tesla P100 can be subdevied | |
| 17:03:57 | pvc_ | noted on this. thanks for your help. :) | |
| 17:07:35 | openstackgerrit | Merged openstack/nova-specs master: Document Stein review priorities https://review.openstack.org/609807 | |
| 17:17:04 | openstack | Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | |
| 17:17:04 | melwitt | dtantsur: I've been meaning to ask you a question about this ironic bug we fixed around RC time https://bugs.launchpad.net/nova/+bug/1787910 in comment #2 you mentioned the regression broke the ironic-inspector CI upstream. do you happen to know why the ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa job we run in nova did not break the same way? | |
| 17:17:17 | dtantsur | melwitt: looking | |
| 17:18:17 | melwitt | thanks. I don't know the differences between the ironic-inspector job and our job | |
| 17:18:35 | dtantsur | melwitt: okay, so tripleo got broken because it still used disk/ram filters with ironic | |
| 17:18:38 | melwitt | trying to learn if there's a test coverage gap we can close to catch more issues | |
| 17:18:54 | dtantsur | as to inspector, I don't remember why exactly I mentioned that. but there is at least one difference | |
| 17:19:17 | dtantsur | ironic CI does not set vcpus/memory_mb on nodes, which leads to them not exposed to nova for some time | |
| 17:19:29 | dtantsur | ironic-inspector, as part of its functioning, discovers these properties and sets them | |
| 17:19:45 | sean-k-mooney | dtantsur: i was under the impression we ended up keeping the disk/ram filter for that reason in rocky and it was going to be fixed in ironic this cycle | |
| 17:20:24 | dtantsur | sean-k-mooney: I don't think it was for this reason, but I may be missing something | |
| 17:20:40 | dtantsur | for ironic these properties (and filters) are optional since.. mmm.. pike? maybe queens | |
| 17:21:23 | melwitt | dtantsur: thanks for the pointer | |
| 17:21:40 | dtantsur | np | |
| 17:21:46 | sean-k-mooney | dtantsur: we defintly had a bug in the RC period related to them during rocky release. mriedem do you remeber what the bug related to ironc and the disk/ram filter was? | |
| 17:22:25 | dtantsur | sean-k-mooney: the link melwitt posted above? yeah, essentially disk/ram filter stopped working for ironic. since nobody cares about this case any more, we just ended up fixing tripleo to not enable these filters. | |
| 17:22:33 | dtantsur | no further investigation was done IIUC | |
| 17:23:17 | sean-k-mooney | dtantsur: maybe not sure | |
| 17:24:08 | melwitt | yeah, looks like it, use of the filters resulted in NoValidHost because of the bug | |
| 17:25:03 | melwitt | so I probably the failure to update_available_resource was showing up in our ironic job, but was hidden because things otherwise worked (without ram/disk filters). which seems like it would be unexpected | |
| 17:25:06 | sean-k-mooney | i think the only reason to use the disk/ram filter today would be if you were using the caching scheduler since it does not use palcement | |
| 17:27:25 | dtantsur | yep | |
| 17:31:10 | openstackgerrit | sean mooney proposed openstack/nova-specs master: Add spec for sriov live migration https://review.openstack.org/605116 | |
| 17:31:48 | melwitt | looks like there were two different bugs in this bug. one was the update_available_resource fail (which wasn't caught by any CI) and then the core/ram/disk filter problem which was "unfixable", that is, only way out was to stop configuring deployments to use the filters | |
| 17:33:16 | melwitt | it just so happened that because tripleo CI was failing because it was using the core/ram/disk filters, they also noticed the update_available_resource failure in the logs | |
| 17:33:44 | sean-k-mooney | brb | |
| 17:34:26 | openstackgerrit | Merged openstack/nova-specs master: Detach and attach boot volumes - Stein https://review.openstack.org/600628 | |
| 17:52:41 | mriedem | efried: do you/anyone care about this anymore? https://review.openstack.org/#/c/560174/ it was mostly just historical documentation right? | |
| 18:27:01 | efried | mriedem: I'm not sure. The information is probably still useful. I guess the last sentence would need to be updated to reflect how we actually solved it. Swhy I hadn't abandoned it yet. | |
| 18:27:46 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955 | |
| 18:33:47 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/queens: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612774 | |
| 18:33:47 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/queens: Move live_migration.pre.start to the start of the method https://review.openstack.org/612773 | |
| 18:35:26 | efried | mriedem: abandoned | |
| 18:35:27 | melwitt | huh, looks no longer possible to filter logstash by n-cpu log type only. I guess not enough people used it | |
| 18:37:42 | artom | Huh, stable/pike is going to be very problematic for ^^ | |
| 18:38:38 | mriedem | melwitt: tags:"screen-n-cpu.txt" | |
| 18:38:52 | melwitt | mriedem: filters out my result when I do | |
| 18:39:20 | artom | Oh wait, our downstream bug is OSP13/queens, so we're good | |
| 18:39:21 | melwitt | oh wait | |
| 18:39:30 | melwitt | mriedem: user error, my bad | |
| 18:40:30 | edleafe | cdent: ^^ Got all the tests passing locally \o/ | |
| 18:40:43 | edleafe | doh! fat fingers ^^ | |
| 18:41:14 | melwitt | I want to create an e-r query and I'm rusty | |
| 18:41:29 | mriedem | well, there are specs to be reviewed if you wanted to do that instead :) | |
| 18:41:43 | melwitt | I'm doing that too | |
| 18:42:25 | melwitt | I thought this would be quicker than it's being | |
| 18:49:17 | melwitt | bah, there's already a query for this. just e-r hasn't posted anything on it | |
| 18:51:18 | mriedem | jroll: you might want to take a gander at this ironic volume-backed resize/cold migrate spec https://review.openstack.org/#/c/449155/ | |
| 18:51:25 | mriedem | i haven't been through it in quite awhile | |
| 18:51:37 | mriedem | but i'm also not very ironically inclined | |
| 18:52:30 | mriedem | seems that tuba guy would also care about this | |
| 18:53:24 | pvc | so sad | |
| 18:53:24 | pvc | sean-k-mooney suddenly root@test-vgpu:/home/ubuntu# nvidia-smi No devices were found | |
| 19:01:01 | cfriesen | since it's spec review day, I'd appreciate some eyes on https://review.openstack.org/#/c/571111/ | |
| 19:01:14 | cfriesen | (the emulated TPM spec) | |
| 19:02:32 | pvc | nervermind me | |
| 19:04:03 | cfriesen | I think the open questions are whether we want to support CRB at this point (and if so how to ask for it), and whether we need to explicitly call out what happens for non-x86 architectures or leave that for the implementation. | |
| 19:58:00 | mriedem | jaypipes: are there any plans for this? https://review.openstack.org/#/c/529135/ | |