| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-23 | |||
| 22:08:55 | melwitt | mriedem: no, just saying, that's why it's stuck | |
| 22:09:21 | melwitt | and maybe, I guess I could ask jaypipes because I thought I saw mention of the idea of an owner attribute in some other spec | |
| 22:09:25 | mriedem | ok, so....if it's stuck, and no one is working on unstucking it, and it's not high enough priority to are, should we just abandon | |
| 22:09:25 | sean-k-mooney | cfriesen: on https://review.openstack.org/#/c/599957 i suggested just adding a hw:cpu_policy=pinned which would pin the vm to one of the shared cpu set cores instead. does that sound better then over commiting dedicated cpus to you? | |
| 22:09:34 | mriedem | *care | |
| 22:09:51 | openstackgerrit | Merged openstack/nova-specs master: Dynamically find releases for move-implemented-specs https://review.openstack.org/592628 | |
| 22:10:08 | sean-k-mooney | cfriesen: oh ya but you have to use isolcpus too if you useing the realtime core extra specs in nova correctly | |
| 22:10:10 | cfriesen | sean-k-mooney: I think you'd need to pin each vCPU in the VM to one of the shared cpu set cores. | |
| 22:10:25 | sean-k-mooney | cfriesen: yes that is what i was suggesting | |
| 22:10:46 | melwitt | mriedem: for the record, I care about it a lot but I can't argue that allocation ownership in placement is a priority given everything else that's going on. I can abandon it on that basis | |
| 22:10:57 | cfriesen | sean-k-mooney: isolcpus means no scheduling, so only works with explicit pinning and one vcpu per pcpu. | |
| 22:11:12 | sean-k-mooney | cfriesen: yep | |
| 22:11:21 | cfriesen | sean-k-mooney: yeah, so pinning to shared cpu set cores makes more sense to me | |
| 22:18:33 | sean-k-mooney | cfriesen: ok that comment is in there spec but while i kind of get this usescase i also think its not a great one. | |
| 22:20:25 | cfriesen | it's purely a small performance optimization compared to just using "shared" | |
| 22:21:08 | cfriesen | which is valid, but it makes the resource tracking really messy | |
| 22:21:20 | sean-k-mooney | cfriesen: well in tushars case they want to have more over subsction with the same perfromce i think | |
| 22:21:40 | cfriesen | sean-k-mooney: their performance document is comparing against "shared" | |
| 22:22:15 | sean-k-mooney | yes but they are not comparing against shared with hw:numa_nodes=1 are they | |
| 22:22:21 | sean-k-mooney | also where is the doc? | |
| 22:22:21 | cfriesen | nope. | |
| 22:22:28 | cfriesen | linked at the bottom of the spec | |
| 22:24:10 | openstackgerrit | Merged openstack/nova-specs master: fix tox python3 overrides https://review.openstack.org/579793 | |
| 22:24:11 | sean-k-mooney | not to kill this entirlaly but the other thing that makes me uncomfortable about this is by doing this you increase the risk of specter or l1tf | |
| 22:25:08 | sean-k-mooney | in a public cloud enve you are now allowing two instace that could be from different host to be pinned to the same core where they will context switch | |
| 22:27:30 | cfriesen | sean-k-mooney: agreed, that seems sketchy. ideally you'd want to ensure only instances from the same tenant were permanently pinned together | |
| 22:27:43 | sean-k-mooney | ya which is a pain | |
| #openstack-nova - 2018-10-24 | |||
| 01:40:02 | litao | hi | |
| 01:56:08 | openstackgerrit | liuming proposed openstack/nova master: Deletes evacuated instance files when source host is ok https://review.openstack.org/605987 | |
| 02:10:49 | openstackgerrit | Vu Cong Tuan proposed openstack/nova-specs master: Switch to stestr https://review.openstack.org/581284 | |
| 02:26:45 | alex_xu | gmann: sorry, I can't join office hour today | |
| 02:27:28 | gmann | alex_xu: ok, i will skip for today then. thanks for informing. | |
| 04:00:42 | openstackgerrit | Merged openstack/nova master: Fix up compute rpcapi version for pike release https://review.openstack.org/612231 | |
| 04:11:51 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Fix up compute rpcapi version for pike release https://review.openstack.org/612561 | |
| 06:19:21 | openstackgerrit | Merged openstack/nova stable/rocky: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611327 | |
| 07:33:32 | openstackgerrit | Yongli He proposed openstack/nova-specs master: add spec "show-server-numa-topology" https://review.openstack.org/612256 | |
| 07:46:44 | pvc | hi anyone sean-k-mooney or bauzas? | |
| 07:50:31 | pvc | i already launch an instance iwth vgpu on it | |
| 07:50:57 | pvc | but which driver should i use? | |
| 07:51:05 | pvc | to run a gpu application | |
| 07:53:56 | bauzas | good morning nova | |
| 08:18:25 | bauzas | pvc: you should use the grid guest driver | |
| 08:19:00 | bauzas | which was originally planned, but not lasting so long | |
| 08:33:55 | pvc | i've successfully run the nvidia-smi bauzas | |
| 08:34:01 | pvc | Product Name : GRID P100-2B4 | |
| 08:34:08 | bauzas | cool | |
| 08:34:10 | pvc | Virtualization mode : VGPU | |
| 08:34:17 | pvc | but i cannot test the tensorflow-gpu :( | |
| 08:34:31 | pvc | do you have any prefer docs for that | |
| 08:44:29 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support delete_on_termination in volume attach api https://review.openstack.org/612949 | |
| 08:46:40 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733 | |
| 08:46:40 | openstack | bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9) | |
| 08:46:40 | openstackgerrit | Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846 | |
| 08:47:40 | pvc_ | are you running tensorflow on your instance bauzas | |
| 08:54:59 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support deleting data volume when destroy instance https://review.openstack.org/580336 | |
| 09:03:56 | pvc_ | hi bauzas u using cuda | |
| 09:06:43 | pvc_ | tensorflow.python.framework.errors_impl.InternalError: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error :( | |
| 09:07:28 | sean-k-mooney | pvc_: are you using one of the Q series mdev-types | |
| 09:07:29 | sean-k-mooney | https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#cuda-open-cl-support-vgpu | |
| 09:08:52 | pvc_ | NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] | |
| 09:09:11 | sean-k-mooney | for you gpu https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#vgpu-types-tesla-p100 | |
| 09:09:59 | sean-k-mooney | you need to fine the mdev-type that correspond to one of P100-1Q P100-2Q P100-4Q P100-8Q or P100-16Q | |
| 09:10:08 | pvc_ | videocard of my instance: 00:05.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:15f8] (rev a1) | |
| 09:10:44 | sean-k-mooney | pvc_: yes that likely not going to change regardless of the mdev type yoou choose | |
| 09:12:26 | pvc_ | im using nvidia-211 | |
| 09:13:57 | sean-k-mooney | sure that does not help us map to the nvidia docs. you will need to look at the vendor data reported for the mdev type and see if you can find the vgpu type | |
| 09:14:31 | pvc_ | GRID P100-2B4 | |
| 09:14:35 | pvc_ | the name of nvidia-211 | |
| 09:15:02 | sean-k-mooney | right so that is a B series vgpu and does not support compute via opencl or cuda | |
| 09:15:46 | sean-k-mooney | pvc_: find the one corresponding to P100-2Q | |
| 09:15:52 | sean-k-mooney | that will be the most similar | |
| 09:15:56 | pvc_ | okay wait | |
| 09:16:33 | pvc_ | cat nvidia-84/name GRID P100-2Q | |
| 09:17:15 | sean-k-mooney | cool so if you set that in the nova.conf and boot a new vm it should work with cuda | |
| 09:17:37 | sean-k-mooney | you shoudl look at https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#vgpu-types-tesla-p100 and determin which one meets your needs | |
| 09:17:44 | pvc_ | wait | |
| 09:17:44 | pvc_ | may i know where i can find that information? | |
| 09:17:46 | pvc_ | thank you i'll try | |
| 09:18:38 | pvc_ | what docs can i read to determine where cuda will run? | |
| 09:18:58 | sean-k-mooney | all the vGPU types that end in Q support cuda acordeing to https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#cuda-open-cl-support-vgpu. | |
| 09:19:08 | sean-k-mooney | its the same doc just later in section 1.6.2 | |
| 09:19:33 | pvc_ | wow that's so cool :) | |
| 09:19:35 | pvc_ | thank you so much | |
| 09:19:43 | pvc_ | i'll lauch again an instance | |
| 09:20:43 | pvc_ | can i use P100-1Q sean-k-mooney? | |
| 09:20:49 | sean-k-mooney | so decodeing this a bit more it appears the number before the Q in there name ing is the amound of ram so the p100-2q has 2GB of fram buffer so you can run 8 of them on the 16GB p100 | |
| 09:20:51 | sean-k-mooney | yes | |
| 09:21:11 | sean-k-mooney | that will simply have 1GB of vRAM allocated to its framebuffer instead | |
| 09:21:29 | sean-k-mooney | that will allow you to run 16 vms on each phyicl p100 | |
| 09:21:32 | pvc_ | 2Q is better right | |
| 09:21:39 | pvc_ | so i can launch 8 instances with 2Q | |
| 09:21:45 | sean-k-mooney | yep | |
| 09:21:57 | pvc_ | i have 2 tesla p100 | |
| 09:22:13 | sean-k-mooney | 8 with 2Q 4 with 4Q 2 with 8Q and 1 with 16Q | |
| 09:22:21 | pvc_ | http://paste.openstack.org/show/732950/ | |
| 09:22:41 | sean-k-mooney | so its a trade off between number of vms you can run and performcne that each vm will have | |
| 09:25:10 | sean-k-mooney | pvc_: what software license do you have by the way. | |
| 09:25:38 | sean-k-mooney | you will need the Quadro vDWS license to be able to used the higher performance Q series vgpu types | |
| 09:26:27 | pvc_ | i didnt use any license for now | |
| 09:26:29 | pvc_ | do i need to install it | |