| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-23 | |||
| 21:46:40 | sean-k-mooney | jaypipes: haha was there a partical pharse that exausted the spellchecking ablity today :) | |
| 21:48:53 | sean-k-mooney | cfriesen: over commiting pinned cpus i think would be less objectionable but the fact we choose hw:cpu_policy=dedicated|shared and not hw:cpu_policy=pinned|floating makes me dislike the idea | |
| 21:49:16 | edleafe | mriedem: gee thanks! | |
| 21:52:29 | cfriesen | sean-k-mooney: once you have more than one instance using a cpu, it's shared. I don't see what this would buy us compared to just using a shared CPU | |
| 21:54:45 | sean-k-mooney | cfriesen: well i very limited cases it may improve you performance as pinning will result in numa affinity for leagacy reasons but i would personally prefer hw:cpu_policy=share + hw:numa_nodes=1 | |
| 21:55:42 | cfriesen | sean-k-mooney: agreed. And if you were using hugepages you'd get the numa affinity anyways. | |
| 21:56:24 | sean-k-mooney | cfriesen: also since we are going to be using the dedictate_cpu_set for allocating realtime cpus i think this is likely to break that also | |
| 21:57:00 | cfriesen | wait...how is a realtime cpu different from a regular dedicated cpu? | |
| 21:58:36 | sean-k-mooney | cfriesen: realtime we set the "nice" value or whatever the priority values is on linux to realtime so it does not get premented if you have a realtime kernel | |
| 21:58:51 | sean-k-mooney | cfriesen: we dont set the tread prioity for dedicated cpus | |
| 22:02:33 | cfriesen | sean-k-mooney: that's what I thought, just making sure. if you've already got a "dedicated" cpu with nothing else running on it, what's the benefit of making the vcpu task realtime? | |
| 22:02:56 | cfriesen | (unless the host hasn't properly moved all kernel work off that cpu) | |
| 22:03:09 | sean-k-mooney | basiclly ^ | |
| 22:03:25 | sean-k-mooney | or there is some backgound process that is on the host that is not confied properly | |
| 22:03:30 | sean-k-mooney | but very little | |
| 22:05:05 | cfriesen | but if the host hasn't moved kernel work off that cpu, and you run the qemu task as realtime, don't you risk priority inversion anyways if the guest is doing a busy-loop and never letting other tasks run? | |
| 22:05:59 | jaypipes | sean-k-mooney: no, just distracted generally :) | |
| 22:06:01 | sean-k-mooney | cfriesen: one if its actully a kernel thread then the kernel will run it if it needs too | |
| 22:06:31 | sean-k-mooney | cfriesen: second if its a user tread the kernel can rescudle it i think if the vm is in a busy loop | |
| 22:06:34 | mriedem | melwitt: can we abandon https://review.openstack.org/#/c/509042/ or are there plans to update that? | |
| 22:07:07 | sean-k-mooney | cfriesen: i think we tell people dont use this unless you have set up your host properly for realtime workloads and properly isolated the cores | |
| 22:07:21 | melwitt | mriedem: I'd like to update it but we need allocation ownership concepts in placement else it's moot | |
| 22:07:33 | mriedem | so is anyone driving that dependency? | |
| 22:07:37 | sean-k-mooney | cfriesen: at least i tell people do ues the realtime feature unless you have set up the host properly | |
| 22:08:44 | cfriesen | sean-k-mooney: on hosts with the RT kernel a bunch of kernel things get run in schedulable threads | |
| 22:08:55 | melwitt | mriedem: no, just saying, that's why it's stuck | |
| 22:09:21 | melwitt | and maybe, I guess I could ask jaypipes because I thought I saw mention of the idea of an owner attribute in some other spec | |
| 22:09:25 | mriedem | ok, so....if it's stuck, and no one is working on unstucking it, and it's not high enough priority to are, should we just abandon | |
| 22:09:25 | sean-k-mooney | cfriesen: on https://review.openstack.org/#/c/599957 i suggested just adding a hw:cpu_policy=pinned which would pin the vm to one of the shared cpu set cores instead. does that sound better then over commiting dedicated cpus to you? | |
| 22:09:34 | mriedem | *care | |
| 22:09:51 | openstackgerrit | Merged openstack/nova-specs master: Dynamically find releases for move-implemented-specs https://review.openstack.org/592628 | |
| 22:10:08 | sean-k-mooney | cfriesen: oh ya but you have to use isolcpus too if you useing the realtime core extra specs in nova correctly | |
| 22:10:10 | cfriesen | sean-k-mooney: I think you'd need to pin each vCPU in the VM to one of the shared cpu set cores. | |
| 22:10:25 | sean-k-mooney | cfriesen: yes that is what i was suggesting | |
| 22:10:46 | melwitt | mriedem: for the record, I care about it a lot but I can't argue that allocation ownership in placement is a priority given everything else that's going on. I can abandon it on that basis | |
| 22:10:57 | cfriesen | sean-k-mooney: isolcpus means no scheduling, so only works with explicit pinning and one vcpu per pcpu. | |
| 22:11:12 | sean-k-mooney | cfriesen: yep | |
| 22:11:21 | cfriesen | sean-k-mooney: yeah, so pinning to shared cpu set cores makes more sense to me | |
| 22:18:33 | sean-k-mooney | cfriesen: ok that comment is in there spec but while i kind of get this usescase i also think its not a great one. | |
| 22:20:25 | cfriesen | it's purely a small performance optimization compared to just using "shared" | |
| 22:21:08 | cfriesen | which is valid, but it makes the resource tracking really messy | |
| 22:21:20 | sean-k-mooney | cfriesen: well in tushars case they want to have more over subsction with the same perfromce i think | |
| 22:21:40 | cfriesen | sean-k-mooney: their performance document is comparing against "shared" | |
| 22:22:15 | sean-k-mooney | yes but they are not comparing against shared with hw:numa_nodes=1 are they | |
| 22:22:21 | sean-k-mooney | also where is the doc? | |
| 22:22:21 | cfriesen | nope. | |
| 22:22:28 | cfriesen | linked at the bottom of the spec | |
| 22:24:10 | openstackgerrit | Merged openstack/nova-specs master: fix tox python3 overrides https://review.openstack.org/579793 | |
| 22:24:11 | sean-k-mooney | not to kill this entirlaly but the other thing that makes me uncomfortable about this is by doing this you increase the risk of specter or l1tf | |
| 22:25:08 | sean-k-mooney | in a public cloud enve you are now allowing two instace that could be from different host to be pinned to the same core where they will context switch | |
| 22:27:30 | cfriesen | sean-k-mooney: agreed, that seems sketchy. ideally you'd want to ensure only instances from the same tenant were permanently pinned together | |
| 22:27:43 | sean-k-mooney | ya which is a pain | |
| #openstack-nova - 2018-10-24 | |||
| 01:40:02 | litao | hi | |
| 01:56:08 | openstackgerrit | liuming proposed openstack/nova master: Deletes evacuated instance files when source host is ok https://review.openstack.org/605987 | |
| 02:10:49 | openstackgerrit | Vu Cong Tuan proposed openstack/nova-specs master: Switch to stestr https://review.openstack.org/581284 | |
| 02:26:45 | alex_xu | gmann: sorry, I can't join office hour today | |
| 02:27:28 | gmann | alex_xu: ok, i will skip for today then. thanks for informing. | |
| 04:00:42 | openstackgerrit | Merged openstack/nova master: Fix up compute rpcapi version for pike release https://review.openstack.org/612231 | |
| 04:11:51 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Fix up compute rpcapi version for pike release https://review.openstack.org/612561 | |
| 06:19:21 | openstackgerrit | Merged openstack/nova stable/rocky: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611327 | |
| 07:33:32 | openstackgerrit | Yongli He proposed openstack/nova-specs master: add spec "show-server-numa-topology" https://review.openstack.org/612256 | |
| 07:46:44 | pvc | hi anyone sean-k-mooney or bauzas? | |
| 07:50:31 | pvc | i already launch an instance iwth vgpu on it | |
| 07:50:57 | pvc | but which driver should i use? | |
| 07:51:05 | pvc | to run a gpu application | |
| 07:53:56 | bauzas | good morning nova | |
| 08:18:25 | bauzas | pvc: you should use the grid guest driver | |
| 08:19:00 | bauzas | which was originally planned, but not lasting so long | |
| 08:33:55 | pvc | i've successfully run the nvidia-smi bauzas | |
| 08:34:01 | pvc | Product Name : GRID P100-2B4 | |
| 08:34:08 | bauzas | cool | |
| 08:34:10 | pvc | Virtualization mode : VGPU | |
| 08:34:17 | pvc | but i cannot test the tensorflow-gpu :( | |
| 08:34:31 | pvc | do you have any prefer docs for that | |
| 08:44:29 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support delete_on_termination in volume attach api https://review.openstack.org/612949 | |
| 08:46:40 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733 | |
| 08:46:40 | openstack | bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9) | |
| 08:46:40 | openstackgerrit | Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846 | |
| 08:47:40 | pvc_ | are you running tensorflow on your instance bauzas | |
| 08:54:59 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support deleting data volume when destroy instance https://review.openstack.org/580336 | |
| 09:03:56 | pvc_ | hi bauzas u using cuda | |
| 09:06:43 | pvc_ | tensorflow.python.framework.errors_impl.InternalError: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error :( | |
| 09:07:28 | sean-k-mooney | pvc_: are you using one of the Q series mdev-types | |
| 09:07:29 | sean-k-mooney | https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#cuda-open-cl-support-vgpu | |
| 09:08:52 | pvc_ | NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] | |
| 09:09:11 | sean-k-mooney | for you gpu https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#vgpu-types-tesla-p100 | |
| 09:09:59 | sean-k-mooney | you need to fine the mdev-type that correspond to one of P100-1Q P100-2Q P100-4Q P100-8Q or P100-16Q | |
| 09:10:08 | pvc_ | videocard of my instance: 00:05.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:15f8] (rev a1) | |
| 09:10:44 | sean-k-mooney | pvc_: yes that likely not going to change regardless of the mdev type yoou choose | |
| 09:12:26 | pvc_ | im using nvidia-211 | |
| 09:13:57 | sean-k-mooney | sure that does not help us map to the nvidia docs. you will need to look at the vendor data reported for the mdev type and see if you can find the vgpu type | |
| 09:14:31 | pvc_ | GRID P100-2B4 | |
| 09:14:35 | pvc_ | the name of nvidia-211 | |
| 09:15:02 | sean-k-mooney | right so that is a B series vgpu and does not support compute via opencl or cuda | |
| 09:15:46 | sean-k-mooney | pvc_: find the one corresponding to P100-2Q | |
| 09:15:52 | sean-k-mooney | that will be the most similar | |
| 09:15:56 | pvc_ | okay wait | |
| 09:16:33 | pvc_ | cat nvidia-84/name GRID P100-2Q | |
| 09:17:15 | sean-k-mooney | cool so if you set that in the nova.conf and boot a new vm it should work with cuda | |
| 09:17:37 | sean-k-mooney | you shoudl look at https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#vgpu-types-tesla-p100 and determin which one meets your needs | |