Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-23
20:52:51 cdent mriedem: meh, I'd let them ride a bit longer, might still be a chance
20:53:42 jaypipes melwitt: yeah, couldn't sleep.
20:53:59 mriedem overcommitting a dedicated pcpu... https://review.openstack.org/#/c/599957/ seems...odd
20:54:08 jaypipes BTW, I have COMPLETELY failed in my sean-k-mooney spellchecker powers today.
20:54:11 mriedem isn't that an oxymoron?
20:54:45 mriedem oxymoron = overcommitted dedicated pcpu
20:54:48 melwitt thinking about that hurts my brain
20:54:54 jaypipes mriedem: yeah, it is, and I chatted with tpatil about that in Denver
20:55:14 melwitt definitely sounds contradictory
20:55:31 jaypipes mriedem: mostly they just need the whole "allow a single compute host to have dedicated stuff and non-dedicated stuff on the same box" thing
20:55:48 jaypipes mriedem: I doubt Tushar will follow up on that spec.
20:55:53 mriedem and then PCPU inventory has an allocation_ratio>1.0?
20:56:48 mriedem b/c this spec https://review.openstack.org/#/c/599957/ is all about many new config options
20:56:54 mriedem which kinda sucks
21:00:38 melwitt they're saying two of the options (cpu_dedicated_set, cpu_shared_set) come from a different spec though, `Standardize CPU resource tracking`
21:02:22 melwitt so only the cpu_pinning_allocation_ratio would be new. which really I guess is meaning pcpu_allocation_ratio, whereas the existing cpu_allocation_ratio technically means vcpu_allocation_ratio?
21:03:18 melwitt so that they separate handling of vcpu vs pcpu
21:05:38 melwitt also interesting, it says based on the prereq spec https://review.openstack.org/#/c/555081 that PCPU inventory will be created with hardcoded allocation ratio of 1.0 and they want to be able to change it/overcommit it. so wouldn't that just be a call to the placement API?
21:06:45 melwitt but I guess they want to be able to set it the same way as is possible for VCPU
21:06:55 melwitt via nova.conf
21:36:09 mriedem edleafe: all sorts of API gross for you to munch on here https://review.openstack.org/#/c/580336/4
21:41:09 cfriesen I think my question about the "overcommit PCPUs" idea is what does it buy you that a low CPU overcommit ratio (with non-dedicated cpus) wouldn't?
21:46:40 sean-k-mooney jaypipes: haha was there a partical pharse that exausted the spellchecking ablity today :)
21:48:53 sean-k-mooney cfriesen: over commiting pinned cpus i think would be less objectionable but the fact we choose hw:cpu_policy=dedicated|shared and not hw:cpu_policy=pinned|floating makes me dislike the idea
21:49:16 edleafe mriedem: gee thanks!
21:52:29 cfriesen sean-k-mooney: once you have more than one instance using a cpu, it's shared. I don't see what this would buy us compared to just using a shared CPU
21:54:45 sean-k-mooney cfriesen: well i very limited cases it may improve you performance as pinning will result in numa affinity for leagacy reasons but i would personally prefer hw:cpu_policy=share + hw:numa_nodes=1
21:55:42 cfriesen sean-k-mooney: agreed. And if you were using hugepages you'd get the numa affinity anyways.
21:56:24 sean-k-mooney cfriesen: also since we are going to be using the dedictate_cpu_set for allocating realtime cpus i think this is likely to break that also
21:57:00 cfriesen wait...how is a realtime cpu different from a regular dedicated cpu?
21:58:36 sean-k-mooney cfriesen: realtime we set the "nice" value or whatever the priority values is on linux to realtime so it does not get premented if you have a realtime kernel
21:58:51 sean-k-mooney cfriesen: we dont set the tread prioity for dedicated cpus
22:02:33 cfriesen sean-k-mooney: that's what I thought, just making sure. if you've already got a "dedicated" cpu with nothing else running on it, what's the benefit of making the vcpu task realtime?
22:02:56 cfriesen (unless the host hasn't properly moved all kernel work off that cpu)
22:03:09 sean-k-mooney basiclly ^
22:03:25 sean-k-mooney or there is some backgound process that is on the host that is not confied properly
22:03:30 sean-k-mooney but very little
22:05:05 cfriesen but if the host hasn't moved kernel work off that cpu, and you run the qemu task as realtime, don't you risk priority inversion anyways if the guest is doing a busy-loop and never letting other tasks run?
22:05:59 jaypipes sean-k-mooney: no, just distracted generally :)
22:06:01 sean-k-mooney cfriesen: one if its actully a kernel thread then the kernel will run it if it needs too
22:06:31 sean-k-mooney cfriesen: second if its a user tread the kernel can rescudle it i think if the vm is in a busy loop
22:06:34 mriedem melwitt: can we abandon https://review.openstack.org/#/c/509042/ or are there plans to update that?
22:07:07 sean-k-mooney cfriesen: i think we tell people dont use this unless you have set up your host properly for realtime workloads and properly isolated the cores
22:07:21 melwitt mriedem: I'd like to update it but we need allocation ownership concepts in placement else it's moot
22:07:33 mriedem so is anyone driving that dependency?
22:07:37 sean-k-mooney cfriesen: at least i tell people do ues the realtime feature unless you have set up the host properly
22:08:44 cfriesen sean-k-mooney: on hosts with the RT kernel a bunch of kernel things get run in schedulable threads
22:08:55 melwitt mriedem: no, just saying, that's why it's stuck
22:09:21 melwitt and maybe, I guess I could ask jaypipes because I thought I saw mention of the idea of an owner attribute in some other spec
22:09:25 sean-k-mooney cfriesen: on https://review.openstack.org/#/c/599957 i suggested just adding a hw:cpu_policy=pinned which would pin the vm to one of the shared cpu set cores instead. does that sound better then over commiting dedicated cpus to you?
22:09:25 mriedem ok, so....if it's stuck, and no one is working on unstucking it, and it's not high enough priority to are, should we just abandon
22:09:34 mriedem *care
22:09:51 openstackgerrit Merged openstack/nova-specs master: Dynamically find releases for move-implemented-specs https://review.openstack.org/592628
22:10:08 sean-k-mooney cfriesen: oh ya but you have to use isolcpus too if you useing the realtime core extra specs in nova correctly
22:10:10 cfriesen sean-k-mooney: I think you'd need to pin each vCPU in the VM to one of the shared cpu set cores.
22:10:25 sean-k-mooney cfriesen: yes that is what i was suggesting
22:10:46 melwitt mriedem: for the record, I care about it a lot but I can't argue that allocation ownership in placement is a priority given everything else that's going on. I can abandon it on that basis
22:10:57 cfriesen sean-k-mooney: isolcpus means no scheduling, so only works with explicit pinning and one vcpu per pcpu.
22:11:12 sean-k-mooney cfriesen: yep
22:11:21 cfriesen sean-k-mooney: yeah, so pinning to shared cpu set cores makes more sense to me
22:18:33 sean-k-mooney cfriesen: ok that comment is in there spec but while i kind of get this usescase i also think its not a great one.
22:20:25 cfriesen it's purely a small performance optimization compared to just using "shared"
22:21:08 cfriesen which is valid, but it makes the resource tracking really messy
22:21:20 sean-k-mooney cfriesen: well in tushars case they want to have more over subsction with the same perfromce i think
22:21:40 cfriesen sean-k-mooney: their performance document is comparing against "shared"
22:22:15 sean-k-mooney yes but they are not comparing against shared with hw:numa_nodes=1 are they
22:22:21 cfriesen nope.
22:22:21 sean-k-mooney also where is the doc?
22:22:28 cfriesen linked at the bottom of the spec
22:24:10 openstackgerrit Merged openstack/nova-specs master: fix tox python3 overrides https://review.openstack.org/579793
22:24:11 sean-k-mooney not to kill this entirlaly but the other thing that makes me uncomfortable about this is by doing this you increase the risk of specter or l1tf
22:25:08 sean-k-mooney in a public cloud enve you are now allowing two instace that could be from different host to be pinned to the same core where they will context switch
22:27:30 cfriesen sean-k-mooney: agreed, that seems sketchy. ideally you'd want to ensure only instances from the same tenant were permanently pinned together
22:27:43 sean-k-mooney ya which is a pain
#openstack-nova - 2018-10-24
01:40:02 litao hi
01:56:08 openstackgerrit liuming proposed openstack/nova master: Deletes evacuated instance files when source host is ok https://review.openstack.org/605987
02:10:49 openstackgerrit Vu Cong Tuan proposed openstack/nova-specs master: Switch to stestr https://review.openstack.org/581284
02:26:45 alex_xu gmann: sorry, I can't join office hour today
02:27:28 gmann alex_xu: ok, i will skip for today then. thanks for informing.
04:00:42 openstackgerrit Merged openstack/nova master: Fix up compute rpcapi version for pike release https://review.openstack.org/612231
04:11:51 openstackgerrit melanie witt proposed openstack/nova stable/rocky: Fix up compute rpcapi version for pike release https://review.openstack.org/612561
06:19:21 openstackgerrit Merged openstack/nova stable/rocky: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611327
07:33:32 openstackgerrit Yongli He proposed openstack/nova-specs master: add spec "show-server-numa-topology" https://review.openstack.org/612256
07:46:44 pvc hi anyone sean-k-mooney or bauzas?
07:50:31 pvc i already launch an instance iwth vgpu on it
07:50:57 pvc but which driver should i use?
07:51:05 pvc to run a gpu application
07:53:56 bauzas good morning nova
08:18:25 bauzas pvc: you should use the grid guest driver
08:19:00 bauzas which was originally planned, but not lasting so long
08:33:55 pvc i've successfully run the nvidia-smi bauzas
08:34:01 pvc Product Name : GRID P100-2B4
08:34:08 bauzas cool
08:34:10 pvc Virtualization mode : VGPU
08:34:17 pvc but i cannot test the tensorflow-gpu :(
08:34:31 pvc do you have any prefer docs for that
08:44:29 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Support delete_on_termination in volume attach api https://review.openstack.org/612949
08:46:40 openstackgerrit Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846
08:46:40 openstack bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9)
08:46:40 openstackgerrit Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733

Earlier   Later