Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-16
15:00:00 bauzas sure
15:01:32 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470
15:01:51 kashyap lyarwood: Hiya, hope the renewed comments make sense: forgot that we also need MIN_QEMU_BLOCKDEV
15:02:09 kashyap lyarwood: And a different, newer version for MIN_LIBVIRT_BLOCKDEV. (Notes in the review :))
15:05:07 artom Remind me again what are the criteria for reporting a VM status as UNKNOWN? Host being down is one of them, right?
15:05:19 artom melwitt, ^^ if you're awake
15:05:21 dansmith yeah or cell down
15:05:29 artom dansmith, aha, ack, thanks!
15:05:42 dansmith it started at cell down only, but I think melwitt changed that recently
15:06:51 artom dansmith, that's the impression I got - not super important, just clearing out old (downstream) BZs
15:12:40 spatel sean-k-mooney: Good morning, if i want to disable Hyper Threading without BIOS setting how do i do that? I was reading AWS article here using offline cpu threads, do you think this is valid way to do that - https://aws.amazon.com/blogs/compute/disabling-intel-hyper-threading-technology-on-amazon-linux/
15:14:22 sean-k-mooney spatel: yes you can do it via the sys virtual file system
15:14:54 sean-k-mooney the bios is the best way to do it as ofline cores will not unpartion the l1 cache
15:15:01 spatel sean-k-mooney: does that perform as good as BIOS setting ?
15:15:10 sean-k-mooney not quite
15:15:27 sean-k-mooney if you disable it in the bios it dobles the l1 cache avaiable to the core
15:15:58 sean-k-mooney when ht is enabled the l1 cache is partioned so the each ht has its onw region of the l1 cache
15:16:14 spatel We found erlang doing pretty good job when disable HT but again i don't want to do that setting from BIOS (its painful) i want to give that control to end users
15:16:41 sean-k-mooney but with it disable at the bios level all the l1 cache is avaiable to the physical core. at least on older intel architecutres
15:17:12 sean-k-mooney spatel: are you using cpu pinning
15:17:21 spatel Yes CPU pinning
15:17:26 sean-k-mooney if so you can use the cpu_thread_policy
15:17:47 sean-k-mooney create multiple flaovrs for the earlang instnace and let them choose
15:18:04 spatel sean-k-mooney: cpu_thread_policy=isolate ?
15:18:16 sean-k-mooney e.g. hw:cpu_thread_policy=prefer vs hw:cpu_threads_policy=isolate
15:18:20 sean-k-mooney yes
15:18:51 spatel sean-k-mooney: I have tried all kind of combination but erlang doesn't like it.
15:19:03 spatel when i run VM on single NUMA perfrmance is really good
15:19:03 sean-k-mooney the sysfs performace delta is pretty small since normally your app/data wont fit in l1 anyway
15:21:31 spatel sean-k-mooney: look at this - https://imgur.com/a/8zapZ8x
15:22:13 spatel To understand better CPU topology i am comparing them with AWS and here what i found
15:22:40 spatel On my openstack VM CPU topo looks very strange
15:23:01 sean-k-mooney the lower image looks like what i would expect
15:23:08 spatel l1d & l1i cache is shared
15:23:36 spatel why aws has l1d and l1i outside
15:23:53 sean-k-mooney i dont know
15:24:06 sean-k-mooney but you can alter this in the libvirt xml i think
15:24:27 spatel That is the problem, I have check with Alicloud and aws and both has perfect CPU topo but my openstack has very odd output
15:24:36 sean-k-mooney this is not something we would expose however
15:24:41 spatel I think it could be QEMU version or bug
15:25:07 sean-k-mooney well what do you mean by odd
15:25:08 spatel I am planning to upgrade my qemu to 4.2 (currently running 2.12)
15:25:19 sean-k-mooney the imgae you provide showing the kvm instance
15:25:25 sean-k-mooney look like real hardware would
15:25:28 spatel qemu-kvm
15:25:43 spatel both are virtual machine
15:26:22 sean-k-mooney sure but looking at https://imgur.com/a/8zapZ8x the bottom image looks corect the top look incorrect
15:27:06 spatel you are saying AWS instance looks incorrect?
15:27:13 sean-k-mooney yes
15:27:35 sean-k-mooney that is the toplogy that we should see if and only if you had HT disabled
15:27:36 spatel If i run same command on my host compute it looks exactly like AWS one
15:27:48 spatel sean-k-mooney: no
15:28:08 spatel Let me show you my two physical compute topo (HT vs non-HT)
15:28:10 spatel hold on..
15:30:04 sean-k-mooney ok so looking locally they have changed how this work in later versions
15:30:37 sean-k-mooney the view that you see in openstack i how it used to work in nehalem and i belive up to sandybridge or ivybridge
15:32:05 sean-k-mooney spatel: as i said the bios seting used to change the toplogy between the aws one and the openstack one at the hardware level
15:32:20 spatel sean-k-mooney: https://imgur.com/a/at3WBBf
15:32:44 spatel This is my two compute host (one has HT enable and second has HT enable)
15:33:13 sean-k-mooney spatel: yep as i said this has changed with different hardware micorarchitecutres
15:33:30 spatel If you look bottom picture (its very similar to AWS virtual instance, that means AWS virtual machine correctly exposing physical topology including cache)
15:33:36 sean-k-mooney spatel: openstack/nova is not currently setting the cpu cache toplogy its decied by libvirt
15:34:02 sean-k-mooney spatel: openstack is not ment to expose the host toplogy by defualt
15:34:19 spatel even in host-passthrough ?
15:34:23 sean-k-mooney correct
15:34:33 sean-k-mooney openstack does not specify the cache toplogy at all
15:34:41 spatel hmmm
15:34:42 sean-k-mooney that is left entirely to libvirt today
15:35:09 sean-k-mooney libvirt allows use to set this but we dont so you get whatever libvirt/qemu decied to provide
15:35:33 spatel hmm! how ALI cloud doing this even they are running openstack
15:35:50 spatel may be they have hack version of software design by them
15:37:54 spatel Anyway so you think BIOS level is best way then i will do with that but it will be growing pain as my cloud growing :(
15:39:23 openstackgerrit John Garbutt proposed openstack/nova master: Update limit APIs https://review.opendev.org/712707
15:39:23 openstackgerrit John Garbutt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/712749
15:39:24 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Enforce unified limits using oslo.limit https://review.opendev.org/615180
15:48:03 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834
15:48:54 lyarwood kashyap: ^ updated btw
15:52:05 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles in os-volumes-attachments policies https://review.opendev.org/710190
15:52:05 openstackgerrit Brin Zhang proposed openstack/nova master: Add PATCH volume attachments api to os-volume_attachments https://review.opendev.org/693828
15:52:06 openstackgerrit Brin Zhang proposed openstack/nova master: Add new policy to PATCH update volume API https://review.opendev.org/711194
15:52:06 openstackgerrit Brin Zhang proposed openstack/nova master: Add functional tests for PATCH volume attachments API https://review.opendev.org/710965
15:57:20 dansmith sean-k-mooney: AFAIK, the cyborg patch that generates the libvirt xml hasn't changed much, and you've tested that at some point with real devices (right?) so we can assume it works without much fanfare?
15:59:21 kashyap lyarwood: Will check; thx
16:00:58 sean-k-mooney dansmith: i havent tested with real device no
16:01:06 dansmith oh I thought you had okay
16:01:19 sean-k-mooney dansmith: i can go specific review that patch however
16:01:28 dansmith presumably sundar has
16:01:37 sean-k-mooney i belive you are correct in that it does not change much
16:01:58 sean-k-mooney dansmith: yes sundar has apparently tested it with the rushcreak fpga card
16:02:24 dansmith I looked over it a while back and I think the only way I'd be able to find stuff really wrong with it is through log examination
16:02:25 dansmith it's pretty straightforward
16:05:09 sean-k-mooney dansmith: i rebased the cyborg devstack pluging multinode this moringin by the way. just to resovle the merge conflict
16:05:16 dansmith I saw, thanks
16:06:12 sean-k-mooney if i rebase it again do you want me to move your host name fix patch lower? hoepfully they will merge soon anyway but that is usefaul outside of multinode testing
16:06:51 dansmith it's not super critical unless it's blocking people.. I put it later just to avoid messing up your series, but obviously it's probably an easy merge.. your call
16:08:00 sean-k-mooney ok if i need to respin i can move it down. i dont think other have really complained about it plus you can always override the host via the local.conf anyway
16:11:03 dansmith yup.. I imagine that's because most people are using more throwaway machines for their testing (and it's probably not getting a very wide audience anyway) but.. yep, not critical and there is a workaround
16:12:52 melwitt artom, dansmith: fyi I didn't change the meaning of the UNKNOWN status, it originally was only for host down and then when the down cells handling was added, it was used for that as well. my change was just a new policy rule to allow UNKNOWN status to be seen by non-admin if indicated by policy
16:13:35 dansmith melwitt: it was originally for cell down only, AFAIR
16:13:38 melwitt this is the logic for host down https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5339-L5351

Earlier   Later