| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-16 | |||
| 15:15:01 | spatel | sean-k-mooney: does that perform as good as BIOS setting ? | |
| 15:15:10 | sean-k-mooney | not quite | |
| 15:15:27 | sean-k-mooney | if you disable it in the bios it dobles the l1 cache avaiable to the core | |
| 15:15:58 | sean-k-mooney | when ht is enabled the l1 cache is partioned so the each ht has its onw region of the l1 cache | |
| 15:16:14 | spatel | We found erlang doing pretty good job when disable HT but again i don't want to do that setting from BIOS (its painful) i want to give that control to end users | |
| 15:16:41 | sean-k-mooney | but with it disable at the bios level all the l1 cache is avaiable to the physical core. at least on older intel architecutres | |
| 15:17:12 | sean-k-mooney | spatel: are you using cpu pinning | |
| 15:17:21 | spatel | Yes CPU pinning | |
| 15:17:26 | sean-k-mooney | if so you can use the cpu_thread_policy | |
| 15:17:47 | sean-k-mooney | create multiple flaovrs for the earlang instnace and let them choose | |
| 15:18:04 | spatel | sean-k-mooney: cpu_thread_policy=isolate ? | |
| 15:18:16 | sean-k-mooney | e.g. hw:cpu_thread_policy=prefer vs hw:cpu_threads_policy=isolate | |
| 15:18:20 | sean-k-mooney | yes | |
| 15:18:51 | spatel | sean-k-mooney: I have tried all kind of combination but erlang doesn't like it. | |
| 15:19:03 | sean-k-mooney | the sysfs performace delta is pretty small since normally your app/data wont fit in l1 anyway | |
| 15:19:03 | spatel | when i run VM on single NUMA perfrmance is really good | |
| 15:21:31 | spatel | sean-k-mooney: look at this - https://imgur.com/a/8zapZ8x | |
| 15:22:13 | spatel | To understand better CPU topology i am comparing them with AWS and here what i found | |
| 15:22:40 | spatel | On my openstack VM CPU topo looks very strange | |
| 15:23:01 | sean-k-mooney | the lower image looks like what i would expect | |
| 15:23:08 | spatel | l1d & l1i cache is shared | |
| 15:23:36 | spatel | why aws has l1d and l1i outside | |
| 15:23:53 | sean-k-mooney | i dont know | |
| 15:24:06 | sean-k-mooney | but you can alter this in the libvirt xml i think | |
| 15:24:27 | spatel | That is the problem, I have check with Alicloud and aws and both has perfect CPU topo but my openstack has very odd output | |
| 15:24:36 | sean-k-mooney | this is not something we would expose however | |
| 15:24:41 | spatel | I think it could be QEMU version or bug | |
| 15:25:07 | sean-k-mooney | well what do you mean by odd | |
| 15:25:08 | spatel | I am planning to upgrade my qemu to 4.2 (currently running 2.12) | |
| 15:25:19 | sean-k-mooney | the imgae you provide showing the kvm instance | |
| 15:25:25 | sean-k-mooney | look like real hardware would | |
| 15:25:28 | spatel | qemu-kvm | |
| 15:25:43 | spatel | both are virtual machine | |
| 15:26:22 | sean-k-mooney | sure but looking at https://imgur.com/a/8zapZ8x the bottom image looks corect the top look incorrect | |
| 15:27:06 | spatel | you are saying AWS instance looks incorrect? | |
| 15:27:13 | sean-k-mooney | yes | |
| 15:27:35 | sean-k-mooney | that is the toplogy that we should see if and only if you had HT disabled | |
| 15:27:36 | spatel | If i run same command on my host compute it looks exactly like AWS one | |
| 15:27:48 | spatel | sean-k-mooney: no | |
| 15:28:08 | spatel | Let me show you my two physical compute topo (HT vs non-HT) | |
| 15:28:10 | spatel | hold on.. | |
| 15:30:04 | sean-k-mooney | ok so looking locally they have changed how this work in later versions | |
| 15:30:37 | sean-k-mooney | the view that you see in openstack i how it used to work in nehalem and i belive up to sandybridge or ivybridge | |
| 15:32:05 | sean-k-mooney | spatel: as i said the bios seting used to change the toplogy between the aws one and the openstack one at the hardware level | |
| 15:32:20 | spatel | sean-k-mooney: https://imgur.com/a/at3WBBf | |
| 15:32:44 | spatel | This is my two compute host (one has HT enable and second has HT enable) | |
| 15:33:13 | sean-k-mooney | spatel: yep as i said this has changed with different hardware micorarchitecutres | |
| 15:33:30 | spatel | If you look bottom picture (its very similar to AWS virtual instance, that means AWS virtual machine correctly exposing physical topology including cache) | |
| 15:33:36 | sean-k-mooney | spatel: openstack/nova is not currently setting the cpu cache toplogy its decied by libvirt | |
| 15:34:02 | sean-k-mooney | spatel: openstack is not ment to expose the host toplogy by defualt | |
| 15:34:19 | spatel | even in host-passthrough ? | |
| 15:34:23 | sean-k-mooney | correct | |
| 15:34:33 | sean-k-mooney | openstack does not specify the cache toplogy at all | |
| 15:34:41 | spatel | hmmm | |
| 15:34:42 | sean-k-mooney | that is left entirely to libvirt today | |
| 15:35:09 | sean-k-mooney | libvirt allows use to set this but we dont so you get whatever libvirt/qemu decied to provide | |
| 15:35:33 | spatel | hmm! how ALI cloud doing this even they are running openstack | |
| 15:35:50 | spatel | may be they have hack version of software design by them | |
| 15:37:54 | spatel | Anyway so you think BIOS level is best way then i will do with that but it will be growing pain as my cloud growing :( | |
| 15:39:23 | openstackgerrit | John Garbutt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/712749 | |
| 15:39:23 | openstackgerrit | John Garbutt proposed openstack/nova master: Update limit APIs https://review.opendev.org/712707 | |
| 15:39:24 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Enforce unified limits using oslo.limit https://review.opendev.org/615180 | |
| 15:48:03 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834 | |
| 15:48:54 | lyarwood | kashyap: ^ updated btw | |
| 15:52:05 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add PATCH volume attachments api to os-volume_attachments https://review.opendev.org/693828 | |
| 15:52:05 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-volumes-attachments policies https://review.opendev.org/710190 | |
| 15:52:06 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add functional tests for PATCH volume attachments API https://review.opendev.org/710965 | |
| 15:52:06 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new policy to PATCH update volume API https://review.opendev.org/711194 | |
| 15:57:20 | dansmith | sean-k-mooney: AFAIK, the cyborg patch that generates the libvirt xml hasn't changed much, and you've tested that at some point with real devices (right?) so we can assume it works without much fanfare? | |
| 15:59:21 | kashyap | lyarwood: Will check; thx | |
| 16:00:58 | sean-k-mooney | dansmith: i havent tested with real device no | |
| 16:01:06 | dansmith | oh I thought you had okay | |
| 16:01:19 | sean-k-mooney | dansmith: i can go specific review that patch however | |
| 16:01:28 | dansmith | presumably sundar has | |
| 16:01:37 | sean-k-mooney | i belive you are correct in that it does not change much | |
| 16:01:58 | sean-k-mooney | dansmith: yes sundar has apparently tested it with the rushcreak fpga card | |
| 16:02:24 | dansmith | I looked over it a while back and I think the only way I'd be able to find stuff really wrong with it is through log examination | |
| 16:02:25 | dansmith | it's pretty straightforward | |
| 16:05:09 | sean-k-mooney | dansmith: i rebased the cyborg devstack pluging multinode this moringin by the way. just to resovle the merge conflict | |
| 16:05:16 | dansmith | I saw, thanks | |
| 16:06:12 | sean-k-mooney | if i rebase it again do you want me to move your host name fix patch lower? hoepfully they will merge soon anyway but that is usefaul outside of multinode testing | |
| 16:06:51 | dansmith | it's not super critical unless it's blocking people.. I put it later just to avoid messing up your series, but obviously it's probably an easy merge.. your call | |
| 16:08:00 | sean-k-mooney | ok if i need to respin i can move it down. i dont think other have really complained about it plus you can always override the host via the local.conf anyway | |
| 16:11:03 | dansmith | yup.. I imagine that's because most people are using more throwaway machines for their testing (and it's probably not getting a very wide audience anyway) but.. yep, not critical and there is a workaround | |
| 16:12:52 | melwitt | artom, dansmith: fyi I didn't change the meaning of the UNKNOWN status, it originally was only for host down and then when the down cells handling was added, it was used for that as well. my change was just a new policy rule to allow UNKNOWN status to be seen by non-admin if indicated by policy | |
| 16:13:35 | dansmith | melwitt: it was originally for cell down only, AFAIR | |
| 16:13:38 | melwitt | this is the logic for host down https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5339-L5351 | |
| 16:13:42 | dansmith | mark-host-down left the status in place | |
| 16:13:58 | dansmith | that's host status | |
| 16:14:09 | sean-k-mooney | dansmith: on a slightly different topic do you have interest in/ time to review the porvider.yaml series? just trying to figure out which redhat cores to bug as a reviewer when its ready. | |
| 16:14:11 | dansmith | he's talking about instance status right? | |
| 16:14:16 | melwitt | wasn't that the question? | |
| 16:14:20 | sean-k-mooney | melwitt: since your here same question ^ | |
| 16:14:26 | melwitt | oh, sorry. sigh | |
| 16:14:32 | dansmith | melwitt: vm status | |
| 16:14:40 | melwitt | well, either way I didn't change the meaning of vm status either | |
| 16:15:02 | dansmith | melwitt: I thought you were proposing the vm status change but okay | |
| 16:16:21 | melwitt | dansmith: I did but you explained why it wouldn't be a good idea and I agreed with your reasoning and updated the spec to stop proposing it. the spec was approved some time after that | |
| 16:16:35 | dansmith | sean-k-mooney: I dunno, I don't have a huge interest in reviewing that | |
| 16:16:41 | melwitt | *I did originally | |