| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-29 | |||
| 18:49:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 18:49:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 18:49:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 18:49:23 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor https://review.opendev.org/678887 | |
| 18:49:23 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 18:49:34 | donnyd | any good docs to point me to for setup? | |
| 18:50:23 | sean-k-mooney | donnyd: i can send you the command or document them in an etherpad if you like | |
| 18:50:46 | sean-k-mooney | over all there is not too much that you need too do | |
| 18:50:54 | sean-k-mooney | we jus tneed to know the name of the flavor | |
| 18:50:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Rename the nova-grenade-live-migration job to nova-grenade-multinode https://review.opendev.org/678841 | |
| 18:50:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cold migrate and resize to nova-grenade-multinode https://review.opendev.org/679210 | |
| 18:51:00 | sean-k-mooney | and you need to add a few extra specs | |
| 18:51:06 | mriedem | artom: ha just did ours | |
| 18:51:34 | sean-k-mooney | and then 2 patches 1 for the project-config repo to add the new lable to nodepool and 1 to nova to add the perodic job | |
| 18:51:35 | mriedem | unbelievably, the teacher didn't comment on my new fresh haircut | |
| 18:51:42 | artom | mriedem, D: | |
| 18:51:56 | mriedem | "oh maya nice to meet....WHO IS THIS HANDSOME DEVIL?!" | |
| 18:51:58 | artom | The nerve of some people. | |
| 18:52:21 | artom | Seriously. The female version of https://www.youtube.com/watch?v=4pXfHLUlZf4. | |
| 18:52:23 | artom | (NFSW) | |
| 18:52:48 | sean-k-mooney | see now there is plasible deniablity | |
| 18:54:18 | donnyd | sure that would work great sean-k-mooney | |
| 18:54:33 | donnyd | I can name things however you like | |
| 18:55:06 | sean-k-mooney | we just need to know the name to put it in the project-config patch so it does not really matter | |
| 19:03:20 | donnyd | have to be something like ubuntu-bionic-fn-numa | |
| 19:03:28 | donnyd | centos-7-fn-numa | |
| 19:03:39 | sean-k-mooney | yep | |
| 19:03:45 | donnyd | something like that.. .whatever you come up with, I will just put it in FN | |
| 19:03:49 | sean-k-mooney | although the ubunutu one is proably enough | |
| 19:04:18 | donnyd | why not just use the label that is already there and i can fixup the flavor | |
| 19:04:26 | sean-k-mooney | ubuntu-bionic-fn-numa is fine the the lable and then we need a flavor name that can be numa or whatever | |
| 19:04:27 | donnyd | ubuntu-bionic-expanded | |
| 19:04:33 | donnyd | ok | |
| 19:05:01 | donnyd | i don't think anyone has ever scheduled a job against it | |
| 19:05:02 | sean-k-mooney | donnyd: we could but other jobs are not expecting muliptle numa nodes | |
| 19:05:21 | sean-k-mooney | well you also have the standard lables right | |
| 19:05:26 | donnyd | other jobs don't run on it right now | |
| 19:05:33 | sean-k-mooney | on FN | |
| 19:05:43 | donnyd | i created a label that had 16G of memory | |
| 19:05:45 | donnyd | yes | |
| 19:05:51 | sean-k-mooney | oh ok | |
| 19:05:57 | donnyd | look in nl02 | |
| 19:06:19 | donnyd | i already have a custom label that is completely unused | |
| 19:07:28 | sean-k-mooney | these ones https://github.com/openstack/project-config/blob/master/nodepool/nl02.openstack.org.yaml#L330 | |
| 19:07:41 | donnyd | yep | |
| 19:07:49 | sean-k-mooney | if you added hw:numa_nodes=2 to those it woudl work too | |
| 19:08:06 | sean-k-mooney | that would be very close to how the intel ci used to run | |
| 19:08:12 | donnyd | what do you mean? | |
| 19:08:20 | donnyd | you mean in project-config | |
| 19:08:28 | sean-k-mooney | no in your nova | |
| 19:08:32 | donnyd | oh | |
| 19:08:45 | donnyd | ok, so I just have to add that meta-data to the flavor | |
| 19:09:11 | sean-k-mooney | openstack flavor set "hw:numa_nodes=2" "8cpu-16GBram-80GBdisk" i think | |
| 19:09:39 | sean-k-mooney | not quite | |
| 19:09:53 | donnyd | maybe --property | |
| 19:10:02 | sean-k-mooney | openstack flavor set --property hw:numa_nodes=2 "8cpu-16GBram-80GBdisk" | |
| 19:10:09 | sean-k-mooney | yep | |
| 19:11:08 | donnyd | ok | |
| 19:11:10 | donnyd | done | |
| 19:11:28 | donnyd | but don't I have to setup something on the hypervisors or OS for NUMA to work? | |
| 19:11:39 | sean-k-mooney | nope | |
| 19:11:50 | donnyd | Ok, well its there | |
| 19:11:55 | sean-k-mooney | you have kvm enabeld correct | |
| 19:12:01 | sean-k-mooney | and netsted virt? | |
| 19:12:10 | donnyd | no... .def not nested | |
| 19:12:15 | sean-k-mooney | kvm is all that is required on the host | |
| 19:12:27 | donnyd | but i can enable nested-virt | |
| 19:12:49 | donnyd | not sure how that would tinker with other jobs | |
| 19:12:54 | sean-k-mooney | ok nested virt woudl be need for use to use kvm in the guest which we would need to test pinning and numa currently | |
| 19:13:08 | sean-k-mooney | the standard jobs hardcode qemu | |
| 19:13:12 | sean-k-mooney | so they wont use it | |
| 19:13:18 | sean-k-mooney | so they should be fine | |
| 19:13:21 | donnyd | ok | |
| 19:13:32 | sean-k-mooney | what os are you running by they way | |
| 19:13:36 | sean-k-mooney | on the host | |
| 19:13:50 | donnyd | It will take a bit to drain each hypervisor so I can unload and reload kvm with nested virt | |
| 19:13:55 | donnyd | ubuntu 18.04 | |
| 19:14:37 | sean-k-mooney | nested virt used to be broken on ubuntu 18.04 but i think they have fixed in in more recent kernels | |
| 19:14:43 | sean-k-mooney | there is no rush | |
| 19:15:28 | sean-k-mooney | i can start getting the job working and then when ever you get a chance to enable nested virt we can do some more testing. | |
| 19:15:33 | donnyd | i stand corrected - nested virt is already enabled too | |
| 19:15:44 | sean-k-mooney | cool | |
| 19:16:15 | donnyd | kernel is 4.15.0-58-generic | |
| 19:16:50 | sean-k-mooney | in that case if i pivot https://review.opendev.org/#/c/652197/ to use nested virt i can add a muti numa version. that targets FN and reuns as a perodic | |
| 19:17:58 | sean-k-mooney | donnyd: am i need to go do a few tings before i finish up for the day | |
| 19:18:05 | donnyd | ok | |
| 19:18:23 | sean-k-mooney | but ill touch base on monday or tueseday next week when i get a chance to look that this again | |
| 19:19:14 | donnyd | Ok, lmk how else i can help | |
| 19:19:28 | donnyd | In my BIOS i have node-interleaving turned off | |
| 19:19:36 | donnyd | is that required for NUMA? | |
| 19:19:40 | sean-k-mooney | no | |
| 19:19:45 | sean-k-mooney | infact if you trun it on | |
| 19:19:49 | sean-k-mooney | it disable numa | |
| 19:20:11 | donnyd | are there any kernel params i can check to see if it will work | |
| 19:20:58 | sean-k-mooney | ls /sys/bus/node/devices/ | |
| 19:21:07 | sean-k-mooney | is ther more then one node | |
| 19:21:34 | sean-k-mooney | if so it should work. you can always try booting a vm with that flavor too | |
| 19:21:41 | sean-k-mooney | if it works then its all good | |
| 19:23:02 | sean-k-mooney | on a numa host it will look like this | |
| 19:23:04 | sean-k-mooney | stack@cloud-3:~/devstack$ ls /sys/bus/node/devices/ | |