| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-29 | |||
| 19:07:41 | donnyd | yep | |
| 19:07:49 | sean-k-mooney | if you added hw:numa_nodes=2 to those it woudl work too | |
| 19:08:06 | sean-k-mooney | that would be very close to how the intel ci used to run | |
| 19:08:12 | donnyd | what do you mean? | |
| 19:08:20 | donnyd | you mean in project-config | |
| 19:08:28 | sean-k-mooney | no in your nova | |
| 19:08:32 | donnyd | oh | |
| 19:08:45 | donnyd | ok, so I just have to add that meta-data to the flavor | |
| 19:09:11 | sean-k-mooney | openstack flavor set "hw:numa_nodes=2" "8cpu-16GBram-80GBdisk" i think | |
| 19:09:39 | sean-k-mooney | not quite | |
| 19:09:53 | donnyd | maybe --property | |
| 19:10:02 | sean-k-mooney | openstack flavor set --property hw:numa_nodes=2 "8cpu-16GBram-80GBdisk" | |
| 19:10:09 | sean-k-mooney | yep | |
| 19:11:08 | donnyd | ok | |
| 19:11:10 | donnyd | done | |
| 19:11:28 | donnyd | but don't I have to setup something on the hypervisors or OS for NUMA to work? | |
| 19:11:39 | sean-k-mooney | nope | |
| 19:11:50 | donnyd | Ok, well its there | |
| 19:11:55 | sean-k-mooney | you have kvm enabeld correct | |
| 19:12:01 | sean-k-mooney | and netsted virt? | |
| 19:12:10 | donnyd | no... .def not nested | |
| 19:12:15 | sean-k-mooney | kvm is all that is required on the host | |
| 19:12:27 | donnyd | but i can enable nested-virt | |
| 19:12:49 | donnyd | not sure how that would tinker with other jobs | |
| 19:12:54 | sean-k-mooney | ok nested virt woudl be need for use to use kvm in the guest which we would need to test pinning and numa currently | |
| 19:13:08 | sean-k-mooney | the standard jobs hardcode qemu | |
| 19:13:12 | sean-k-mooney | so they wont use it | |
| 19:13:18 | sean-k-mooney | so they should be fine | |
| 19:13:21 | donnyd | ok | |
| 19:13:32 | sean-k-mooney | what os are you running by they way | |
| 19:13:36 | sean-k-mooney | on the host | |
| 19:13:50 | donnyd | It will take a bit to drain each hypervisor so I can unload and reload kvm with nested virt | |
| 19:13:55 | donnyd | ubuntu 18.04 | |
| 19:14:37 | sean-k-mooney | nested virt used to be broken on ubuntu 18.04 but i think they have fixed in in more recent kernels | |
| 19:14:43 | sean-k-mooney | there is no rush | |
| 19:15:28 | sean-k-mooney | i can start getting the job working and then when ever you get a chance to enable nested virt we can do some more testing. | |
| 19:15:33 | donnyd | i stand corrected - nested virt is already enabled too | |
| 19:15:44 | sean-k-mooney | cool | |
| 19:16:15 | donnyd | kernel is 4.15.0-58-generic | |
| 19:16:50 | sean-k-mooney | in that case if i pivot https://review.opendev.org/#/c/652197/ to use nested virt i can add a muti numa version. that targets FN and reuns as a perodic | |
| 19:17:58 | sean-k-mooney | donnyd: am i need to go do a few tings before i finish up for the day | |
| 19:18:05 | donnyd | ok | |
| 19:18:23 | sean-k-mooney | but ill touch base on monday or tueseday next week when i get a chance to look that this again | |
| 19:19:14 | donnyd | Ok, lmk how else i can help | |
| 19:19:28 | donnyd | In my BIOS i have node-interleaving turned off | |
| 19:19:36 | donnyd | is that required for NUMA? | |
| 19:19:40 | sean-k-mooney | no | |
| 19:19:45 | sean-k-mooney | infact if you trun it on | |
| 19:19:49 | sean-k-mooney | it disable numa | |
| 19:20:11 | donnyd | are there any kernel params i can check to see if it will work | |
| 19:20:58 | sean-k-mooney | ls /sys/bus/node/devices/ | |
| 19:21:07 | sean-k-mooney | is ther more then one node | |
| 19:21:34 | sean-k-mooney | if so it should work. you can always try booting a vm with that flavor too | |
| 19:21:41 | sean-k-mooney | if it works then its all good | |
| 19:23:02 | sean-k-mooney | on a numa host it will look like this | |
| 19:23:04 | sean-k-mooney | stack@cloud-3:~/devstack$ ls /sys/bus/node/devices/ | |
| 19:23:06 | sean-k-mooney | node0 node1 | |
| 19:23:21 | sean-k-mooney | or in your case you should have node 0-3 | |
| 19:31:18 | openstackgerrit | Merged openstack/nova master: rt: soften warning case in _remove_deleted_instances_allocations https://review.opendev.org/673873 | |
| 19:33:11 | donnyd | sean-k-mooney: I will check it out and report back | |
| 19:35:22 | donnyd | node0 node1 node2 node3 | |
| 19:35:22 | donnyd | ubuntu@compute-1:~$ sudo ls /sys/bus/node/devices/ | |
| 19:37:39 | donnyd | woot woot | |
| 19:38:05 | sean-k-mooney | yep that all looks fine | |
| 19:39:08 | donnyd | another plus is that flavor also already has a bit more for memory, so that may be helpful on the nested virt side | |
| 19:39:32 | sean-k-mooney | it will be helpful for hugepage testing in the vm | |
| 19:40:05 | sean-k-mooney | because it means its less of a blancing act between the memroy needed for openstack vs the nested vms | |
| 19:40:25 | donnyd | i don't have hugepages enabled | |
| 19:40:33 | sean-k-mooney | you dont need them on the host | |
| 19:40:41 | sean-k-mooney | if you look at https://review.opendev.org/#/c/652197/20/playbooks/nfv/nfv.yaml | |
| 19:41:00 | sean-k-mooney | you see in my curent nfv job i ame allocating only a limited amount | |
| 19:41:01 | donnyd | do i look for the same parameters inside the vm? | |
| 19:41:24 | sean-k-mooney | how do you mean | |
| 19:41:39 | sean-k-mooney | do you mean the cpu flags | |
| 19:41:52 | donnyd | looking for numa nodes inside the instance launched with that flavor | |
| 19:42:00 | donnyd | the instance created fine | |
| 19:42:09 | donnyd | but how do i tell if its right | |
| 19:42:15 | sean-k-mooney | oh ya if you run the sam command in the vm you shoudl see 2 | |
| 19:42:24 | prometheanfire | how's the lxml issue coming? | |
| 19:42:41 | sean-k-mooney | i need to rework it to use the helper function | |
| 19:42:44 | donnyd | there is no /sys/bus/node dir | |
| 19:42:45 | sean-k-mooney | but its fixed | |
| 19:43:05 | sean-k-mooney | really? | |
| 19:43:09 | donnyd | no | |
| 19:43:24 | sean-k-mooney | what kernel/os is the vm using | |
| 19:43:31 | donnyd | i am going to try again with a different image and maybe not use one i am developing on | |
| 19:43:45 | sean-k-mooney | prometheanfire: https://review.opendev.org/#/c/678964/ | |
| 19:44:14 | sean-k-mooney | ill respin it sortly to use self.assertXmlEqual | |
| 19:44:25 | sean-k-mooney | but once thats done it should be good to go. | |
| 19:44:33 | sean-k-mooney | prometheanfire: can you unabandon your change by the ay | |
| 19:44:39 | sean-k-mooney | *way | |
| 19:44:49 | sean-k-mooney | prometheanfire: https://review.opendev.org/#/c/673848/ | |
| 19:45:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Test heal port allocations in nova-next https://review.opendev.org/669879 | |
| 19:45:55 | prometheanfire | the lxml change? | |
| 19:45:58 | donnyd | yea, i tried with a different image and its good to go | |
| 19:46:10 | prometheanfire | done | |
| 19:46:21 | sean-k-mooney | donnyd: cool can you provide the output of lcpu from the guest too | |
| 19:46:25 | prometheanfire | should I update it to 4.4.1? | |
| 19:46:35 | sean-k-mooney | i just want to check the flags | |
| 19:46:50 | sean-k-mooney | prometheanfire: i used the latest version on pypi for my testing | |