| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 18:08:15 | openstackgerrit | Jack Ding proposed openstack/nova master: Add HPET timer support for x86 guests https://review.openstack.org/605902 | |
| 18:08:48 | sean-k-mooney | mnaser: right am in that case would you be better with a raw image instead of qcow if your always preallocating | |
| 18:09:57 | mnaser | sean-k-mooney: right, i'm thinking that might be the next path, raw files on disk | |
| 18:10:03 | mnaser | but then we lose a lot of qcow2 features | |
| 18:10:03 | sean-k-mooney | mnaser: if you are also supporting ceph or boot form volume raw can often be better too even if you are using more space for glance / image cache | |
| 18:10:27 | sean-k-mooney | mnaser: like live snapshot | |
| 18:10:36 | mnaser | yeah, a lot.. unfortunately | |
| 18:11:02 | sean-k-mooney | mnaser: i dont think anyone would object if you had a way to fix the bug but did not cause others | |
| 18:11:20 | mnaser | sean-k-mooney: yep.. its just that unfortunately there was no documentation as to why that was an issue with backing images | |
| 18:11:24 | mnaser | so thats what im trying to research | |
| 18:13:33 | sean-k-mooney | mnaser: its got to have something to do with the overlays that we create | |
| 18:13:50 | mnaser | yeah it looks like it's not really a possibility | |
| 18:13:51 | mnaser | :< | |
| 18:14:14 | sean-k-mooney | mnaser: mdbooth and kashyap should be able to confirm tomorow when they are back online | |
| 18:14:22 | mnaser | i'll wait to hear | |
| 18:14:40 | mnaser | now to find ways to benchmark this server | |
| 18:14:44 | mnaser | server/vm that is | |
| 18:15:23 | mnaser | http://paste.openstack.org/show/731425/ | |
| 18:16:16 | sean-k-mooney | is that a vm with 468G of ram | |
| 18:16:34 | sean-k-mooney | sorry 472 | |
| 18:17:00 | sean-k-mooney | i also like the insane amount of gpus | |
| 18:17:34 | sean-k-mooney | may i sugges you use it to play minecrat tootally how you should benchmark | |
| 18:17:46 | sean-k-mooney | mnaser: also i hear bitcoin is a thing :) | |
| 18:18:19 | mnaser | sean-k-mooney: aha, we're rolling out gpus and we have instances with 472G of ram, 48 (dedicated) threads, 1.8T of PCI-e NVMe storage.. | |
| 18:18:33 | sean-k-mooney | mnaser: actully on a serious note you are an operator of a cloud with vgpus correct | |
| 18:18:51 | mnaser | sean-k-mooney: no vgpu support, only dedicated gpus (as far as we've planned) | |
| 18:19:17 | mnaser | part of this is MAYBE seeing if we can get some vGPU CI.. if possible, but i hear there are some more complicated reasons why its not possible | |
| 18:19:19 | sean-k-mooney | ah well does the lack of vgpu numa affinity effect your decision to use vgpus or deploy gpus in the cloud in general | |
| 18:19:46 | sean-k-mooney | mnaser: actully it might be useing complicated trick | |
| 18:20:36 | sean-k-mooney | e.g. nested virt + q35 chipset + viommu + pci passthoug of phyical gpu PF to host vm | |
| 18:20:58 | mnaser | i think we're starting to roll things out by having dedicated gpus to see market demand for it (we've had some). unfortunately the other thing that's coming to mind is i'm thinking that users who need gpu levels of performance probably would want 100% of it | |
| 18:21:25 | mnaser | we can make nested virt available for gpu instances so maybe thats possible | |
| 18:21:33 | sean-k-mooney | mnaser: have you talked to bauzas about possible vgpu ci? | |
| 18:21:57 | mnaser | sean-k-mooney: we briefly talked about it.. dansmith mentioned concerns about iommu and stuff that's beyond my level of comprehension :) | |
| 18:22:24 | mnaser | but we plan to provide at least 1 or 2 instances to openstack CI *if* there's a use case that makes sense | |
| 18:22:39 | dansmith | mnaser: he said viommu, so if that's a thing now then maybe it's doable | |
| 18:22:58 | sean-k-mooney | dansmith: yes it is but we have not enabled it in nova yet | |
| 18:23:06 | sean-k-mooney | but its trival so we could | |
| 18:23:42 | sean-k-mooney | well its a flavor extraspec + xml generation and other crap but its not technical very hard to do we just have not done it yet | |
| 18:24:15 | mnaser | i'd be more than happy to provide 1 or 2 instances with a gpu | |
| 18:25:07 | sean-k-mooney | dansmith: i was added in libvrt 2.1 and qemu 3.4 https://libvirt.org/formatdomain.html#elementsIommu | |
| 18:26:44 | nicolasbock | <freenode_mri "nicolasbock: you'll likely need "> I had to also update `instances.node` but then the allocation was updated correctly | |
| 18:26:51 | sean-k-mooney | mnaser: thats very generous. it would certenly help if we could actully test vgpu the upstream ci even if it was an experimtal job that did not run on all patches | |
| 18:27:42 | mnaser | while i wrap things up here i can push up a patch to add 1 or 2. we'll probably do it with min-servers: 0 and max-servers: 2 to start with | |
| 18:28:43 | mriedem | efried: i've replied in https://review.openstack.org/#/c/606122/ | |
| 18:29:07 | efried | ack | |
| 18:29:40 | spatel | sean-k-mooney: currently i have "intel_iommu=on" in grub.conf, should i add "iommu=pt" too? | |
| 18:30:29 | sean-k-mooney | spatel: "iommu=pt" is not requried but advised | |
| 18:30:52 | efried | mriedem: +2 | |
| 18:30:58 | sean-k-mooney | spatel: this is my cmdline on my sriov systems BOOT_IMAGE=/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=2cca5edf-cbcc-4f0d-91df-df438bbd56c5 ro crashkernel=auto rhgb quiet intel_iommu=on iommu=pt pci=assign-busses,realloc | |
| 18:30:58 | spatel | will add that :) | |
| 18:31:19 | spatel | are you using SR-IOV? | |
| 18:31:25 | spatel | or DPDK? | |
| 18:31:37 | mriedem | efried: thanks | |
| 18:31:44 | mriedem | lazy-load can be a cruel mistress | |
| 18:31:49 | sean-k-mooney | spatel: pci=assign-busses,realloc is to work around some hardware bugs where my bios does not allocate enough iommu space | |
| 18:31:52 | efried | srsly | |
| 18:32:01 | spatel | nice! | |
| 18:32:08 | sean-k-mooney | spatel: iommu=pt is need for dpdk but not sriov | |
| 18:32:17 | spatel | oh! make sense | |
| 18:32:28 | sean-k-mooney | i enable it always so i can deploy both and swap between them | |
| 18:32:48 | spatel | sean-k-mooney: i have created new flavor (15 vCPU / 14G memory ) and i got this error | |
| 18:32:56 | spatel | ERROR (BadRequest): Instance CPUs and/or memory cannot be evenly distributed across instance NUMA nodes. Explicit assignment of CPUs and memory to nodes is required (HTTP 400) (Request-ID: req-400663e1-75d1-4bbc-a06b-07dcfd845be6) | |
| 18:33:19 | spatel | This is what i have in flavor hw:cpu_policy='dedicated', hw:numa_nodes='2' | |
| 18:33:54 | sean-k-mooney | spatel: yes the error could be improved. the vcpus needs to be devisable by the number of numa nodes othere wise you have to tell us how many cpus to put on each numa node | |
| 18:34:51 | sean-k-mooney | spatel: so i would jsut set it to 14 vcpus and 14G memory | |
| 18:34:58 | spatel | cool!! | |
| 18:35:00 | spatel | doing it | |
| 18:35:44 | sean-k-mooney | spatel: since you are optimising your flavors and given your usecase i would also recomment enableing hugepage memroy for the vm | |
| 18:36:29 | sean-k-mooney | it will give you a 30-40% performacne boost in many workloads but require you to allocate hugepages on the host first via the kernel command line ideally | |
| 18:36:49 | spatel | I have this setting in grub "hugepagesz=2M hugepages=2048 transparent_hugepage=never" | |
| 18:37:30 | sean-k-mooney | ah cool that will only allcoate 4G of hugepates form the 32 you have total. | |
| 18:37:40 | spatel | one more question i have 32G memory so what number should be good for number of pages? | |
| 18:37:57 | spatel | yes i have 32G memory | |
| 18:38:28 | spatel | i heard 1G is better for hugepage | |
| 18:38:33 | sean-k-mooney | haha i was getting to that next. :) i would recommend between 24-28G of hugepages leave 6-8 for the host | |
| 18:38:55 | sean-k-mooney | spatel: it depends for some workloads yes for most it does not matter | |
| 18:39:21 | sean-k-mooney | hugepages cannot be subdevided so if you use 1G hugepges the ram in you flavor must be a multiple of 1G | |
| 18:39:22 | spatel | my application doesn't need lots of memory because its RTP traffic voip | |
| 18:39:50 | spatel | hmm! make sense | |
| 18:40:00 | sean-k-mooney | spatel: in your case i doubt you will see a difference and 2MB hugepages will give you more granularity | |
| 18:40:15 | spatel | lets stick to 2M then :) | |
| 18:40:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add post-test hook for testing evacuate https://review.openstack.org/602174 | |
| 18:40:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397 | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400 | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix InstanceNotFound during _destroy_evacuated_instances https://review.openstack.org/606122 | |
| 18:41:00 | openstack | bug 1794996 in OpenStack Compute (nova) "_destroy_evacuated_instances fails and kills n-cpu startup if lazy-loading flavor on a deleted instance" [High,In progress] https://launchpad.net/bugs/1794996 - Assigned to Matt Riedemann (mriedem) | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug 1794996 https://review.openstack.org/606106 | |
| 18:41:26 | spatel | sean-k-mooney: should i use this? hugepagesz=2M hugepages=15360 | |
| 18:41:41 | spatel | it will give 30G | |
| 18:41:56 | spatel | let me try to make it 28G | |
| 18:42:03 | spatel | keep 4G for OS | |
| 18:42:27 | sean-k-mooney | the hugepage memory will not be availabel to normal os process so 2MB is likely too tight for a compute node | |
| 18:43:28 | sean-k-mooney | 4G should be ok but used to give ^G as my safty margin that said i did not need that much of a margin | |
| 18:44:17 | spatel | In that case let me give 8G to OS (keep 24G for VM) | |
| 18:44:32 | sean-k-mooney | spatel: i would set it to 12 288 | |
| 18:44:37 | sean-k-mooney | which is 24G | |
| 18:45:03 | spatel | hugepagesz=2M hugepages=12288 - DONE! going to reboot compute node | |
| 18:45:31 | spatel | Do you use isolcpus= CPUAffinity ? | |
| 18:45:47 | spatel | I was reading about that not sure i need to worry about that or not | |