| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 18:28:43 | mriedem | efried: i've replied in https://review.openstack.org/#/c/606122/ | |
| 18:29:07 | efried | ack | |
| 18:29:40 | spatel | sean-k-mooney: currently i have "intel_iommu=on" in grub.conf, should i add "iommu=pt" too? | |
| 18:30:29 | sean-k-mooney | spatel: "iommu=pt" is not requried but advised | |
| 18:30:52 | efried | mriedem: +2 | |
| 18:30:58 | spatel | will add that :) | |
| 18:30:58 | sean-k-mooney | spatel: this is my cmdline on my sriov systems BOOT_IMAGE=/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=UUID=2cca5edf-cbcc-4f0d-91df-df438bbd56c5 ro crashkernel=auto rhgb quiet intel_iommu=on iommu=pt pci=assign-busses,realloc | |
| 18:31:19 | spatel | are you using SR-IOV? | |
| 18:31:25 | spatel | or DPDK? | |
| 18:31:37 | mriedem | efried: thanks | |
| 18:31:44 | mriedem | lazy-load can be a cruel mistress | |
| 18:31:49 | sean-k-mooney | spatel: pci=assign-busses,realloc is to work around some hardware bugs where my bios does not allocate enough iommu space | |
| 18:31:52 | efried | srsly | |
| 18:32:01 | spatel | nice! | |
| 18:32:08 | sean-k-mooney | spatel: iommu=pt is need for dpdk but not sriov | |
| 18:32:17 | spatel | oh! make sense | |
| 18:32:28 | sean-k-mooney | i enable it always so i can deploy both and swap between them | |
| 18:32:48 | spatel | sean-k-mooney: i have created new flavor (15 vCPU / 14G memory ) and i got this error | |
| 18:32:56 | spatel | ERROR (BadRequest): Instance CPUs and/or memory cannot be evenly distributed across instance NUMA nodes. Explicit assignment of CPUs and memory to nodes is required (HTTP 400) (Request-ID: req-400663e1-75d1-4bbc-a06b-07dcfd845be6) | |
| 18:33:19 | spatel | This is what i have in flavor hw:cpu_policy='dedicated', hw:numa_nodes='2' | |
| 18:33:54 | sean-k-mooney | spatel: yes the error could be improved. the vcpus needs to be devisable by the number of numa nodes othere wise you have to tell us how many cpus to put on each numa node | |
| 18:34:51 | sean-k-mooney | spatel: so i would jsut set it to 14 vcpus and 14G memory | |
| 18:34:58 | spatel | cool!! | |
| 18:35:00 | spatel | doing it | |
| 18:35:44 | sean-k-mooney | spatel: since you are optimising your flavors and given your usecase i would also recomment enableing hugepage memroy for the vm | |
| 18:36:29 | sean-k-mooney | it will give you a 30-40% performacne boost in many workloads but require you to allocate hugepages on the host first via the kernel command line ideally | |
| 18:36:49 | spatel | I have this setting in grub "hugepagesz=2M hugepages=2048 transparent_hugepage=never" | |
| 18:37:30 | sean-k-mooney | ah cool that will only allcoate 4G of hugepates form the 32 you have total. | |
| 18:37:40 | spatel | one more question i have 32G memory so what number should be good for number of pages? | |
| 18:37:57 | spatel | yes i have 32G memory | |
| 18:38:28 | spatel | i heard 1G is better for hugepage | |
| 18:38:33 | sean-k-mooney | haha i was getting to that next. :) i would recommend between 24-28G of hugepages leave 6-8 for the host | |
| 18:38:55 | sean-k-mooney | spatel: it depends for some workloads yes for most it does not matter | |
| 18:39:21 | sean-k-mooney | hugepages cannot be subdevided so if you use 1G hugepges the ram in you flavor must be a multiple of 1G | |
| 18:39:22 | spatel | my application doesn't need lots of memory because its RTP traffic voip | |
| 18:39:50 | spatel | hmm! make sense | |
| 18:40:00 | sean-k-mooney | spatel: in your case i doubt you will see a difference and 2MB hugepages will give you more granularity | |
| 18:40:15 | spatel | lets stick to 2M then :) | |
| 18:40:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add post-test hook for testing evacuate https://review.openstack.org/602174 | |
| 18:40:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397 | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug 1794996 https://review.openstack.org/606106 | |
| 18:41:00 | openstack | bug 1794996 in OpenStack Compute (nova) "_destroy_evacuated_instances fails and kills n-cpu startup if lazy-loading flavor on a deleted instance" [High,In progress] https://launchpad.net/bugs/1794996 - Assigned to Matt Riedemann (mriedem) | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix InstanceNotFound during _destroy_evacuated_instances https://review.openstack.org/606122 | |
| 18:41:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400 | |
| 18:41:26 | spatel | sean-k-mooney: should i use this? hugepagesz=2M hugepages=15360 | |
| 18:41:41 | spatel | it will give 30G | |
| 18:41:56 | spatel | let me try to make it 28G | |
| 18:42:03 | spatel | keep 4G for OS | |
| 18:42:27 | sean-k-mooney | the hugepage memory will not be availabel to normal os process so 2MB is likely too tight for a compute node | |
| 18:43:28 | sean-k-mooney | 4G should be ok but used to give ^G as my safty margin that said i did not need that much of a margin | |
| 18:44:17 | spatel | In that case let me give 8G to OS (keep 24G for VM) | |
| 18:44:32 | sean-k-mooney | spatel: i would set it to 12 288 | |
| 18:44:37 | sean-k-mooney | which is 24G | |
| 18:45:03 | spatel | hugepagesz=2M hugepages=12288 - DONE! going to reboot compute node | |
| 18:45:31 | spatel | Do you use isolcpus= CPUAffinity ? | |
| 18:45:47 | spatel | I was reading about that not sure i need to worry about that or not | |
| 18:46:03 | sean-k-mooney | i would then also reduce the max size vms to 10 or 12 GB ram for your largset flavor so you can alway boot at least 2 of them | |
| 18:46:29 | sean-k-mooney | isolcpus is not the same as cpuaffintiy | |
| 18:47:00 | sean-k-mooney | i generally avoid isolcpus= it is a rather large hammer to reach for | |
| 18:47:16 | mriedem | bauzas: i've -2ed https://review.openstack.org/#/c/599208/ as we discussed yesterday | |
| 18:47:17 | sean-k-mooney | it should only be used for realtime instances even then its tricky to use correctly | |
| 18:47:34 | spatel | ok! got it | |
| 18:47:45 | sean-k-mooney | spatel: generally i would only suggest usign it to isolage cores allcoated to ovs-dpdk if you chose to depoly it | |
| 18:48:53 | sean-k-mooney | spatel: dont get me wronge isolcpus= has a place but its only somting i reach for when i have no other options left and i really really need it | |
| 18:49:07 | spatel | I will soon deploy dpdk (believe me) | |
| 18:49:57 | mnaser | sean-k-mooney: https://review.openstack.org/#/c/607686/ .. ill push up a patch to test things out when possible (or at least something to confirm its working) | |
| 18:50:01 | spatel | in flavor i should set hw:mem_page_size='2048' right ? | |
| 18:50:07 | mnaser | so maybe if you want to start figuring out nova dependencies | |
| 18:52:41 | dansmith | mriedem: melwitt tssurya: cells meeting today? I have an appointment the hour before, but I will probably be back in time | |
| 18:52:51 | sean-k-mooney | spatel: you can but i prefer seting hw:mem_page_size=large | |
| 18:53:05 | sean-k-mooney | spatel: that will work with both 1G and 2MB hugepages | |
| 18:53:15 | spatel | done! let me do that | |
| 18:53:21 | dansmith | side note, mriedem melwitt: This is easy early utility stuff we can merge in front of the down cell stuff: https://review.openstack.org/#/c/594947/ | |
| 18:54:01 | tssurya | dansmith: the most important question I had was the best way to get the "type" of exception from the utility ^ | |
| 18:54:22 | dansmith | type? | |
| 18:54:25 | tssurya | we could also do it during the meeting if others also have topics | |
| 18:54:43 | mriedem | dansmith: i was holding off on that one until i knew what was going on further in the series | |
| 18:54:52 | tssurya | yea for instance a TimeOut/DBonnectionError expception versus InstanceNotFound exception | |
| 18:55:09 | tssurya | as of now we always return the "raised_exception_sentinel" which is not that useful | |
| 18:55:27 | tssurya | because based on the type of exception we have to handle it differently | |
| 18:55:31 | nicolasbock | Fixing the migration is more difficult it seems: I successfully updated the DB with the correct hypervisor and `server show` was now showing the correct hypervisor information | |
| 18:55:37 | mriedem | please hold, i have to sell something to a craigslist weirdo real quick | |
| 18:55:38 | dansmith | tssurya: by timeout you mean an rpc timeout, not the did_not_respond_sentinel I assume? | |
| 18:55:46 | nicolasbock | I ran `server migrate` which failed with `[Errno 2] No such file or directory: '/var/lib/nova/instances/2aa3a324-bf22-4e0c-912a-d7c52f59f1fd/disk` | |
| 18:55:58 | nicolasbock | So the disk didn't make it in the first migration | |
| 18:56:04 | nicolasbock | I verified that the disk is still on the old host | |
| 18:56:18 | sean-k-mooney | mnaser: that spec would allow testing quite alot of featue espcially if it supproted nested virt | |
| 18:56:29 | nicolasbock | Since I am in the middle of open heart surgery anyway I figured I just rsync the disk to the current hypervisor | |
| 18:56:35 | tssurya | dansmith: TimeOut was just an example, my main problem is to filter the "InstanceNotFOund" from others for nvoa show | |
| 18:56:37 | tssurya | nvoa show* | |
| 18:56:41 | tssurya | nova show* | |
| 18:56:41 | nicolasbock | So that worked | |
| 18:56:44 | mnaser | sean-k-mooney: these vms have nested virt | |
| 18:56:57 | nicolasbock | However, migrate is now refusing to migrate since the VM is in an ERROR state | |
| 18:57:08 | dansmith | tssurya: yeah | |
| 18:57:15 | mnaser | nicolasbock: nova reset-state --active | |
| 18:57:16 | nicolasbock | I can't `nova reset-state` either, it says `Reset state for server 2aa3a324-bf22-4e0c-912a-d7c52f59f1fd succeeded; new state is error` | |
| 18:57:22 | tssurya | as of now, when we get the InstanceNotFound, utility hides this returns the sentinel, I try to go and make a minimal construct when I shouldn't be | |
| 18:57:23 | nicolasbock | which isn't all that helpful :( | |
| 18:57:30 | dansmith | tssurya: probably have to get away from the sentinel object I guess | |