| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-23 | |||
| 14:25:17 | sean-k-mooney | it is not ment to the cpus multiplied by the allocation ration | |
| 14:25:25 | stephenfin | That's wrong | |
| 14:25:34 | sean-k-mooney | no that is how it was ment to work | |
| 14:25:38 | sean-k-mooney | and how it has always worked | |
| 14:25:48 | sean-k-mooney | it has never used the allocation ration | |
| 14:25:48 | stephenfin | Create a load of guests with 'hw:numa_nodes=1' on two node host with 6 CPUs per node | |
| 14:26:00 | stephenfin | How many host CPUs are free if you boot those guests? | |
| 14:26:29 | stephenfin | The os-hypervisors information makes no sense in a world of overcommit | |
| 14:26:43 | stephenfin | which is why we're planning to drop it in Wallaby | |
| 14:26:55 | sean-k-mooney | stephenfin: recyclehero is try ing to boot a 10 vcpu guest on an 8 cpu host | |
| 14:27:01 | sean-k-mooney | which we have never supported | |
| 14:27:19 | sean-k-mooney | because 2 gust cpus cant consome the same host cpu enven if they are floating | |
| 14:27:23 | stephenfin | sean-k-mooney: yes, that's true | |
| 14:28:11 | stephenfin | sean-k-mooney: but we have 'max_unit' to track that | |
| 14:29:01 | stephenfin | recyclehero: Oh, that was an interesting point on the output of e.g. 'openstack resource provider inventory list' | |
| 14:29:18 | stephenfin | I'd have expected the total column to show the total inclusive of allocation_ratio | |
| 14:29:41 | stephenfin | but it doesn't; it shows the total before the allocation ratio is applied | |
| 14:29:41 | sean-k-mooney | no | |
| 14:29:50 | stephenfin | so you have to do that maths yourself :) | |
| 14:29:57 | sean-k-mooney | yes | |
| 14:30:06 | stephenfin | sean-k-mooney: make up your mind, man! | |
| 14:30:07 | stephenfin | :P | |
| 14:30:09 | sean-k-mooney | so that the different values are not depenent on each other | |
| 14:30:17 | sean-k-mooney | you have to do the math yourself | |
| 14:30:34 | sean-k-mooney | multipleying total by allocation ratio - reserved | |
| 14:32:13 | stephenfin | sean-k-mooney: as for that earlier issue, this entire stack is failing with it :( https://review.opendev.org/#/q/topic:bp/compact-db-migrations-wallaby | |
| 14:32:29 | stephenfin | I don't think it's something I've done wrong, but I'm waiting to see if another patch fails first | |
| 14:32:55 | sean-k-mooney | stephenfin: the import issue if its the same one it due to an oom issue causeing the import to fail | |
| 14:33:01 | sean-k-mooney | if not then rbd is not isntalled | |
| 14:33:12 | sean-k-mooney | because ceph-common is not installed | |
| 14:33:20 | stephenfin | oh, no, look down | |
| 14:33:22 | sean-k-mooney | i didnt look too closelon | |
| 14:33:32 | stephenfin | ignore the rbd thing | |
| 14:33:37 | stephenfin | that's lyarwood's work | |
| 14:34:59 | sean-k-mooney | oh line 861 | |
| 14:35:18 | sean-k-mooney | it did not scole down fro some reason | |
| 14:35:36 | sean-k-mooney | TypeError: Parameterized generics cannot be used with class or instance checks | |
| 14:35:42 | stephenfin | Yeah | |
| 14:35:47 | stephenfin | I've never seen that before | |
| 14:35:52 | stephenfin | it's buried in eventlet too | |
| 14:36:26 | sean-k-mooney | so its coming from the ytpeing model | |
| 14:37:15 | stephenfin | mypy hasn't been updated recently though, and typing is stdlib now | |
| 14:37:18 | sean-k-mooney | form the libvirt treadpool proxy | |
| 14:37:31 | sean-k-mooney | stephenfin: yes it si from the standardlib | |
| 14:37:48 | stephenfin | crap, new eventlet release yesterday | |
| 14:38:13 | sean-k-mooney | to fix dnspython support? | |
| 14:38:23 | sean-k-mooney | we likely need to cap it untill we figure this out | |
| 14:39:39 | stephenfin | gibi: Looks like we might have a problem with the new release of eventlet ^ | |
| 14:39:50 | stephenfin | https://zuul.opendev.org/t/openstack/build/c85b0734fb524eefb67449105528b0a2/log/logs/screen-n-cpu.txt#861 | |
| 14:40:18 | sean-k-mooney | https://github.com/eventlet/eventlet/commit/9d731373e2325588a33229b425100ad538eed766 | |
| 14:40:23 | sean-k-mooney | this is apparently what changed | |
| 14:42:04 | sean-k-mooney | oh they fixed my ssl bug i think https://github.com/eventlet/eventlet/commit/7b1aa58d77550caabba3ca2e0b1bb79626c83d9c | |
| 14:42:35 | sean-k-mooney | oh no its a different one | |
| 14:43:55 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/632 was our issue | |
| 14:44:27 | stephenfin | yeah, the last working build for https://review.opendev.org/#/c/758396/ used eventlet===0.26.1 | |
| 14:44:35 | stephenfin | https://zuul.opendev.org/t/openstack/build/bf1f2b730848408b99c61e82068eb73b/log/job-output.txt#6355 | |
| 14:44:46 | stephenfin | the failing one uses eventlet===0.28.0 | |
| 14:44:50 | stephenfin | https://zuul.opendev.org/t/openstack/build/00e3a2793a1040b29b944b944c9da795/log/job-output.txt | |
| 14:44:59 | stephenfin | so eventlet has borked us \o/ | |
| 14:45:04 | sean-k-mooney | they have done a buch of release in the last month | |
| 14:46:02 | sean-k-mooney | i guess need to file a bug | |
| 14:52:43 | stephenfin | Hmm, we also bumped our version of mypy, greenlet and libvirt-python recently, so it could be those too. This will be fun | |
| 15:24:08 | recyclehero | sean-k-mooney stephenfin thanks guys. I really appreciate ur help. I guess it was okay after we did the 'None' patch in OSA but the data displayed was confusing. | |
| 15:24:51 | sean-k-mooney | it was a design choice that they made very early on in horizon | |
| 15:25:10 | sean-k-mooney | they wanted ti to deisplay your actual resouce avaiablyt not your over commit availablity | |
| 15:25:46 | sean-k-mooney | im sure it could be changed but it would be a change in behavior of what that dashboard shows | |
| 15:26:47 | recyclehero | but anyway IMO 10 out of 8 is not a good thing to show especially in red. its like ur doing something wrong | |
| 15:27:14 | sean-k-mooney | its in red because you are over commiting | |
| 15:27:22 | sean-k-mooney | which in maycases is wrong | |
| 15:27:43 | sean-k-mooney | for disk or memory its potentially dangrous | |
| 15:27:53 | sean-k-mooney | disk can cause data loss and vm hangs | |
| 15:28:08 | sean-k-mooney | memory without swap can cause the vms to be killed | |
| 15:28:43 | sean-k-mooney | cpu if its too high will also eventually cause nova to stop working properly due to to high load | |
| 15:29:28 | recyclehero | BTW I am using kernel shared memory + lots of swap. i hope it works well | |
| 15:29:54 | sean-k-mooney | if you are using non numa guest then it can | |
| 15:30:28 | recyclehero | with this and using same distros for instances I hope I can get near 2.0 for memory | |
| 15:30:31 | sean-k-mooney | recyclehero: you need to have enough swap to fully account for the over commit | |
| 15:30:51 | recyclehero | lots of swap ssd | |
| 15:30:54 | sean-k-mooney | so you have allcoated at least 2 times your memroy for swap | |
| 15:32:02 | recyclehero | equal memory wont work. I have 12GB Ram + 12 GB swap. you are suggesting 24GB swap for allocation ratio of 2.0? | |
| 15:32:04 | sean-k-mooney | the guest uses more memory then in the falvor for the qemu process to use and back device es with | |
| 15:32:16 | sean-k-mooney | recyclehero: yes | |
| 15:32:22 | sean-k-mooney | you shoudl have 24G | |
| 15:32:28 | sean-k-mooney | of swap | |
| 15:32:46 | sean-k-mooney | you could go a little lowwer but you wotn get to 2.0 with only 12G | |
| 15:33:03 | sean-k-mooney | you will hit oom issues | |
| 15:33:39 | sean-k-mooney | you have not accounted for the qemu overhead or nova/os memory usage | |
| 15:33:47 | recyclehero | good to know. I should I make changes while its a fresh deployement. | |
| 15:34:05 | recyclehero | I have counted nova/os memory in the reserverd. | |
| 15:34:24 | recyclehero | but not for the qemu overhead | |
| 15:34:51 | sean-k-mooney | the qemu over head can be small or lager depening on how my cores or devices an instance has | |
| 15:35:11 | sean-k-mooney | it can be as small as ~100MB or well over a GB | |
| 15:35:35 | sean-k-mooney | per instance | |
| 15:36:25 | sean-k-mooney | so you need to caulate what the maxium amount of overhead will be for you smallest and larges flavor by booting them then figure out how many of them will run on each host and set teh reserved value accourdignly | |
| 15:36:49 | sean-k-mooney | or allocate enough swap to copenstate | |
| 15:39:18 | recyclehero | my strategy is reserved for nova+os and swap to conpensate for overcommiting | |
| 15:40:00 | sean-k-mooney | ya but again that is ignoring qemu overhead | |
| 15:40:21 | sean-k-mooney | which is ment to be acounted for in reserved or swap | |
| 15:40:45 | sean-k-mooney | in generall i advise never overcommiting memroy or disks for what its worth | |
| 15:40:50 | sean-k-mooney | only cpus | |