| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-23 | |||
| 14:23:38 | stephenfin | So no overcommit. If an instance is using a host CPU, no other instance can use it | |
| 14:24:02 | sean-k-mooney | well for non pcpus its also not required | |
| 14:24:13 | sean-k-mooney | *alllow | |
| 14:24:21 | stephenfin | recyclehero: As for the 10 of 8, Horizon is probably retrieving stats from nova's 'os-hypervisors' API which is notoriously broken | |
| 14:24:25 | sean-k-mooney | we do not allow 2 guest cpus to use the same host cpu | |
| 14:24:42 | sean-k-mooney | stephenfin: horizon is doing the right thing | |
| 14:24:55 | sean-k-mooney | its ment to report how many cpus are avaiable and it doe suse os hyperviros | |
| 14:25:17 | sean-k-mooney | it is not ment to the cpus multiplied by the allocation ration | |
| 14:25:25 | stephenfin | That's wrong | |
| 14:25:34 | sean-k-mooney | no that is how it was ment to work | |
| 14:25:38 | sean-k-mooney | and how it has always worked | |
| 14:25:48 | sean-k-mooney | it has never used the allocation ration | |
| 14:25:48 | stephenfin | Create a load of guests with 'hw:numa_nodes=1' on two node host with 6 CPUs per node | |
| 14:26:00 | stephenfin | How many host CPUs are free if you boot those guests? | |
| 14:26:29 | stephenfin | The os-hypervisors information makes no sense in a world of overcommit | |
| 14:26:43 | stephenfin | which is why we're planning to drop it in Wallaby | |
| 14:26:55 | sean-k-mooney | stephenfin: recyclehero is try ing to boot a 10 vcpu guest on an 8 cpu host | |
| 14:27:01 | sean-k-mooney | which we have never supported | |
| 14:27:19 | sean-k-mooney | because 2 gust cpus cant consome the same host cpu enven if they are floating | |
| 14:27:23 | stephenfin | sean-k-mooney: yes, that's true | |
| 14:28:11 | stephenfin | sean-k-mooney: but we have 'max_unit' to track that | |
| 14:29:01 | stephenfin | recyclehero: Oh, that was an interesting point on the output of e.g. 'openstack resource provider inventory list' | |
| 14:29:18 | stephenfin | I'd have expected the total column to show the total inclusive of allocation_ratio | |
| 14:29:41 | stephenfin | but it doesn't; it shows the total before the allocation ratio is applied | |
| 14:29:41 | sean-k-mooney | no | |
| 14:29:50 | stephenfin | so you have to do that maths yourself :) | |
| 14:29:57 | sean-k-mooney | yes | |
| 14:30:06 | stephenfin | sean-k-mooney: make up your mind, man! | |
| 14:30:07 | stephenfin | :P | |
| 14:30:09 | sean-k-mooney | so that the different values are not depenent on each other | |
| 14:30:17 | sean-k-mooney | you have to do the math yourself | |
| 14:30:34 | sean-k-mooney | multipleying total by allocation ratio - reserved | |
| 14:32:13 | stephenfin | sean-k-mooney: as for that earlier issue, this entire stack is failing with it :( https://review.opendev.org/#/q/topic:bp/compact-db-migrations-wallaby | |
| 14:32:29 | stephenfin | I don't think it's something I've done wrong, but I'm waiting to see if another patch fails first | |
| 14:32:55 | sean-k-mooney | stephenfin: the import issue if its the same one it due to an oom issue causeing the import to fail | |
| 14:33:01 | sean-k-mooney | if not then rbd is not isntalled | |
| 14:33:12 | sean-k-mooney | because ceph-common is not installed | |
| 14:33:20 | stephenfin | oh, no, look down | |
| 14:33:22 | sean-k-mooney | i didnt look too closelon | |
| 14:33:32 | stephenfin | ignore the rbd thing | |
| 14:33:37 | stephenfin | that's lyarwood's work | |
| 14:34:59 | sean-k-mooney | oh line 861 | |
| 14:35:18 | sean-k-mooney | it did not scole down fro some reason | |
| 14:35:36 | sean-k-mooney | TypeError: Parameterized generics cannot be used with class or instance checks | |
| 14:35:42 | stephenfin | Yeah | |
| 14:35:47 | stephenfin | I've never seen that before | |
| 14:35:52 | stephenfin | it's buried in eventlet too | |
| 14:36:26 | sean-k-mooney | so its coming from the ytpeing model | |
| 14:37:15 | stephenfin | mypy hasn't been updated recently though, and typing is stdlib now | |
| 14:37:18 | sean-k-mooney | form the libvirt treadpool proxy | |
| 14:37:31 | sean-k-mooney | stephenfin: yes it si from the standardlib | |
| 14:37:48 | stephenfin | crap, new eventlet release yesterday | |
| 14:38:13 | sean-k-mooney | to fix dnspython support? | |
| 14:38:23 | sean-k-mooney | we likely need to cap it untill we figure this out | |
| 14:39:39 | stephenfin | gibi: Looks like we might have a problem with the new release of eventlet ^ | |
| 14:39:50 | stephenfin | https://zuul.opendev.org/t/openstack/build/c85b0734fb524eefb67449105528b0a2/log/logs/screen-n-cpu.txt#861 | |
| 14:40:18 | sean-k-mooney | https://github.com/eventlet/eventlet/commit/9d731373e2325588a33229b425100ad538eed766 | |
| 14:40:23 | sean-k-mooney | this is apparently what changed | |
| 14:42:04 | sean-k-mooney | oh they fixed my ssl bug i think https://github.com/eventlet/eventlet/commit/7b1aa58d77550caabba3ca2e0b1bb79626c83d9c | |
| 14:42:35 | sean-k-mooney | oh no its a different one | |
| 14:43:55 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/632 was our issue | |
| 14:44:27 | stephenfin | yeah, the last working build for https://review.opendev.org/#/c/758396/ used eventlet===0.26.1 | |
| 14:44:35 | stephenfin | https://zuul.opendev.org/t/openstack/build/bf1f2b730848408b99c61e82068eb73b/log/job-output.txt#6355 | |
| 14:44:46 | stephenfin | the failing one uses eventlet===0.28.0 | |
| 14:44:50 | stephenfin | https://zuul.opendev.org/t/openstack/build/00e3a2793a1040b29b944b944c9da795/log/job-output.txt | |
| 14:44:59 | stephenfin | so eventlet has borked us \o/ | |
| 14:45:04 | sean-k-mooney | they have done a buch of release in the last month | |
| 14:46:02 | sean-k-mooney | i guess need to file a bug | |
| 14:52:43 | stephenfin | Hmm, we also bumped our version of mypy, greenlet and libvirt-python recently, so it could be those too. This will be fun | |
| 15:24:08 | recyclehero | sean-k-mooney stephenfin thanks guys. I really appreciate ur help. I guess it was okay after we did the 'None' patch in OSA but the data displayed was confusing. | |
| 15:24:51 | sean-k-mooney | it was a design choice that they made very early on in horizon | |
| 15:25:10 | sean-k-mooney | they wanted ti to deisplay your actual resouce avaiablyt not your over commit availablity | |
| 15:25:46 | sean-k-mooney | im sure it could be changed but it would be a change in behavior of what that dashboard shows | |
| 15:26:47 | recyclehero | but anyway IMO 10 out of 8 is not a good thing to show especially in red. its like ur doing something wrong | |
| 15:27:14 | sean-k-mooney | its in red because you are over commiting | |
| 15:27:22 | sean-k-mooney | which in maycases is wrong | |
| 15:27:43 | sean-k-mooney | for disk or memory its potentially dangrous | |
| 15:27:53 | sean-k-mooney | disk can cause data loss and vm hangs | |
| 15:28:08 | sean-k-mooney | memory without swap can cause the vms to be killed | |
| 15:28:43 | sean-k-mooney | cpu if its too high will also eventually cause nova to stop working properly due to to high load | |
| 15:29:28 | recyclehero | BTW I am using kernel shared memory + lots of swap. i hope it works well | |
| 15:29:54 | sean-k-mooney | if you are using non numa guest then it can | |
| 15:30:28 | recyclehero | with this and using same distros for instances I hope I can get near 2.0 for memory | |
| 15:30:31 | sean-k-mooney | recyclehero: you need to have enough swap to fully account for the over commit | |
| 15:30:51 | recyclehero | lots of swap ssd | |
| 15:30:54 | sean-k-mooney | so you have allcoated at least 2 times your memroy for swap | |
| 15:32:02 | recyclehero | equal memory wont work. I have 12GB Ram + 12 GB swap. you are suggesting 24GB swap for allocation ratio of 2.0? | |
| 15:32:04 | sean-k-mooney | the guest uses more memory then in the falvor for the qemu process to use and back device es with | |
| 15:32:16 | sean-k-mooney | recyclehero: yes | |
| 15:32:22 | sean-k-mooney | you shoudl have 24G | |
| 15:32:28 | sean-k-mooney | of swap | |
| 15:32:46 | sean-k-mooney | you could go a little lowwer but you wotn get to 2.0 with only 12G | |
| 15:33:03 | sean-k-mooney | you will hit oom issues | |
| 15:33:39 | sean-k-mooney | you have not accounted for the qemu overhead or nova/os memory usage | |
| 15:33:47 | recyclehero | good to know. I should I make changes while its a fresh deployement. | |
| 15:34:05 | recyclehero | I have counted nova/os memory in the reserverd. | |
| 15:34:24 | recyclehero | but not for the qemu overhead | |
| 15:34:51 | sean-k-mooney | the qemu over head can be small or lager depening on how my cores or devices an instance has | |
| 15:35:11 | sean-k-mooney | it can be as small as ~100MB or well over a GB | |
| 15:35:35 | sean-k-mooney | per instance | |