| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-23 | |||
| 14:36:26 | sean-k-mooney | so its coming from the ytpeing model | |
| 14:37:15 | stephenfin | mypy hasn't been updated recently though, and typing is stdlib now | |
| 14:37:18 | sean-k-mooney | form the libvirt treadpool proxy | |
| 14:37:31 | sean-k-mooney | stephenfin: yes it si from the standardlib | |
| 14:37:48 | stephenfin | crap, new eventlet release yesterday | |
| 14:38:13 | sean-k-mooney | to fix dnspython support? | |
| 14:38:23 | sean-k-mooney | we likely need to cap it untill we figure this out | |
| 14:39:39 | stephenfin | gibi: Looks like we might have a problem with the new release of eventlet ^ | |
| 14:39:50 | stephenfin | https://zuul.opendev.org/t/openstack/build/c85b0734fb524eefb67449105528b0a2/log/logs/screen-n-cpu.txt#861 | |
| 14:40:18 | sean-k-mooney | https://github.com/eventlet/eventlet/commit/9d731373e2325588a33229b425100ad538eed766 | |
| 14:40:23 | sean-k-mooney | this is apparently what changed | |
| 14:42:04 | sean-k-mooney | oh they fixed my ssl bug i think https://github.com/eventlet/eventlet/commit/7b1aa58d77550caabba3ca2e0b1bb79626c83d9c | |
| 14:42:35 | sean-k-mooney | oh no its a different one | |
| 14:43:55 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/632 was our issue | |
| 14:44:27 | stephenfin | yeah, the last working build for https://review.opendev.org/#/c/758396/ used eventlet===0.26.1 | |
| 14:44:35 | stephenfin | https://zuul.opendev.org/t/openstack/build/bf1f2b730848408b99c61e82068eb73b/log/job-output.txt#6355 | |
| 14:44:46 | stephenfin | the failing one uses eventlet===0.28.0 | |
| 14:44:50 | stephenfin | https://zuul.opendev.org/t/openstack/build/00e3a2793a1040b29b944b944c9da795/log/job-output.txt | |
| 14:44:59 | stephenfin | so eventlet has borked us \o/ | |
| 14:45:04 | sean-k-mooney | they have done a buch of release in the last month | |
| 14:46:02 | sean-k-mooney | i guess need to file a bug | |
| 14:52:43 | stephenfin | Hmm, we also bumped our version of mypy, greenlet and libvirt-python recently, so it could be those too. This will be fun | |
| 15:24:08 | recyclehero | sean-k-mooney stephenfin thanks guys. I really appreciate ur help. I guess it was okay after we did the 'None' patch in OSA but the data displayed was confusing. | |
| 15:24:51 | sean-k-mooney | it was a design choice that they made very early on in horizon | |
| 15:25:10 | sean-k-mooney | they wanted ti to deisplay your actual resouce avaiablyt not your over commit availablity | |
| 15:25:46 | sean-k-mooney | im sure it could be changed but it would be a change in behavior of what that dashboard shows | |
| 15:26:47 | recyclehero | but anyway IMO 10 out of 8 is not a good thing to show especially in red. its like ur doing something wrong | |
| 15:27:14 | sean-k-mooney | its in red because you are over commiting | |
| 15:27:22 | sean-k-mooney | which in maycases is wrong | |
| 15:27:43 | sean-k-mooney | for disk or memory its potentially dangrous | |
| 15:27:53 | sean-k-mooney | disk can cause data loss and vm hangs | |
| 15:28:08 | sean-k-mooney | memory without swap can cause the vms to be killed | |
| 15:28:43 | sean-k-mooney | cpu if its too high will also eventually cause nova to stop working properly due to to high load | |
| 15:29:28 | recyclehero | BTW I am using kernel shared memory + lots of swap. i hope it works well | |
| 15:29:54 | sean-k-mooney | if you are using non numa guest then it can | |
| 15:30:28 | recyclehero | with this and using same distros for instances I hope I can get near 2.0 for memory | |
| 15:30:31 | sean-k-mooney | recyclehero: you need to have enough swap to fully account for the over commit | |
| 15:30:51 | recyclehero | lots of swap ssd | |
| 15:30:54 | sean-k-mooney | so you have allcoated at least 2 times your memroy for swap | |
| 15:32:02 | recyclehero | equal memory wont work. I have 12GB Ram + 12 GB swap. you are suggesting 24GB swap for allocation ratio of 2.0? | |
| 15:32:04 | sean-k-mooney | the guest uses more memory then in the falvor for the qemu process to use and back device es with | |
| 15:32:16 | sean-k-mooney | recyclehero: yes | |
| 15:32:22 | sean-k-mooney | you shoudl have 24G | |
| 15:32:28 | sean-k-mooney | of swap | |
| 15:32:46 | sean-k-mooney | you could go a little lowwer but you wotn get to 2.0 with only 12G | |
| 15:33:03 | sean-k-mooney | you will hit oom issues | |
| 15:33:39 | sean-k-mooney | you have not accounted for the qemu overhead or nova/os memory usage | |
| 15:33:47 | recyclehero | good to know. I should I make changes while its a fresh deployement. | |
| 15:34:05 | recyclehero | I have counted nova/os memory in the reserverd. | |
| 15:34:24 | recyclehero | but not for the qemu overhead | |
| 15:34:51 | sean-k-mooney | the qemu over head can be small or lager depening on how my cores or devices an instance has | |
| 15:35:11 | sean-k-mooney | it can be as small as ~100MB or well over a GB | |
| 15:35:35 | sean-k-mooney | per instance | |
| 15:36:25 | sean-k-mooney | so you need to caulate what the maxium amount of overhead will be for you smallest and larges flavor by booting them then figure out how many of them will run on each host and set teh reserved value accourdignly | |
| 15:36:49 | sean-k-mooney | or allocate enough swap to copenstate | |
| 15:39:18 | recyclehero | my strategy is reserved for nova+os and swap to conpensate for overcommiting | |
| 15:40:00 | sean-k-mooney | ya but again that is ignoring qemu overhead | |
| 15:40:21 | sean-k-mooney | which is ment to be acounted for in reserved or swap | |
| 15:40:45 | sean-k-mooney | in generall i advise never overcommiting memroy or disks for what its worth | |
| 15:40:50 | sean-k-mooney | only cpus | |
| 15:41:13 | sean-k-mooney | and even then only cpus at a max of 4.0 | |
| 15:42:03 | sean-k-mooney | if you want to hevialy over commit memory you might find https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/libvirt-file-backed-memory.html intersting too | |
| 15:42:29 | sean-k-mooney | its uses files on disk that are cached in ram instead of relying on swap | |
| 15:45:31 | recyclehero | thanks sean-k-mooney for sharing ur xp | |
| 15:49:34 | sean-k-mooney | we get enough of bug reported by customer related to OOM that i always advise people against it unless the really really know what they are doing | |
| 15:50:08 | sean-k-mooney | people coming form vmware often dont know that vmware always uses file backed memroy wehn you are over commiting behind the scenes | |
| 15:58:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Make JSON valid https://review.opendev.org/759466 | |
| 15:58:59 | stephenfin | melwitt: lyarwood: That will fix our currently broken docs build ^ | |
| 15:59:19 | stephenfin | I don't know what changed :( Neither Sphinx nor Pygments have seen releases recently :-\ | |
| 15:59:30 | stephenfin | and that doc hasn't been changed | |
| 16:00:13 | melwitt | tbh I don't like the sound of "don't know what changed" :/ | |
| 16:01:36 | stephenfin | neither do I, but it's the best I can offer atm :) | |
| 16:01:42 | melwitt | also I don't understand how the change fixes it, what is it doing, shorten the string? | |
| 16:02:09 | stephenfin | no, what's there isn't valid JSON | |
| 16:02:25 | stephenfin | you can't escape newlines with backslash | |
| 16:02:31 | melwitt | ohh | |
| 16:02:43 | stephenfin | so the parser is rightly complaining | |
| 16:02:49 | melwitt | I see | |
| 16:02:55 | stephenfin | I just don't know why it's suddenly doing it now | |
| 16:02:58 | melwitt | so something started validating it | |
| 16:03:01 | melwitt | yeah | |
| 16:03:11 | stephenfin | but Sphinx has has this validation for years | |
| 16:03:19 | stephenfin | it's more than something has stopped ignoring that | |
| 16:03:51 | stephenfin | hmm, new release of that | |
| 16:04:58 | melwitt | yeah last time I helped with a docs job break it was bc of a new release of openstackdocstheme. so seems likely to be the cause | |
| 16:06:57 | melwitt | and I agree with you on what's likely happened, something started being strict(er) about validation | |
| 16:10:40 | stephenfin | not that, 2.2.6 hasn't made it to u-c yet | |
| 16:12:35 | stephenfin | Ahh, pygments 2.7.1 came in recently. That's our culprit (change I1bc1fe988e3cfd4824dd7b4a9327a67db46c1533) | |
| 16:14:40 | melwitt | \o/ | |
| 16:16:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Make JSON valid https://review.opendev.org/759466 | |
| 16:17:01 | stephenfin | I didn't see the change because I was searching for pygments (lower case p) :( | |
| 16:17:03 | stephenfin | vim-- | |
| 16:17:08 | stephenfin | melwitt: ^ | |
| 16:17:14 | melwitt | heh | |
| 16:17:20 | melwitt | on it | |
| 16:17:35 | stephenfin | The gate is still hopelessly broken due to this libvirt/eventlet issue but that's one thing off the list | |
| 16:20:00 | melwitt | ++ | |
| 16:20:58 | lyarwood | stephenfin: sorry wasn't watching irc upstream | |
| 16:21:05 | lyarwood | stephenfin: what's the eventlet issue? | |
| 16:21:28 | stephenfin | ||