| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-23 | |||
| 13:43:01 | sean-k-mooney | you have only 8 cores on the host | |
| 13:43:12 | recyclehero | but I want to overcommit :(( | |
| 13:43:13 | sean-k-mooney | you cant oversubseribe against your self | |
| 13:43:22 | sean-k-mooney | launch 2 5 core ones | |
| 13:43:35 | recyclehero | aha ok | |
| 13:49:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: virt: Remove 'change_instance_metadata' API https://review.opendev.org/749316 | |
| 13:59:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Liberty database migrations https://review.opendev.org/758397 | |
| 14:14:33 | stephenfin | this looks new :-\ https://zuul.opendev.org/t/openstack/build/c85b0734fb524eefb67449105528b0a2/log/logs/screen-n-cpu.txt#861 | |
| 14:19:47 | recyclehero | stephenfin: I am seeing 10 VCPU on my project so I guess it is ok. thank you | |
| 14:20:23 | recyclehero | but on hoizon in hypervisor VCPU usage is 10 of 8 | |
| 14:20:46 | recyclehero | stephenfin: before getting of your shoulder, what is the logic that one cat overcommit self? | |
| 14:20:56 | recyclehero | cant | |
| 14:20:59 | sean-k-mooney | recyclehero: 10 of 8 is correct | |
| 14:21:05 | sean-k-mooney | that is what should be in horizon | |
| 14:21:31 | sean-k-mooney | stephenfin: nope | |
| 14:21:41 | sean-k-mooney | stephenfin: we have seen that downstream before | |
| 14:21:57 | sean-k-mooney | stephenfin: we belive it happens due to out of memory | |
| 14:22:16 | recyclehero | sean-k-mooney: I mean why is that that I cant have a single x>8 cores instance? | |
| 14:22:38 | recyclehero | stephenfin: wrong mention | |
| 14:23:27 | stephenfin | recyclehero: PCPU is how we track pinned CPUs. Pinned CPUs means each instance CPU is assigned its own host CPU | |
| 14:23:38 | stephenfin | So no overcommit. If an instance is using a host CPU, no other instance can use it | |
| 14:24:02 | sean-k-mooney | well for non pcpus its also not required | |
| 14:24:13 | sean-k-mooney | *alllow | |
| 14:24:21 | stephenfin | recyclehero: As for the 10 of 8, Horizon is probably retrieving stats from nova's 'os-hypervisors' API which is notoriously broken | |
| 14:24:25 | sean-k-mooney | we do not allow 2 guest cpus to use the same host cpu | |
| 14:24:42 | sean-k-mooney | stephenfin: horizon is doing the right thing | |
| 14:24:55 | sean-k-mooney | its ment to report how many cpus are avaiable and it doe suse os hyperviros | |
| 14:25:17 | sean-k-mooney | it is not ment to the cpus multiplied by the allocation ration | |
| 14:25:25 | stephenfin | That's wrong | |
| 14:25:34 | sean-k-mooney | no that is how it was ment to work | |
| 14:25:38 | sean-k-mooney | and how it has always worked | |
| 14:25:48 | sean-k-mooney | it has never used the allocation ration | |
| 14:25:48 | stephenfin | Create a load of guests with 'hw:numa_nodes=1' on two node host with 6 CPUs per node | |
| 14:26:00 | stephenfin | How many host CPUs are free if you boot those guests? | |
| 14:26:29 | stephenfin | The os-hypervisors information makes no sense in a world of overcommit | |
| 14:26:43 | stephenfin | which is why we're planning to drop it in Wallaby | |
| 14:26:55 | sean-k-mooney | stephenfin: recyclehero is try ing to boot a 10 vcpu guest on an 8 cpu host | |
| 14:27:01 | sean-k-mooney | which we have never supported | |
| 14:27:19 | sean-k-mooney | because 2 gust cpus cant consome the same host cpu enven if they are floating | |
| 14:27:23 | stephenfin | sean-k-mooney: yes, that's true | |
| 14:28:11 | stephenfin | sean-k-mooney: but we have 'max_unit' to track that | |
| 14:29:01 | stephenfin | recyclehero: Oh, that was an interesting point on the output of e.g. 'openstack resource provider inventory list' | |
| 14:29:18 | stephenfin | I'd have expected the total column to show the total inclusive of allocation_ratio | |
| 14:29:41 | stephenfin | but it doesn't; it shows the total before the allocation ratio is applied | |
| 14:29:41 | sean-k-mooney | no | |
| 14:29:50 | stephenfin | so you have to do that maths yourself :) | |
| 14:29:57 | sean-k-mooney | yes | |
| 14:30:06 | stephenfin | sean-k-mooney: make up your mind, man! | |
| 14:30:07 | stephenfin | :P | |
| 14:30:09 | sean-k-mooney | so that the different values are not depenent on each other | |
| 14:30:17 | sean-k-mooney | you have to do the math yourself | |
| 14:30:34 | sean-k-mooney | multipleying total by allocation ratio - reserved | |
| 14:32:13 | stephenfin | sean-k-mooney: as for that earlier issue, this entire stack is failing with it :( https://review.opendev.org/#/q/topic:bp/compact-db-migrations-wallaby | |
| 14:32:29 | stephenfin | I don't think it's something I've done wrong, but I'm waiting to see if another patch fails first | |
| 14:32:55 | sean-k-mooney | stephenfin: the import issue if its the same one it due to an oom issue causeing the import to fail | |
| 14:33:01 | sean-k-mooney | if not then rbd is not isntalled | |
| 14:33:12 | sean-k-mooney | because ceph-common is not installed | |
| 14:33:20 | stephenfin | oh, no, look down | |
| 14:33:22 | sean-k-mooney | i didnt look too closelon | |
| 14:33:32 | stephenfin | ignore the rbd thing | |
| 14:33:37 | stephenfin | that's lyarwood's work | |
| 14:34:59 | sean-k-mooney | oh line 861 | |
| 14:35:18 | sean-k-mooney | it did not scole down fro some reason | |
| 14:35:36 | sean-k-mooney | TypeError: Parameterized generics cannot be used with class or instance checks | |
| 14:35:42 | stephenfin | Yeah | |
| 14:35:47 | stephenfin | I've never seen that before | |
| 14:35:52 | stephenfin | it's buried in eventlet too | |
| 14:36:26 | sean-k-mooney | so its coming from the ytpeing model | |
| 14:37:15 | stephenfin | mypy hasn't been updated recently though, and typing is stdlib now | |
| 14:37:18 | sean-k-mooney | form the libvirt treadpool proxy | |
| 14:37:31 | sean-k-mooney | stephenfin: yes it si from the standardlib | |
| 14:37:48 | stephenfin | crap, new eventlet release yesterday | |
| 14:38:13 | sean-k-mooney | to fix dnspython support? | |
| 14:38:23 | sean-k-mooney | we likely need to cap it untill we figure this out | |
| 14:39:39 | stephenfin | gibi: Looks like we might have a problem with the new release of eventlet ^ | |
| 14:39:50 | stephenfin | https://zuul.opendev.org/t/openstack/build/c85b0734fb524eefb67449105528b0a2/log/logs/screen-n-cpu.txt#861 | |
| 14:40:18 | sean-k-mooney | https://github.com/eventlet/eventlet/commit/9d731373e2325588a33229b425100ad538eed766 | |
| 14:40:23 | sean-k-mooney | this is apparently what changed | |
| 14:42:04 | sean-k-mooney | oh they fixed my ssl bug i think https://github.com/eventlet/eventlet/commit/7b1aa58d77550caabba3ca2e0b1bb79626c83d9c | |
| 14:42:35 | sean-k-mooney | oh no its a different one | |
| 14:43:55 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/632 was our issue | |
| 14:44:27 | stephenfin | yeah, the last working build for https://review.opendev.org/#/c/758396/ used eventlet===0.26.1 | |
| 14:44:35 | stephenfin | https://zuul.opendev.org/t/openstack/build/bf1f2b730848408b99c61e82068eb73b/log/job-output.txt#6355 | |
| 14:44:46 | stephenfin | the failing one uses eventlet===0.28.0 | |
| 14:44:50 | stephenfin | https://zuul.opendev.org/t/openstack/build/00e3a2793a1040b29b944b944c9da795/log/job-output.txt | |
| 14:44:59 | stephenfin | so eventlet has borked us \o/ | |
| 14:45:04 | sean-k-mooney | they have done a buch of release in the last month | |
| 14:46:02 | sean-k-mooney | i guess need to file a bug | |
| 14:52:43 | stephenfin | Hmm, we also bumped our version of mypy, greenlet and libvirt-python recently, so it could be those too. This will be fun | |
| 15:24:08 | recyclehero | sean-k-mooney stephenfin thanks guys. I really appreciate ur help. I guess it was okay after we did the 'None' patch in OSA but the data displayed was confusing. | |
| 15:24:51 | sean-k-mooney | it was a design choice that they made very early on in horizon | |
| 15:25:10 | sean-k-mooney | they wanted ti to deisplay your actual resouce avaiablyt not your over commit availablity | |
| 15:25:46 | sean-k-mooney | im sure it could be changed but it would be a change in behavior of what that dashboard shows | |
| 15:26:47 | recyclehero | but anyway IMO 10 out of 8 is not a good thing to show especially in red. its like ur doing something wrong | |
| 15:27:14 | sean-k-mooney | its in red because you are over commiting | |
| 15:27:22 | sean-k-mooney | which in maycases is wrong | |
| 15:27:43 | sean-k-mooney | for disk or memory its potentially dangrous | |
| 15:27:53 | sean-k-mooney | disk can cause data loss and vm hangs | |
| 15:28:08 | sean-k-mooney | memory without swap can cause the vms to be killed | |
| 15:28:43 | sean-k-mooney | cpu if its too high will also eventually cause nova to stop working properly due to to high load | |