| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-23 | |||
| 13:33:20 | sean-k-mooney | https://github.com/openstack/nova/blob/6a5e158756edc6c01bb23c26dcfc4c3a3df703dc/nova/notifications/base.py#L211-L252 | |
| 13:33:50 | sean-k-mooney | that never read it and the object does nto hold that value https://github.com/openstack/nova/blob/6a5e158756edc6c01bb23c26dcfc4c3a3df703dc/nova/notifications/objects/instance.py#L287-L317 | |
| 13:34:22 | sean-k-mooney | Diavel: so if you are using the versioned notificaiton which is what you should be using you wont get it | |
| 13:34:37 | sean-k-mooney | it can be added but it cannot be bakcported | |
| 13:35:15 | recyclehero | noonedeadpunk, sean-k-mooney : I set inital_cpu_allocation_ratio via this config overrides. | |
| 13:35:55 | recyclehero | sean-k-mooney: I read the spec but dont know why its still overriding my config | |
| 13:35:58 | sean-k-mooney | Diavel: so you will need to use the unversion notification and then add it to the versioned notificaton for wallaby | |
| 13:36:19 | recyclehero | because I have set it None, I mean they are not there. thats None right? | |
| 13:36:28 | Diavel | sean-k-mooney: got it.. shit I missed that, sorry and thanks again | |
| 13:36:40 | Diavel | sean-k-mooney: I am using the versioned one | |
| 13:37:13 | sean-k-mooney | Diavel: no worries we can totally add it but they are versioned so we cant backprot it | |
| 13:37:36 | sean-k-mooney | Diavel: it would be good to add it so that you dont have to use the unversioned ones | |
| 13:37:49 | Diavel | sean-k-mooney: yeah I totally agree | |
| 13:38:11 | sean-k-mooney | recyclehero: initial is only used when a node if first deployed | |
| 13:38:34 | sean-k-mooney | since it has a default setting it to none will result in it still using the default | |
| 13:41:00 | recyclehero | sean-k-mooney: I didnt completly get the second one. I did the patch which I sent. so cpu_allocation_ratio is not present in nova.conf. also I ahve set the inital_cpu_allocation_ratio:16 which is present in nova.conf | |
| 13:41:14 | recyclehero | it was a clean deploy | |
| 13:41:31 | recyclehero | and I think it did use the iniita ratio because it is set to 16 | |
| 13:41:39 | recyclehero | the problem lies it total I guess | |
| 13:41:54 | recyclehero | you were asking me before to do some tests. | |
| 13:41:59 | sean-k-mooney | well total shoudl be equal to the number of cores | |
| 13:42:08 | sean-k-mooney | not 16*number of cores | |
| 13:42:19 | sean-k-mooney | and horizon should not change | |
| 13:42:38 | recyclehero | oh so It means I have it set now. let me check lunching a 10 core instance | |
| 13:42:55 | sean-k-mooney | you cant | |
| 13:43:01 | sean-k-mooney | you have only 8 cores on the host | |
| 13:43:12 | recyclehero | but I want to overcommit :(( | |
| 13:43:13 | sean-k-mooney | you cant oversubseribe against your self | |
| 13:43:22 | sean-k-mooney | launch 2 5 core ones | |
| 13:43:35 | recyclehero | aha ok | |
| 13:49:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: virt: Remove 'change_instance_metadata' API https://review.opendev.org/749316 | |
| 13:59:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Liberty database migrations https://review.opendev.org/758397 | |
| 14:14:33 | stephenfin | this looks new :-\ https://zuul.opendev.org/t/openstack/build/c85b0734fb524eefb67449105528b0a2/log/logs/screen-n-cpu.txt#861 | |
| 14:19:47 | recyclehero | stephenfin: I am seeing 10 VCPU on my project so I guess it is ok. thank you | |
| 14:20:23 | recyclehero | but on hoizon in hypervisor VCPU usage is 10 of 8 | |
| 14:20:46 | recyclehero | stephenfin: before getting of your shoulder, what is the logic that one cat overcommit self? | |
| 14:20:56 | recyclehero | cant | |
| 14:20:59 | sean-k-mooney | recyclehero: 10 of 8 is correct | |
| 14:21:05 | sean-k-mooney | that is what should be in horizon | |
| 14:21:31 | sean-k-mooney | stephenfin: nope | |
| 14:21:41 | sean-k-mooney | stephenfin: we have seen that downstream before | |
| 14:21:57 | sean-k-mooney | stephenfin: we belive it happens due to out of memory | |
| 14:22:16 | recyclehero | sean-k-mooney: I mean why is that that I cant have a single x>8 cores instance? | |
| 14:22:38 | recyclehero | stephenfin: wrong mention | |
| 14:23:27 | stephenfin | recyclehero: PCPU is how we track pinned CPUs. Pinned CPUs means each instance CPU is assigned its own host CPU | |
| 14:23:38 | stephenfin | So no overcommit. If an instance is using a host CPU, no other instance can use it | |
| 14:24:02 | sean-k-mooney | well for non pcpus its also not required | |
| 14:24:13 | sean-k-mooney | *alllow | |
| 14:24:21 | stephenfin | recyclehero: As for the 10 of 8, Horizon is probably retrieving stats from nova's 'os-hypervisors' API which is notoriously broken | |
| 14:24:25 | sean-k-mooney | we do not allow 2 guest cpus to use the same host cpu | |
| 14:24:42 | sean-k-mooney | stephenfin: horizon is doing the right thing | |
| 14:24:55 | sean-k-mooney | its ment to report how many cpus are avaiable and it doe suse os hyperviros | |
| 14:25:17 | sean-k-mooney | it is not ment to the cpus multiplied by the allocation ration | |
| 14:25:25 | stephenfin | That's wrong | |
| 14:25:34 | sean-k-mooney | no that is how it was ment to work | |
| 14:25:38 | sean-k-mooney | and how it has always worked | |
| 14:25:48 | sean-k-mooney | it has never used the allocation ration | |
| 14:25:48 | stephenfin | Create a load of guests with 'hw:numa_nodes=1' on two node host with 6 CPUs per node | |
| 14:26:00 | stephenfin | How many host CPUs are free if you boot those guests? | |
| 14:26:29 | stephenfin | The os-hypervisors information makes no sense in a world of overcommit | |
| 14:26:43 | stephenfin | which is why we're planning to drop it in Wallaby | |
| 14:26:55 | sean-k-mooney | stephenfin: recyclehero is try ing to boot a 10 vcpu guest on an 8 cpu host | |
| 14:27:01 | sean-k-mooney | which we have never supported | |
| 14:27:19 | sean-k-mooney | because 2 gust cpus cant consome the same host cpu enven if they are floating | |
| 14:27:23 | stephenfin | sean-k-mooney: yes, that's true | |
| 14:28:11 | stephenfin | sean-k-mooney: but we have 'max_unit' to track that | |
| 14:29:01 | stephenfin | recyclehero: Oh, that was an interesting point on the output of e.g. 'openstack resource provider inventory list' | |
| 14:29:18 | stephenfin | I'd have expected the total column to show the total inclusive of allocation_ratio | |
| 14:29:41 | stephenfin | but it doesn't; it shows the total before the allocation ratio is applied | |
| 14:29:41 | sean-k-mooney | no | |
| 14:29:50 | stephenfin | so you have to do that maths yourself :) | |
| 14:29:57 | sean-k-mooney | yes | |
| 14:30:06 | stephenfin | sean-k-mooney: make up your mind, man! | |
| 14:30:07 | stephenfin | :P | |
| 14:30:09 | sean-k-mooney | so that the different values are not depenent on each other | |
| 14:30:17 | sean-k-mooney | you have to do the math yourself | |
| 14:30:34 | sean-k-mooney | multipleying total by allocation ratio - reserved | |
| 14:32:13 | stephenfin | sean-k-mooney: as for that earlier issue, this entire stack is failing with it :( https://review.opendev.org/#/q/topic:bp/compact-db-migrations-wallaby | |
| 14:32:29 | stephenfin | I don't think it's something I've done wrong, but I'm waiting to see if another patch fails first | |
| 14:32:55 | sean-k-mooney | stephenfin: the import issue if its the same one it due to an oom issue causeing the import to fail | |
| 14:33:01 | sean-k-mooney | if not then rbd is not isntalled | |
| 14:33:12 | sean-k-mooney | because ceph-common is not installed | |
| 14:33:20 | stephenfin | oh, no, look down | |
| 14:33:22 | sean-k-mooney | i didnt look too closelon | |
| 14:33:32 | stephenfin | ignore the rbd thing | |
| 14:33:37 | stephenfin | that's lyarwood's work | |
| 14:34:59 | sean-k-mooney | oh line 861 | |
| 14:35:18 | sean-k-mooney | it did not scole down fro some reason | |
| 14:35:36 | sean-k-mooney | TypeError: Parameterized generics cannot be used with class or instance checks | |
| 14:35:42 | stephenfin | Yeah | |
| 14:35:47 | stephenfin | I've never seen that before | |
| 14:35:52 | stephenfin | it's buried in eventlet too | |
| 14:36:26 | sean-k-mooney | so its coming from the ytpeing model | |
| 14:37:15 | stephenfin | mypy hasn't been updated recently though, and typing is stdlib now | |
| 14:37:18 | sean-k-mooney | form the libvirt treadpool proxy | |
| 14:37:31 | sean-k-mooney | stephenfin: yes it si from the standardlib | |
| 14:37:48 | stephenfin | crap, new eventlet release yesterday | |
| 14:38:13 | sean-k-mooney | to fix dnspython support? | |
| 14:38:23 | sean-k-mooney | we likely need to cap it untill we figure this out | |
| 14:39:39 | stephenfin | gibi: Looks like we might have a problem with the new release of eventlet ^ | |