Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-26
10:26:05 lyarwood awesome thanks :)
10:35:47 elod lyarwood , gibi : https://review.opendev.org/#/c/748156/
10:37:31 elod seems minor version bump required anyway, but if you see other patch that should listed also that requires minor bump, too, just let me know and I'll update the commit message
10:39:54 gibi elod: +1
10:47:36 openstackgerrit Stephen Finucane proposed openstack/nova master: Deprecate filters that have been replaced by placement filters https://review.opendev.org/745605
10:58:07 rmart04 Hi Stephenfin, thanks for this info. I'm just digesting what you've written. I am currently not using huge pages, but I think what you are saying is that one of the numa nodes ran out of memory, not the entire host. I'm not sure how reserved_host_memory_mb will help when the host has lots of memory available (250GB), i'm guessing we just ran out of memory on one particuar numa node? I should mention, we have
10:58:07 rmart04 one large VM on this host with numa_nodes:2.
10:59:05 stephenfin rmart04: Yes, exactly. We lock memory to a single NUMA node so it the instance tries to allocate memory and there's none on _that node_, you're flat out of luck
11:00:32 stephenfin Does the per-node memory allocation of that very large VM bump up against the per-host node total? Are your DIMMs evenly split between host NUMA nodes?
11:01:45 stephenfin i.e. you've 250GB available in total, but perhaps only a small bit on one particular NUMA node
11:01:58 sean-k-mooney the OOM reaper in the kernel runs per numa node so if a numa node runs out of memory it will start killing process if the kerenl needs to allcoate even if the other node is entirely empty
11:02:12 rmart04 So Host has 768GB split evenly across numa nodes. There is one instance, 750GB in size, and has seen more than half its memory used successfully in the past. I'm not sure how to see how much memory a particular numa node has available? Is that easy to see?
11:02:31 sean-k-mooney if you are using numa but not pinning or hugepages
11:02:41 alex_xu stephenfin: are we ensure the operator finish the pin_cpu_set to dedicated_cpu_set/sahred_cpu_set change before update to V release?
11:02:46 sean-k-mooney then we will place all the vms on numa node 0 and never use the second numa node
11:02:55 sean-k-mooney im working on fixing that this morning
11:03:00 rmart04 cpu_policy is dedicated, so I believed its pinned
11:03:20 alex_xu stephenfin: or the operator can do that change when upgrade U to V?
11:03:33 sean-k-mooney ah in that case we will stack all vms on the first numa node untill we run our of pinned cpus then use the next one
11:04:16 stephenfin alex_xu: A user can continue to use the legacy vcpu_pin_set in U. I'm hoping to drop support for it in V, but I have too many series in flight at the moment so it might slip to W
11:04:19 stephenfin alex_xu: If that's what you mean?
11:04:20 sean-k-mooney alex_xu: i think the plan is to remove vcpu_pin_set in v so they have to upgrade in U
11:04:33 sean-k-mooney ideally they would do it in train
11:04:41 sean-k-mooney but they can do it in U
11:05:42 sean-k-mooney rmart04: you can see it in /sys
11:05:47 sean-k-mooney ill get the path on sec
11:06:20 alex_xu stephenfin: sean-k-mooney ,I'm thinking of 744021 may stop the upgrade from U to V if the operate transft from legacy to new config in that time
11:06:53 sean-k-mooney rmart04: cat /sys/bus/node/devices/node*/meminfo
11:07:31 alex_xu if we force that legacy config change finished in U, then it should be fine
11:08:09 sean-k-mooney alex_xu: its there because we had customer upgraind into an invlaid config
11:08:13 stephenfin alex_xu: I don't understand the issue
11:08:41 sean-k-mooney they had vms with the old configuration where they were using isolate but the new config options
11:08:44 stephenfin A user should never get into a situation where they have new-style config but a pinned instance is consuming VCPU instead of PCPU
11:08:48 sean-k-mooney so they were using vcpus instead of pcpus
11:09:02 sean-k-mooney even thogh the host reported pcpus
11:09:10 stephenfin This patch should be backported to Ussuri and Train
11:09:26 sean-k-mooney yes
11:10:14 stephenfin Is there something upgrade specific I'm missing, perhaps?
11:10:14 sean-k-mooney alex_xu: it was an edgecase we forgot to block, we taught it would be blocked by the numa toplogy filter but the check was missing
11:10:32 alex_xu if the host with legacy config is running isolated thread instance, but the 744021 stop the isolated thread instance? (I'm still try to ramp up my mind)
11:10:53 sean-k-mooney alex_xu: if its using legacy configs it will allow it to boot
11:11:01 sean-k-mooney alex_xu: it only changes the behavior for new configs
11:11:43 alex_xu oh, right
11:12:14 sean-k-mooney alex_xu: we dont allow the shared and dedicated set to overlap so the only time they contian the same values is if we are using the legacy config options
11:12:51 sean-k-mooney so and host_cell.pcpuset != host_cell.cpuset
11:13:05 sean-k-mooney is an indirect check for is the host using new config options
11:13:08 lyarwood elod: sorry missed that https://review.opendev.org/#/c/747357/ wasn't included
11:13:11 sean-k-mooney that will also work in the schduler
11:13:14 lyarwood elod: can we wait for that and then release?
11:13:23 sean-k-mooney since we cant actully check the config directly there
11:14:33 jsuchome lyarwood: gibi stephenfin https://review.opendev.org/#/c/574301/ finally green again, any further changes necessary?
11:15:50 elod lyarwood: oh, of course, sorry, I forgot to check what else on the queue :S
11:16:30 openstackgerrit Merged openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522
11:17:00 lyarwood jsuchome: https://review.opendev.org/#/c/746904/ - I just need to respin this after lunch and I think we should be good to go
11:17:43 alex_xu sean-k-mooney: but with new config, why we meet the case fallback to query VCPU?
11:18:44 alex_xu fallback query should find the host with legacy config
11:19:53 sean-k-mooney alex_xu: not if all host are using the new configs and you forget to trun off the fallback
11:20:17 sean-k-mooney the fallback can select the vcpu on host with the new config options
11:20:33 sean-k-mooney there is noting in the fallback query to prevent that
11:20:52 sean-k-mooney we were relying on the numa toplogy filter to handel that case and this check was missing
11:21:44 sean-k-mooney alex_xu: for what its worth i was 99% certin the numa toplogy filter would block this untill i repoduced the issue locally
11:23:40 alex_xu sean-k-mooney: if forget turn the fallback off, the scheduler will chocie the host with shared_cpu_set?
11:24:11 alex_xu oh, it is not, since host.pcpu is empty
11:28:41 sean-k-mooney so with the fall back enabled on a cloudl with all new config options
11:28:52 sean-k-mooney and smt enabled
11:29:33 sean-k-mooney when we do the initall query the hyperthreading forbiden trait will cause the PCPU queury to retrun no hosts since they all have smt enabled
11:29:41 alex_xu sean-k-mooney: it is new deployed host, with dedicated and shared cpu set, that allows the mixed instance. But the fallback still enabled, then in the case of there is no more pCPU, then scheduler fallback to VCPU. Then host with shared cpu set allow that scheduling?
11:29:53 sean-k-mooney then the fallback with vcpus will return allocation agains tthe vcpu inventories
11:30:23 sean-k-mooney alex_xu: yes basicly
11:31:30 sean-k-mooney so what happens is we end up claiming vCPUs in placement and then pin the vm to the cpu_dedicated_set cpus
11:31:42 alex_xu if that is the case, a shared thread instance also can be scheduled to that host, right?
11:31:52 sean-k-mooney yes
11:32:03 alex_xu emm...that is still wrong
11:32:33 sean-k-mooney well shared instance and pinned can mix on the same host as of train
11:32:36 sean-k-mooney that is ok
11:32:41 sean-k-mooney but the allocation are wrong
11:32:44 alex_xu so we should ask the user turn off the fallback before upgrading to V?
11:33:05 sean-k-mooney we shoudl remove it when we removed vcpu_pin_set
11:33:31 sean-k-mooney with https://review.opendev.org/#/c/744021/3 its safe to leave it on
11:33:50 sean-k-mooney as placement will still do the wrong thing but the numa toplogy filter or the hardware module on teh compute host will block it
11:33:51 alex_xu but it doesn't fix the case for shared thread instance, right?
11:34:03 sean-k-mooney that is not broken
11:34:27 sean-k-mooney if you have cpu_policy=shared then it will request vcpus
11:34:40 noonedeadpunk hey everyone! I was wondering if it's possible to prohibit users to use ephemeral storage? I was trying to look through the policies but the only thing I found and was related is to allow use zero disk flavors. While it allows to use cinder volumes, it doesn't prohibits ephemeral ones...
11:34:41 sean-k-mooney the fallback only applies to cpu_policy=dedicated
11:35:05 sean-k-mooney and cpu_thread_policy only applies to cpu_polcy=dedicated
11:35:18 alex_xu yes
11:35:24 noonedeadpunk Eventually while I was looking I saw other ppl was also wondering about that opportuninty
11:35:26 sean-k-mooney noonedeadpunk: yes you can
11:35:28 sean-k-mooney via config
11:35:41 alex_xu I mean the case cpu_thread_policy is shared and cpu_policy is dedicated
11:35:59 sean-k-mooney oh in that case we dont claim vcpus at all
11:36:04 sean-k-mooney well we will
11:36:13 sean-k-mooney because of the bug
11:36:41 sean-k-mooney alex_xu: the vm is still being pinned to the cpu_dedicated_set
11:37:00 sean-k-mooney and its emulator treads will stll be pinned to the cpu_shared_set
11:37:13 sean-k-mooney we just wont claim PCPUs in plament
11:37:29 sean-k-mooney noonedeadpunk: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices
11:37:44 sean-k-mooney noonedeadpunk: you can set that to 0
11:37:44 alex_xu wait, the filter will stop that, since there is no enough pcpu in hostnuma.pcpu

Earlier   Later