Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-26
08:54:49 stephenfin alex_xu: I suspect it impacts the mixed CPU work too
08:58:47 openstackgerrit Brin Zhang proposed openstack/nova master: Cyborg evacuate support https://review.opendev.org/715326
09:01:18 stephenfin rmart04: Using the dedicated CPU policy implies a single-node NUMA topology by default, and using a NUMA topology implies consuming memory from a single host NUMA node for each guest NUMA node
09:02:44 stephenfin rmart04: That's a roundabout way of saying that I suspect you haven't reserved memory on per-node basis and may have allocated something like hugepages unevenly across NUMA nodes
09:05:08 stephenfin rmart04: nova won't let you overcommit against a host NUMA node using nova instances, but it can't account for host processes or huge page allocations consuming memory on that node. You've to do that manually
09:07:03 stephenfin rmart04: There are two config options of value here: 'reserved_host_memory_mb', which is a total amount of memory to reserve across all NUMA nodes, and 'reserved_huge_pages' which is a per-node reservation that applies when you're using explicit pagesizes
09:07:23 bauzas stephenfin: gibi: fwiw, after the provider series, I was heading to review your vTPM series
09:07:47 stephenfin rmart04: I suggest whacking up the first ('reserved_host_memory_mb'). It would be good to inspect idle memory consumption without instances on the host and use that value with some buffer.
09:08:34 stephenfin rmart04: The second is rarely used. Unless you're using explicit page sizes for all instances, I wouldn't worry about it for now
09:08:43 stephenfin rmart04: Hope that helps
09:08:53 stephenfin bauzas: awesome. Thanks :)
09:13:22 lyarwood stephenfin: just catching up with some downstream stuff but I'll get into your reviews after that
09:16:44 bauzas alex_xu: gibi: I have a logging concern with https://review.opendev.org/#/c/693460/36
09:17:52 bauzas alex_xu: gibi: or we will get a lot of logs if operators don't use this new feature :)
09:18:21 gibi bauzas: you mean https://review.opendev.org/#/c/693460/36/nova/compute/resource_tracker.py@120 ?
09:18:43 gibi bauzas: I see one extra log per compute node at compute service startup
09:19:29 gibi also that config has a default value so it is set even if not specified in the config file
09:19:41 bauzas gibi: correct, this won't be very talkative, but we then try to merge inventories even if the operator doesn't use it
09:19:59 bauzas gibi: https://review.opendev.org/#/c/693460/36/nova/compute/resource_tracker.py@1165
09:20:09 bauzas and https://review.opendev.org/#/c/676029/51/nova/compute/provider_config.py@391
09:20:16 bauzas all of this seems not needed
09:22:09 bauzas gibi: I would have preferred a way to avoid calling all of this if no YAML files were there
09:23:49 gibi Can you check from oslo config if the config value is set explicitly by the user or just the default value is applied by oslo?
09:24:31 gibi because current calling down to get_provider_configs is the only way to detect if there are config files
09:28:38 gibi regarding the actual logging: I can see a reasoning to log what provider config files are loaded if any instead of logging if nothing is loaded
09:39:45 stephenfin real trivial patch for the VMWare driver here https://review.opendev.org/#/c/713565/
09:41:14 openstackgerrit Stephen Finucane proposed openstack/nova master: Move revert resize under semaphore https://review.opendev.org/747746
09:41:15 openstackgerrit Stephen Finucane proposed openstack/nova master: Cleanup 'drop_move_claim' and '_drop_move_claim' https://review.opendev.org/747747
09:41:15 openstackgerrit Stephen Finucane proposed openstack/nova master: manager: Address TODO https://review.opendev.org/747748
09:41:16 openstackgerrit Stephen Finucane proposed openstack/nova master: manager: Move context manager up one level https://review.opendev.org/745674
09:41:16 openstackgerrit Stephen Finucane proposed openstack/nova master: compute: Add type hints for resize functions https://review.opendev.org/745341
09:41:17 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: compute: Add more type hints for resize functions https://review.opendev.org/745675
09:42:22 stephenfin lyarwood: are you respinning that nova-image-download-via-rbd series again today to add the missing '__init__.py' file etc.?
09:45:42 openstackgerrit Merged openstack/nova stable/ussuri: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747972
09:46:35 gibi stephenfin: I left a question in https://review.opendev.org/#/c/741995/10/nova/compute/manager.py@a4932
09:52:11 lyarwood stephenfin: yeah
09:54:47 stephenfin gibi: replied
09:55:54 gibi stephenfin: thanks it is make sense now.
09:57:20 bauzas gibi: sorry, missed your ping
09:57:22 stephenfin gibi: fwiw, that patch is identical to this bug fix for a ResourceTracker race. I just didn't want to make the series depend on each other https://review.opendev.org/#/c/744958/
09:57:55 stephenfin gibi: Have been bugging melwitt and dansmith for reviews on that though so you're off the hook for reviews there :P
09:58:15 gibi stephenfin: if it is identical then I can add my +2 there :)
09:59:00 stephenfin yup, iirc cherry-picked and change-id modified to mark them as independent
10:03:17 bauzas stephenfin: just saw https://review.opendev.org/#/c/747744/
10:03:21 bauzas will try to look at it too
10:04:13 stephenfin nice one
10:06:56 bauzas stephenfin: starting to look at the vTPM series but... wow, I need a beer :p
10:07:31 gibi isn't it a bit too early for a beer? ;)
10:08:05 lyarwood blasphemy
10:08:13 alex_xu stephenfin: checking now
10:08:32 alex_xu bauzas: yea, as gibi said, it is just one log at startup, should be ok
10:08:52 bauzas alex_xu: I changed my thoughts
10:09:08 bauzas I don't wanna hold for a nit
10:09:18 bauzas but... this change needs a rebase either way :)
10:10:42 bauzas gibi: technically, it's beer o'time
10:10:53 bauzas ten past nonn
10:10:56 bauzas noon
10:11:09 bauzas if that's too early, then I don't know when
10:12:45 alex_xu bauzas: yea, that is due to tony_su rebase the previous patch, actually it needn't, just just gate failed, he should recheck instead of rebase.
10:13:40 bauzas alex_xu: the branch I see from gerrit pulls the old revision
10:14:11 bauzas alex_xu: what he can do tho is wait for the dependent change to be merged and then indeed a recheck should be fine
10:16:43 gibi bauzas: I rest my case, it is really afternoon now
10:18:42 openstackgerrit Stephen Finucane proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460
10:18:45 alex_xu bauzas: yea, that is my thought also
10:18:57 stephenfin alex_xu, bauzas: There you go
10:18:58 elod lyarwood: the persist_xml patch has merged in ussuri. what do you think about triggering a release in ussuri? do you mind if i create a release patch?
10:19:09 alex_xu stephenfin: hah, thanks :)
10:19:49 bauzas stephenfin: thanks
10:20:24 elod lyarwood: or should we wait until it gets merged all the way till stein and release ussuri / train / stein together?
10:22:44 openstackgerrit Stephen Finucane proposed openstack/nova master: Deprecate filters that have been replaced by placement filters https://review.opendev.org/745605
10:23:36 gibi bauzas: enjoy
10:23:56 lyarwood elod: if you have time feel free to create the release change for ussuri now
10:24:09 lyarwood elod: I was going to wait but there's nothing stopping us from releasing as it lands in each branch tbh
10:25:37 elod lyarwood: ok, thanks, doing it now
10:26:05 lyarwood awesome thanks :)
10:35:47 elod lyarwood , gibi : https://review.opendev.org/#/c/748156/
10:37:31 elod seems minor version bump required anyway, but if you see other patch that should listed also that requires minor bump, too, just let me know and I'll update the commit message
10:39:54 gibi elod: +1
10:47:36 openstackgerrit Stephen Finucane proposed openstack/nova master: Deprecate filters that have been replaced by placement filters https://review.opendev.org/745605
10:58:07 rmart04 Hi Stephenfin, thanks for this info. I'm just digesting what you've written. I am currently not using huge pages, but I think what you are saying is that one of the numa nodes ran out of memory, not the entire host. I'm not sure how reserved_host_memory_mb will help when the host has lots of memory available (250GB), i'm guessing we just ran out of memory on one particuar numa node? I should mention, we have
10:58:07 rmart04 one large VM on this host with numa_nodes:2.
10:59:05 stephenfin rmart04: Yes, exactly. We lock memory to a single NUMA node so it the instance tries to allocate memory and there's none on _that node_, you're flat out of luck
11:00:32 stephenfin Does the per-node memory allocation of that very large VM bump up against the per-host node total? Are your DIMMs evenly split between host NUMA nodes?
11:01:45 stephenfin i.e. you've 250GB available in total, but perhaps only a small bit on one particular NUMA node
11:01:58 sean-k-mooney the OOM reaper in the kernel runs per numa node so if a numa node runs out of memory it will start killing process if the kerenl needs to allcoate even if the other node is entirely empty
11:02:12 rmart04 So Host has 768GB split evenly across numa nodes. There is one instance, 750GB in size, and has seen more than half its memory used successfully in the past. I'm not sure how to see how much memory a particular numa node has available? Is that easy to see?
11:02:31 sean-k-mooney if you are using numa but not pinning or hugepages
11:02:41 alex_xu stephenfin: are we ensure the operator finish the pin_cpu_set to dedicated_cpu_set/sahred_cpu_set change before update to V release?
11:02:46 sean-k-mooney then we will place all the vms on numa node 0 and never use the second numa node
11:02:55 sean-k-mooney im working on fixing that this morning
11:03:00 rmart04 cpu_policy is dedicated, so I believed its pinned
11:03:20 alex_xu stephenfin: or the operator can do that change when upgrade U to V?
11:03:33 sean-k-mooney ah in that case we will stack all vms on the first numa node untill we run our of pinned cpus then use the next one
11:04:16 stephenfin alex_xu: A user can continue to use the legacy vcpu_pin_set in U. I'm hoping to drop support for it in V, but I have too many series in flight at the moment so it might slip to W
11:04:19 stephenfin alex_xu: If that's what you mean?
11:04:20 sean-k-mooney alex_xu: i think the plan is to remove vcpu_pin_set in v so they have to upgrade in U
11:04:33 sean-k-mooney ideally they would do it in train
11:04:41 sean-k-mooney but they can do it in U
11:05:42 sean-k-mooney rmart04: you can see it in /sys
11:05:47 sean-k-mooney ill get the path on sec

Earlier   Later