Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-26
09:41:16 openstackgerrit Stephen Finucane proposed openstack/nova master: compute: Add type hints for resize functions https://review.opendev.org/745341
09:41:16 openstackgerrit Stephen Finucane proposed openstack/nova master: manager: Move context manager up one level https://review.opendev.org/745674
09:41:17 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: compute: Add more type hints for resize functions https://review.opendev.org/745675
09:42:22 stephenfin lyarwood: are you respinning that nova-image-download-via-rbd series again today to add the missing '__init__.py' file etc.?
09:45:42 openstackgerrit Merged openstack/nova stable/ussuri: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747972
09:46:35 gibi stephenfin: I left a question in https://review.opendev.org/#/c/741995/10/nova/compute/manager.py@a4932
09:52:11 lyarwood stephenfin: yeah
09:54:47 stephenfin gibi: replied
09:55:54 gibi stephenfin: thanks it is make sense now.
09:57:20 bauzas gibi: sorry, missed your ping
09:57:22 stephenfin gibi: fwiw, that patch is identical to this bug fix for a ResourceTracker race. I just didn't want to make the series depend on each other https://review.opendev.org/#/c/744958/
09:57:55 stephenfin gibi: Have been bugging melwitt and dansmith for reviews on that though so you're off the hook for reviews there :P
09:58:15 gibi stephenfin: if it is identical then I can add my +2 there :)
09:59:00 stephenfin yup, iirc cherry-picked and change-id modified to mark them as independent
10:03:17 bauzas stephenfin: just saw https://review.opendev.org/#/c/747744/
10:03:21 bauzas will try to look at it too
10:04:13 stephenfin nice one
10:06:56 bauzas stephenfin: starting to look at the vTPM series but... wow, I need a beer :p
10:07:31 gibi isn't it a bit too early for a beer? ;)
10:08:05 lyarwood blasphemy
10:08:13 alex_xu stephenfin: checking now
10:08:32 alex_xu bauzas: yea, as gibi said, it is just one log at startup, should be ok
10:08:52 bauzas alex_xu: I changed my thoughts
10:09:08 bauzas I don't wanna hold for a nit
10:09:18 bauzas but... this change needs a rebase either way :)
10:10:42 bauzas gibi: technically, it's beer o'time
10:10:53 bauzas ten past nonn
10:10:56 bauzas noon
10:11:09 bauzas if that's too early, then I don't know when
10:12:45 alex_xu bauzas: yea, that is due to tony_su rebase the previous patch, actually it needn't, just just gate failed, he should recheck instead of rebase.
10:13:40 bauzas alex_xu: the branch I see from gerrit pulls the old revision
10:14:11 bauzas alex_xu: what he can do tho is wait for the dependent change to be merged and then indeed a recheck should be fine
10:16:43 gibi bauzas: I rest my case, it is really afternoon now
10:18:42 openstackgerrit Stephen Finucane proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460
10:18:45 alex_xu bauzas: yea, that is my thought also
10:18:57 stephenfin alex_xu, bauzas: There you go
10:18:58 elod lyarwood: the persist_xml patch has merged in ussuri. what do you think about triggering a release in ussuri? do you mind if i create a release patch?
10:19:09 alex_xu stephenfin: hah, thanks :)
10:19:49 bauzas stephenfin: thanks
10:20:24 elod lyarwood: or should we wait until it gets merged all the way till stein and release ussuri / train / stein together?
10:22:44 openstackgerrit Stephen Finucane proposed openstack/nova master: Deprecate filters that have been replaced by placement filters https://review.opendev.org/745605
10:23:36 gibi bauzas: enjoy
10:23:56 lyarwood elod: if you have time feel free to create the release change for ussuri now
10:24:09 lyarwood elod: I was going to wait but there's nothing stopping us from releasing as it lands in each branch tbh
10:25:37 elod lyarwood: ok, thanks, doing it now
10:26:05 lyarwood awesome thanks :)
10:35:47 elod lyarwood , gibi : https://review.opendev.org/#/c/748156/
10:37:31 elod seems minor version bump required anyway, but if you see other patch that should listed also that requires minor bump, too, just let me know and I'll update the commit message
10:39:54 gibi elod: +1
10:47:36 openstackgerrit Stephen Finucane proposed openstack/nova master: Deprecate filters that have been replaced by placement filters https://review.opendev.org/745605
10:58:07 rmart04 one large VM on this host with numa_nodes:2.
10:58:07 rmart04 Hi Stephenfin, thanks for this info. I'm just digesting what you've written. I am currently not using huge pages, but I think what you are saying is that one of the numa nodes ran out of memory, not the entire host. I'm not sure how reserved_host_memory_mb will help when the host has lots of memory available (250GB), i'm guessing we just ran out of memory on one particuar numa node? I should mention, we have
10:59:05 stephenfin rmart04: Yes, exactly. We lock memory to a single NUMA node so it the instance tries to allocate memory and there's none on _that node_, you're flat out of luck
11:00:32 stephenfin Does the per-node memory allocation of that very large VM bump up against the per-host node total? Are your DIMMs evenly split between host NUMA nodes?
11:01:45 stephenfin i.e. you've 250GB available in total, but perhaps only a small bit on one particular NUMA node
11:01:58 sean-k-mooney the OOM reaper in the kernel runs per numa node so if a numa node runs out of memory it will start killing process if the kerenl needs to allcoate even if the other node is entirely empty
11:02:12 rmart04 So Host has 768GB split evenly across numa nodes. There is one instance, 750GB in size, and has seen more than half its memory used successfully in the past. I'm not sure how to see how much memory a particular numa node has available? Is that easy to see?
11:02:31 sean-k-mooney if you are using numa but not pinning or hugepages
11:02:41 alex_xu stephenfin: are we ensure the operator finish the pin_cpu_set to dedicated_cpu_set/sahred_cpu_set change before update to V release?
11:02:46 sean-k-mooney then we will place all the vms on numa node 0 and never use the second numa node
11:02:55 sean-k-mooney im working on fixing that this morning
11:03:00 rmart04 cpu_policy is dedicated, so I believed its pinned
11:03:20 alex_xu stephenfin: or the operator can do that change when upgrade U to V?
11:03:33 sean-k-mooney ah in that case we will stack all vms on the first numa node untill we run our of pinned cpus then use the next one
11:04:16 stephenfin alex_xu: A user can continue to use the legacy vcpu_pin_set in U. I'm hoping to drop support for it in V, but I have too many series in flight at the moment so it might slip to W
11:04:19 stephenfin alex_xu: If that's what you mean?
11:04:20 sean-k-mooney alex_xu: i think the plan is to remove vcpu_pin_set in v so they have to upgrade in U
11:04:33 sean-k-mooney ideally they would do it in train
11:04:41 sean-k-mooney but they can do it in U
11:05:42 sean-k-mooney rmart04: you can see it in /sys
11:05:47 sean-k-mooney ill get the path on sec
11:06:20 alex_xu stephenfin: sean-k-mooney ,I'm thinking of 744021 may stop the upgrade from U to V if the operate transft from legacy to new config in that time
11:06:53 sean-k-mooney rmart04: cat /sys/bus/node/devices/node*/meminfo
11:07:31 alex_xu if we force that legacy config change finished in U, then it should be fine
11:08:09 sean-k-mooney alex_xu: its there because we had customer upgraind into an invlaid config
11:08:13 stephenfin alex_xu: I don't understand the issue
11:08:41 sean-k-mooney they had vms with the old configuration where they were using isolate but the new config options
11:08:44 stephenfin A user should never get into a situation where they have new-style config but a pinned instance is consuming VCPU instead of PCPU
11:08:48 sean-k-mooney so they were using vcpus instead of pcpus
11:09:02 sean-k-mooney even thogh the host reported pcpus
11:09:10 stephenfin This patch should be backported to Ussuri and Train
11:09:26 sean-k-mooney yes
11:10:14 sean-k-mooney alex_xu: it was an edgecase we forgot to block, we taught it would be blocked by the numa toplogy filter but the check was missing
11:10:14 stephenfin Is there something upgrade specific I'm missing, perhaps?
11:10:32 alex_xu if the host with legacy config is running isolated thread instance, but the 744021 stop the isolated thread instance? (I'm still try to ramp up my mind)
11:10:53 sean-k-mooney alex_xu: if its using legacy configs it will allow it to boot
11:11:01 sean-k-mooney alex_xu: it only changes the behavior for new configs
11:11:43 alex_xu oh, right
11:12:14 sean-k-mooney alex_xu: we dont allow the shared and dedicated set to overlap so the only time they contian the same values is if we are using the legacy config options
11:12:51 sean-k-mooney so and host_cell.pcpuset != host_cell.cpuset
11:13:05 sean-k-mooney is an indirect check for is the host using new config options
11:13:08 lyarwood elod: sorry missed that https://review.opendev.org/#/c/747357/ wasn't included
11:13:11 sean-k-mooney that will also work in the schduler
11:13:14 lyarwood elod: can we wait for that and then release?
11:13:23 sean-k-mooney since we cant actully check the config directly there
11:14:33 jsuchome lyarwood: gibi stephenfin https://review.opendev.org/#/c/574301/ finally green again, any further changes necessary?
11:15:50 elod lyarwood: oh, of course, sorry, I forgot to check what else on the queue :S
11:16:30 openstackgerrit Merged openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522
11:17:00 lyarwood jsuchome: https://review.opendev.org/#/c/746904/ - I just need to respin this after lunch and I think we should be good to go
11:17:43 alex_xu sean-k-mooney: but with new config, why we meet the case fallback to query VCPU?

Earlier   Later