Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-06-10
16:13:04 sean-k-mooney the soft lockup seams ot be related to a memroy allcotion
16:14:05 sean-k-mooney i wonder if that is related to the amount of ram we are usign for the vms but its got 128mb
16:14:34 sean-k-mooney so it shoudl not be a proablem
16:15:14 sean-k-mooney 18.269292] Freeing unused decrypted memory: 2036K
16:15:16 sean-k-mooney [ 18.395056] Freeing unused kernel image (initmem) memory: 3244K
16:15:18 sean-k-mooney [ 18.397470] Write protecting the kernel read-only data: 30720k
16:15:20 sean-k-mooney [ 18.419370] Freeing unused kernel image (text/rodata gap) memory: 2036K
16:15:22 sean-k-mooney [ 18.433426] Freeing unused kernel image (rodata/data gap) memory: 1448K
16:15:30 sean-k-mooney it has already freed the ram disk and early allcoations
16:19:32 sean-k-mooney cirros-0.6.2-x86_64-disk.img so that pretty new
16:19:56 sean-k-mooney or at least its using a ubuntu 22.04 kernel
16:20:27 sean-k-mooney it look like eitehr a real out of memory issue or a kernel bug to me
16:21:12 sean-k-mooney 0.6.2 was released in may so we coudl perhaps ask for a 0
16:21:28 sean-k-mooney 0.6.3 release with the latest hardwer enabeling kernel package?
17:00:58 melwitt it seems like all of the swift put_object failures I see occur in the nova-lvm job on tests that have nothing to do with volumes. I don't get it Jun 10 04:45:27.576181 np0037692500 devstack@g-api.service[86519]: ERROR glance.common.wsgi Got error from Swift: put_object('glance', 'b1cfd7cf-ff23-4b05-be12-48ce3e14d593-00001', ...) failure and no ability to reset contents for reupload..
17:01:39 sean-k-mooney hum... we dont need swift in that job
17:01:50 sean-k-mooney the nova-lvm jobs is ment to test nova with the lvm storage backend
17:01:55 sean-k-mooney and nova never talks to swift
17:01:59 melwitt that's where glance is uploading images though
17:02:11 sean-k-mooney right but it could and proably should be using the file backend
17:02:19 sean-k-mooney not the swift backend
17:02:21 melwitt oh, yeah
17:04:48 dansmith why "should" ?
17:07:21 sean-k-mooney as in why it should not be using swift?
17:07:27 dansmith yeah
17:07:27 sean-k-mooney just to keep the job smaller
17:07:42 sean-k-mooney its not really imporant to testing the lvm storag backend
17:07:47 dansmith the default config is to use swift though right?
17:07:51 sean-k-mooney no file
17:08:02 sean-k-mooney although maybe that depend on if swift is deployed
17:08:21 dansmith I think if swift is deployed (which it is in the integrated job, AFAIK) then we'll use it
17:08:47 sean-k-mooney swift is deployed by default in devstack although i always turn it off so im used to glance using file
17:08:51 dansmith I don't see that the LVM job is configuring it specifically.. and point being just that the fewer deltas from the base job the easier it is to reason about failures and why they're different
17:09:04 sean-k-mooney or cinder
17:09:28 sean-k-mooney well i would not expect the base job to use swift for glance
17:09:39 dansmith I would unless we don't deploy swift
17:10:15 dansmith I don't see an "enable_service" for the nova-lvm job that is opting into swift, is my point
17:12:12 dansmith okay but maybe integrated-compute does disable it?
17:12:56 sean-k-mooney i dont know i might have actully bee looking at the multi store job not lvm
17:13:17 sean-k-mooney the multi store job does turn it off http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/controller/logs/_.localrc_auto.txt
17:14:37 sean-k-mooney maybe i misread the job name but i dont have a nova-lvm tab open so im kind of confuged why i tought it was nova-lvm
17:14:46 dansmith okay, tbh I thought more of our jobs were running with swift-backed glance since it's the default, but perhaps "most" is not quite right
17:15:21 dansmith https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L151
17:15:22 sean-k-mooney so i would have personaly tough glance is rarely backed by swift and file or cinder backend woudl be more common
17:15:38 dansmith not me :)
17:16:05 dansmith backing it by file requires shared storage and I think doing so by cinder is rather expensive, IIRC
17:16:10 sean-k-mooney i guess im just used to swift not being deployed in the enviornment i look at
17:17:44 sean-k-mooney actully we were talking about timeout an dthe integreated compute job and its deritives
17:17:47 sean-k-mooney https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L141
17:17:53 sean-k-mooney looks like its already reduced to 4
17:18:42 sean-k-mooney i guess we clampped it to 4 because of failure when we rased the default for the rest to 6
17:22:47 sean-k-mooney dansmith: looking at a clople of the tiemout they are all on rax but spread across 3 datacenters rax-iad, rax-dfw and rax-ord
17:30:31 sean-k-mooney dansmith: https://tinyurl.com/3bf6nhkh
17:31:31 sean-k-mooney actull i think i need to grop by somethign
17:31:37 sean-k-mooney so that i only get on ehit per job
17:32:14 dansmith yeah, it wouldn't surprise me at all if rax is the oldest/slowest of the providers
17:32:25 dansmith maybe we could group only smaller jobs to run on rax
17:32:41 sean-k-mooney they also run our vms on xen not kvm i belive
17:38:08 sean-k-mooney ok this is better https://tinyurl.com/35dfcfcy
17:38:37 sean-k-mooney only 72 hits mostly rax
17:39:10 sean-k-mooney what would be useful is if we could break down the average job runtime by provider
#openstack-nova - 2024-06-11
06:05:18 hamidlotfi_ Hi there,
06:05:18 hamidlotfi_ An instance whose volume is on CEPH, after abnormally restarting the Compute server where that instance is located, that instance does not come up and displays the following error:
06:05:22 hamidlotfi_ https://www.irccloud.com/pastebin/n5z4qgB7/
06:05:27 hamidlotfi_ I checked all of those values for the disk_cachemodes parameter,
06:05:44 hamidlotfi_ network=directsync
06:05:44 hamidlotfi_ network=none
06:05:44 hamidlotfi_ network=writethrough
06:05:44 hamidlotfi_ but in all of the values, the result is the same and I had that error.
08:09:51 opendevreview Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665
11:16:56 sean-k-mooney gibi: can you review https://review.opendev.org/c/openstack/nova/+/710848 and the proceedign repoducer again
11:17:38 sean-k-mooney when you have time of course but this has been arrond for a long time and if you agree i would like to back port that to stable branches
12:09:25 gibi added to my queue
13:15:02 WJeffs Hey, I might be being blind, but is there any options for Numa placement when allocating them to a VM. Say we have a HV with 8 numa groups across 2 cpus, and I want to allocate all the cores from 2 groups, thats no issue, but it gives me 1+6, but is there a way we can make it always use numa on same physical cpu at all times?
13:59:06 sean-k-mooney WJeffs: form the same cpu package no. if you have a numa affined guest and ask for 2 numa nodes we will either pack or balance the vm across the numa node depenign on a config option
13:59:39 sean-k-mooney WJeffs: in the past the beahvior was to pack leading to poor performance we now default to spreading
14:00:12 sean-k-mooney WJeffs: we did have a request ot supprot socket level optimiations for amd cpus in the past
14:00:21 sean-k-mooney but we have not had time to implemtn that
15:00:51 WJeffs sean-k-mooney: ah it is for AMD Genoa, I guess that is the same part, we are already creating the numa configuration inside the VM and that works great and gives us good performance, now we are trying to test some extra configurations and found that the numas are spread across sockets
15:01:20 WJeffs I'll have a look for the request/blueprint and see if there is anything we can do to help on this.
15:05:22 sean-k-mooney WJeffs: this is basicaly all contolled by https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2449-L2511 currently
15:05:47 sean-k-mooney to do what you want we would need to add the socket info int the numa toplogy
15:06:10 sean-k-mooney and then add an addtional sort here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2481
15:06:18 sean-k-mooney to sort the nodes by socket
15:07:37 sean-k-mooney sorry not there just below it https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2488
15:07:43 sean-k-mooney the les is for pinned cpus
15:38:27 WJeffs sean-k-mooney: Thanks, I'll take a look and see if there is something can do to help
15:40:00 sean-k-mooney the main gap right now is we dont track the socket id in the host numa topogy data stucture
15:40:07 sean-k-mooney or the numa distance btween numa ndoes
15:40:13 sean-k-mooney since that really waht you want to minimise
15:40:34 sean-k-mooney libvirt provide both we jsut dont collect it right now
15:41:00 WJeffs yea, thats the main thing we are trying to do. So its collect from libvirt -> use that data that needs to be added.
15:41:29 sean-k-mooney ya so artom looked at this brifly a year or two ago
15:42:00 sean-k-mooney we have a downstream tracker in our backlog to eventually look at this https://issues.redhat.com/browse/OSPRH-36
15:42:15 sean-k-mooney but we jsut didnt have time or custoemr ask to priories it
15:42:21 artom I even have/had WIP patches up, at some point
15:42:35 artom Just the basics to collect those distance from libvirt
15:42:35 sean-k-mooney https://review.opendev.org/q/topic:%22bp/libvirt-smarter-cpu-placement%22
15:42:39 sean-k-mooney yep ^
15:43:06 sean-k-mooney there in merge conflict but its proably pretty trivial

Earlier   Later