Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-06-10
14:11:38 sean-k-mooney we are asking for all the sharing provider in the aggrate our host is a member of
14:11:47 sean-k-mooney that what the in: query is doing
14:12:16 sean-k-mooney the agg_uuids var is the list of aggreates the current host is a member of
14:13:12 sean-k-mooney we tyically expect only one RP in an aggrate to have MISC_SHARES_VIA_AGGREGATE althogh that is not strictly required
14:13:34 gibi ahh I misremembered the meaning of 'in:' the query
14:13:48 gibi that is not AND, that is OR
14:13:48 sean-k-mooney well member_of=in:%
14:14:21 sean-k-mooney so member_of is "filter by aggartge" and in is the same meaing as container or in in python
14:14:42 sean-k-mooney i.e. filter by aggrage where the aggreate uuid is in "list of uuids"
14:15:05 sean-k-mooney and we prefilter the aggreate list to only the ones that this host is in
14:16:19 sean-k-mooney im still -1 on sahid patch as is because i woudl like to see a repoducer test and then test coverage for this
14:16:44 sean-k-mooney but i think exteranlly the beahivor is the same but internally there is now pagination
14:17:16 sean-k-mooney i dont think hardcodeing the pagination to 200 uuids however is a good thing
14:17:31 sean-k-mooney sicne that depend on yoru loadbalnacer config
14:21:03 sahid sorry guys I had to switch to a meeting.
14:21:13 sahid fair enough I will add a configurable
14:24:40 gibi yeah I was wrong as I thought we query for PRs that are in _all_ the aggregates but we query for RPs that are in _any_ of the aggregates. I will drop my -1
14:25:28 sean-k-mooney oh you tought the in was an and not an or just read that
14:25:39 sean-k-mooney ya its an or of all the uuids in the list
14:26:14 sean-k-mooney in is basically AnyOf rather then AllOf
14:27:12 gibi yeah
14:27:36 gibi and as it is an OR I agree that we can do multiple queries and combine the results
14:28:03 sean-k-mooney im kind of ok in the preodics
14:28:39 sean-k-mooney im not sure that is ok for the schduler but the schduler scales the query based on teh guest request
14:28:44 sean-k-mooney not the number of aggrates that exist
14:28:49 sean-k-mooney so i think tis less of a problem there
15:19:08 opendevreview Will Szumski proposed openstack/nova stable/2023.2: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921686
15:19:37 opendevreview Will Szumski proposed openstack/nova stable/2023.1: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921687
15:50:26 opendevreview Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665
16:07:03 melwitt dansmith: here's one of the kernel panics I got recently https://fbf109e10bffd8a657b2-ace28e47b1213bdab56305960a7f8056.ssl.cf5.rackcdn.com/919986/2/check/tempest-integrated-compute/4b51a9c/testr_results.html
16:07:49 melwitt and this https://86989508e8a17de7a050-16e5a6d00ae68a8470fe17c7f38ebfca.ssl.cf2.rackcdn.com/921586/1/check/tempest-integrated-compute-rbac-old-defaults/8980965/testr_results.html
16:08:06 dansmith okay yeah that first one looks like what we had seen before
16:08:18 dansmith and the second
16:10:19 melwitt this one also looks like https://5edd60feb751137d1041-26db3adf591a82ec37c96a7d3180086f.ssl.cf5.rackcdn.com/870932/41/check/nova-next/be498dc/testr_results.html
16:11:00 melwitt this one is different https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/testr_results.html
16:11:42 melwitt (soft lock up ^)
16:12:43 melwitt this one is also different https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d1c/919991/4/check/tempest-integrated-compute-enforce-scope-new-defaults/d1c7ab9/testr_results.html
16:13:04 sean-k-mooney the soft lockup seams ot be related to a memroy allcotion
16:14:05 sean-k-mooney i wonder if that is related to the amount of ram we are usign for the vms but its got 128mb
16:14:34 sean-k-mooney so it shoudl not be a proablem
16:15:14 sean-k-mooney 18.269292] Freeing unused decrypted memory: 2036K
16:15:16 sean-k-mooney [ 18.395056] Freeing unused kernel image (initmem) memory: 3244K
16:15:18 sean-k-mooney [ 18.397470] Write protecting the kernel read-only data: 30720k
16:15:20 sean-k-mooney [ 18.419370] Freeing unused kernel image (text/rodata gap) memory: 2036K
16:15:22 sean-k-mooney [ 18.433426] Freeing unused kernel image (rodata/data gap) memory: 1448K
16:15:30 sean-k-mooney it has already freed the ram disk and early allcoations
16:19:32 sean-k-mooney cirros-0.6.2-x86_64-disk.img so that pretty new
16:19:56 sean-k-mooney or at least its using a ubuntu 22.04 kernel
16:20:27 sean-k-mooney it look like eitehr a real out of memory issue or a kernel bug to me
16:21:12 sean-k-mooney 0.6.2 was released in may so we coudl perhaps ask for a 0
16:21:28 sean-k-mooney 0.6.3 release with the latest hardwer enabeling kernel package?
17:00:58 melwitt it seems like all of the swift put_object failures I see occur in the nova-lvm job on tests that have nothing to do with volumes. I don't get it Jun 10 04:45:27.576181 np0037692500 devstack@g-api.service[86519]: ERROR glance.common.wsgi Got error from Swift: put_object('glance', 'b1cfd7cf-ff23-4b05-be12-48ce3e14d593-00001', ...) failure and no ability to reset contents for reupload..
17:01:39 sean-k-mooney hum... we dont need swift in that job
17:01:50 sean-k-mooney the nova-lvm jobs is ment to test nova with the lvm storage backend
17:01:55 sean-k-mooney and nova never talks to swift
17:01:59 melwitt that's where glance is uploading images though
17:02:11 sean-k-mooney right but it could and proably should be using the file backend
17:02:19 sean-k-mooney not the swift backend
17:02:21 melwitt oh, yeah
17:04:48 dansmith why "should" ?
17:07:21 sean-k-mooney as in why it should not be using swift?
17:07:27 dansmith yeah
17:07:27 sean-k-mooney just to keep the job smaller
17:07:42 sean-k-mooney its not really imporant to testing the lvm storag backend
17:07:47 dansmith the default config is to use swift though right?
17:07:51 sean-k-mooney no file
17:08:02 sean-k-mooney although maybe that depend on if swift is deployed
17:08:21 dansmith I think if swift is deployed (which it is in the integrated job, AFAIK) then we'll use it
17:08:47 sean-k-mooney swift is deployed by default in devstack although i always turn it off so im used to glance using file
17:08:51 dansmith I don't see that the LVM job is configuring it specifically.. and point being just that the fewer deltas from the base job the easier it is to reason about failures and why they're different
17:09:04 sean-k-mooney or cinder
17:09:28 sean-k-mooney well i would not expect the base job to use swift for glance
17:09:39 dansmith I would unless we don't deploy swift
17:10:15 dansmith I don't see an "enable_service" for the nova-lvm job that is opting into swift, is my point
17:12:12 dansmith okay but maybe integrated-compute does disable it?
17:12:56 sean-k-mooney i dont know i might have actully bee looking at the multi store job not lvm
17:13:17 sean-k-mooney the multi store job does turn it off http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/controller/logs/_.localrc_auto.txt
17:14:37 sean-k-mooney maybe i misread the job name but i dont have a nova-lvm tab open so im kind of confuged why i tought it was nova-lvm
17:14:46 dansmith okay, tbh I thought more of our jobs were running with swift-backed glance since it's the default, but perhaps "most" is not quite right
17:15:21 dansmith https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L151
17:15:22 sean-k-mooney so i would have personaly tough glance is rarely backed by swift and file or cinder backend woudl be more common
17:15:38 dansmith not me :)
17:16:05 dansmith backing it by file requires shared storage and I think doing so by cinder is rather expensive, IIRC
17:16:10 sean-k-mooney i guess im just used to swift not being deployed in the enviornment i look at
17:17:44 sean-k-mooney actully we were talking about timeout an dthe integreated compute job and its deritives
17:17:47 sean-k-mooney https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L141
17:17:53 sean-k-mooney looks like its already reduced to 4
17:18:42 sean-k-mooney i guess we clampped it to 4 because of failure when we rased the default for the rest to 6
17:22:47 sean-k-mooney dansmith: looking at a clople of the tiemout they are all on rax but spread across 3 datacenters rax-iad, rax-dfw and rax-ord
17:30:31 sean-k-mooney dansmith: https://tinyurl.com/3bf6nhkh
17:31:31 sean-k-mooney actull i think i need to grop by somethign
17:31:37 sean-k-mooney so that i only get on ehit per job
17:32:14 dansmith yeah, it wouldn't surprise me at all if rax is the oldest/slowest of the providers
17:32:25 dansmith maybe we could group only smaller jobs to run on rax
17:32:41 sean-k-mooney they also run our vms on xen not kvm i belive
17:38:08 sean-k-mooney ok this is better https://tinyurl.com/35dfcfcy
17:38:37 sean-k-mooney only 72 hits mostly rax
17:39:10 sean-k-mooney what would be useful is if we could break down the average job runtime by provider
#openstack-nova - 2024-06-11
06:05:18 hamidlotfi_ Hi there,
06:05:18 hamidlotfi_ An instance whose volume is on CEPH, after abnormally restarting the Compute server where that instance is located, that instance does not come up and displays the following error:

Earlier   Later