| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-10 | |||
| 14:09:11 | sean-k-mooney | the main issues is that there coudl be a chagne in the assocation during the iterations | |
| 14:09:15 | gibi | no, that is my point, we cannot just combine the results | |
| 14:09:41 | sean-k-mooney | this is the set of sharing resouce providers that share with this host | |
| 14:09:48 | sean-k-mooney | why can we not just combine? | |
| 14:09:57 | gibi | no this is all the RPs that in an aggregate | |
| 14:10:10 | gibi | this is not just a single RP | |
| 14:10:20 | sean-k-mooney | no | |
| 14:10:24 | sean-k-mooney | "/resource_providers?member_of=in:%s&required=%s" % ( | |
| 14:10:26 | sean-k-mooney | aggs, os_traits.MISC_SHARES_VIA_AGGREGATE) | |
| 14:10:46 | sean-k-mooney | this is only the resouce providers in teh aggrate with the os_traits.MISC_SHARES_VIA_AGGREGATE triat | |
| 14:10:50 | sean-k-mooney | so only the sharing providers | |
| 14:10:55 | gibi | but not for a single host | |
| 14:11:13 | gibi | there is no 'this host' filter | |
| 14:11:16 | gibi | in the queru | |
| 14:11:18 | gibi | in the query | |
| 14:11:21 | sean-k-mooney | there does not need to be | |
| 14:11:38 | sean-k-mooney | we are asking for all the sharing provider in the aggrate our host is a member of | |
| 14:11:47 | sean-k-mooney | that what the in: query is doing | |
| 14:12:16 | sean-k-mooney | the agg_uuids var is the list of aggreates the current host is a member of | |
| 14:13:12 | sean-k-mooney | we tyically expect only one RP in an aggrate to have MISC_SHARES_VIA_AGGREGATE althogh that is not strictly required | |
| 14:13:34 | gibi | ahh I misremembered the meaning of 'in:' the query | |
| 14:13:48 | sean-k-mooney | well member_of=in:% | |
| 14:13:48 | gibi | that is not AND, that is OR | |
| 14:14:21 | sean-k-mooney | so member_of is "filter by aggartge" and in is the same meaing as container or in in python | |
| 14:14:42 | sean-k-mooney | i.e. filter by aggrage where the aggreate uuid is in "list of uuids" | |
| 14:15:05 | sean-k-mooney | and we prefilter the aggreate list to only the ones that this host is in | |
| 14:16:19 | sean-k-mooney | im still -1 on sahid patch as is because i woudl like to see a repoducer test and then test coverage for this | |
| 14:16:44 | sean-k-mooney | but i think exteranlly the beahivor is the same but internally there is now pagination | |
| 14:17:16 | sean-k-mooney | i dont think hardcodeing the pagination to 200 uuids however is a good thing | |
| 14:17:31 | sean-k-mooney | sicne that depend on yoru loadbalnacer config | |
| 14:21:03 | sahid | sorry guys I had to switch to a meeting. | |
| 14:21:13 | sahid | fair enough I will add a configurable | |
| 14:24:40 | gibi | yeah I was wrong as I thought we query for PRs that are in _all_ the aggregates but we query for RPs that are in _any_ of the aggregates. I will drop my -1 | |
| 14:25:28 | sean-k-mooney | oh you tought the in was an and not an or just read that | |
| 14:25:39 | sean-k-mooney | ya its an or of all the uuids in the list | |
| 14:26:14 | sean-k-mooney | in is basically AnyOf rather then AllOf | |
| 14:27:12 | gibi | yeah | |
| 14:27:36 | gibi | and as it is an OR I agree that we can do multiple queries and combine the results | |
| 14:28:03 | sean-k-mooney | im kind of ok in the preodics | |
| 14:28:39 | sean-k-mooney | im not sure that is ok for the schduler but the schduler scales the query based on teh guest request | |
| 14:28:44 | sean-k-mooney | not the number of aggrates that exist | |
| 14:28:49 | sean-k-mooney | so i think tis less of a problem there | |
| 15:19:08 | opendevreview | Will Szumski proposed openstack/nova stable/2023.2: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921686 | |
| 15:19:37 | opendevreview | Will Szumski proposed openstack/nova stable/2023.1: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921687 | |
| 15:50:26 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | |
| 16:07:03 | melwitt | dansmith: here's one of the kernel panics I got recently https://fbf109e10bffd8a657b2-ace28e47b1213bdab56305960a7f8056.ssl.cf5.rackcdn.com/919986/2/check/tempest-integrated-compute/4b51a9c/testr_results.html | |
| 16:07:49 | melwitt | and this https://86989508e8a17de7a050-16e5a6d00ae68a8470fe17c7f38ebfca.ssl.cf2.rackcdn.com/921586/1/check/tempest-integrated-compute-rbac-old-defaults/8980965/testr_results.html | |
| 16:08:06 | dansmith | okay yeah that first one looks like what we had seen before | |
| 16:08:18 | dansmith | and the second | |
| 16:10:19 | melwitt | this one also looks like https://5edd60feb751137d1041-26db3adf591a82ec37c96a7d3180086f.ssl.cf5.rackcdn.com/870932/41/check/nova-next/be498dc/testr_results.html | |
| 16:11:00 | melwitt | this one is different https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/testr_results.html | |
| 16:11:42 | melwitt | (soft lock up ^) | |
| 16:12:43 | melwitt | this one is also different https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d1c/919991/4/check/tempest-integrated-compute-enforce-scope-new-defaults/d1c7ab9/testr_results.html | |
| 16:13:04 | sean-k-mooney | the soft lockup seams ot be related to a memroy allcotion | |
| 16:14:05 | sean-k-mooney | i wonder if that is related to the amount of ram we are usign for the vms but its got 128mb | |
| 16:14:34 | sean-k-mooney | so it shoudl not be a proablem | |
| 16:15:14 | sean-k-mooney | 18.269292] Freeing unused decrypted memory: 2036K | |
| 16:15:16 | sean-k-mooney | [ 18.395056] Freeing unused kernel image (initmem) memory: 3244K | |
| 16:15:18 | sean-k-mooney | [ 18.397470] Write protecting the kernel read-only data: 30720k | |
| 16:15:20 | sean-k-mooney | [ 18.419370] Freeing unused kernel image (text/rodata gap) memory: 2036K | |
| 16:15:22 | sean-k-mooney | [ 18.433426] Freeing unused kernel image (rodata/data gap) memory: 1448K | |
| 16:15:30 | sean-k-mooney | it has already freed the ram disk and early allcoations | |
| 16:19:32 | sean-k-mooney | cirros-0.6.2-x86_64-disk.img so that pretty new | |
| 16:19:56 | sean-k-mooney | or at least its using a ubuntu 22.04 kernel | |
| 16:20:27 | sean-k-mooney | it look like eitehr a real out of memory issue or a kernel bug to me | |
| 16:21:12 | sean-k-mooney | 0.6.2 was released in may so we coudl perhaps ask for a 0 | |
| 16:21:28 | sean-k-mooney | 0.6.3 release with the latest hardwer enabeling kernel package? | |
| 17:00:58 | melwitt | it seems like all of the swift put_object failures I see occur in the nova-lvm job on tests that have nothing to do with volumes. I don't get it Jun 10 04:45:27.576181 np0037692500 devstack@g-api.service[86519]: ERROR glance.common.wsgi Got error from Swift: put_object('glance', 'b1cfd7cf-ff23-4b05-be12-48ce3e14d593-00001', ...) failure and no ability to reset contents for reupload.. | |
| 17:01:39 | sean-k-mooney | hum... we dont need swift in that job | |
| 17:01:50 | sean-k-mooney | the nova-lvm jobs is ment to test nova with the lvm storage backend | |
| 17:01:55 | sean-k-mooney | and nova never talks to swift | |
| 17:01:59 | melwitt | that's where glance is uploading images though | |
| 17:02:11 | sean-k-mooney | right but it could and proably should be using the file backend | |
| 17:02:19 | sean-k-mooney | not the swift backend | |
| 17:02:21 | melwitt | oh, yeah | |
| 17:04:48 | dansmith | why "should" ? | |
| 17:07:21 | sean-k-mooney | as in why it should not be using swift? | |
| 17:07:27 | sean-k-mooney | just to keep the job smaller | |
| 17:07:27 | dansmith | yeah | |
| 17:07:42 | sean-k-mooney | its not really imporant to testing the lvm storag backend | |
| 17:07:47 | dansmith | the default config is to use swift though right? | |
| 17:07:51 | sean-k-mooney | no file | |
| 17:08:02 | sean-k-mooney | although maybe that depend on if swift is deployed | |
| 17:08:21 | dansmith | I think if swift is deployed (which it is in the integrated job, AFAIK) then we'll use it | |
| 17:08:47 | sean-k-mooney | swift is deployed by default in devstack although i always turn it off so im used to glance using file | |
| 17:08:51 | dansmith | I don't see that the LVM job is configuring it specifically.. and point being just that the fewer deltas from the base job the easier it is to reason about failures and why they're different | |
| 17:09:04 | sean-k-mooney | or cinder | |
| 17:09:28 | sean-k-mooney | well i would not expect the base job to use swift for glance | |
| 17:09:39 | dansmith | I would unless we don't deploy swift | |
| 17:10:15 | dansmith | I don't see an "enable_service" for the nova-lvm job that is opting into swift, is my point | |
| 17:12:12 | dansmith | okay but maybe integrated-compute does disable it? | |
| 17:12:56 | sean-k-mooney | i dont know i might have actully bee looking at the multi store job not lvm | |
| 17:13:17 | sean-k-mooney | the multi store job does turn it off http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/controller/logs/_.localrc_auto.txt | |
| 17:14:37 | sean-k-mooney | maybe i misread the job name but i dont have a nova-lvm tab open so im kind of confuged why i tought it was nova-lvm | |
| 17:14:46 | dansmith | okay, tbh I thought more of our jobs were running with swift-backed glance since it's the default, but perhaps "most" is not quite right | |
| 17:15:21 | dansmith | https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L151 | |
| 17:15:22 | sean-k-mooney | so i would have personaly tough glance is rarely backed by swift and file or cinder backend woudl be more common | |
| 17:15:38 | dansmith | not me :) | |
| 17:16:05 | dansmith | backing it by file requires shared storage and I think doing so by cinder is rather expensive, IIRC | |
| 17:16:10 | sean-k-mooney | i guess im just used to swift not being deployed in the enviornment i look at | |