| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-10 | |||
| 14:03:33 | sean-k-mooney | yes i know | |
| 14:03:59 | sean-k-mooney | but but what im saying is we shoudl not be calling that when generating the allcoation candiate request | |
| 14:04:55 | sean-k-mooney | but your right that we have to be carful which is why i was orginally pushing back on shaids approch. but if it only affecting the resouce-tracker cache that less fo a concern | |
| 14:06:33 | gibi | even if it is only used in the tracker we cannot allow to have wrong data in the tracker, maybe tomorrow a new feature will start using that data. | |
| 14:06:47 | sean-k-mooney | sure | |
| 14:07:01 | sean-k-mooney | but i never said we shoudl | |
| 14:07:03 | gibi | so I'm hard -1 in the current version of the patch | |
| 14:08:00 | gibi | I guess I'm not arguing against you but against the patch itself :) | |
| 14:08:02 | sean-k-mooney | so the current verison of the patch should not chagne the behaivor | |
| 14:08:19 | sean-k-mooney | either we would loop over all the aggreates and return the comined set | |
| 14:08:39 | sean-k-mooney | or raise the same excption we raised previosusly | |
| 14:09:11 | sean-k-mooney | the main issues is that there coudl be a chagne in the assocation during the iterations | |
| 14:09:15 | gibi | no, that is my point, we cannot just combine the results | |
| 14:09:41 | sean-k-mooney | this is the set of sharing resouce providers that share with this host | |
| 14:09:48 | sean-k-mooney | why can we not just combine? | |
| 14:09:57 | gibi | no this is all the RPs that in an aggregate | |
| 14:10:10 | gibi | this is not just a single RP | |
| 14:10:20 | sean-k-mooney | no | |
| 14:10:24 | sean-k-mooney | "/resource_providers?member_of=in:%s&required=%s" % ( | |
| 14:10:26 | sean-k-mooney | aggs, os_traits.MISC_SHARES_VIA_AGGREGATE) | |
| 14:10:46 | sean-k-mooney | this is only the resouce providers in teh aggrate with the os_traits.MISC_SHARES_VIA_AGGREGATE triat | |
| 14:10:50 | sean-k-mooney | so only the sharing providers | |
| 14:10:55 | gibi | but not for a single host | |
| 14:11:13 | gibi | there is no 'this host' filter | |
| 14:11:16 | gibi | in the queru | |
| 14:11:18 | gibi | in the query | |
| 14:11:21 | sean-k-mooney | there does not need to be | |
| 14:11:38 | sean-k-mooney | we are asking for all the sharing provider in the aggrate our host is a member of | |
| 14:11:47 | sean-k-mooney | that what the in: query is doing | |
| 14:12:16 | sean-k-mooney | the agg_uuids var is the list of aggreates the current host is a member of | |
| 14:13:12 | sean-k-mooney | we tyically expect only one RP in an aggrate to have MISC_SHARES_VIA_AGGREGATE althogh that is not strictly required | |
| 14:13:34 | gibi | ahh I misremembered the meaning of 'in:' the query | |
| 14:13:48 | sean-k-mooney | well member_of=in:% | |
| 14:13:48 | gibi | that is not AND, that is OR | |
| 14:14:21 | sean-k-mooney | so member_of is "filter by aggartge" and in is the same meaing as container or in in python | |
| 14:14:42 | sean-k-mooney | i.e. filter by aggrage where the aggreate uuid is in "list of uuids" | |
| 14:15:05 | sean-k-mooney | and we prefilter the aggreate list to only the ones that this host is in | |
| 14:16:19 | sean-k-mooney | im still -1 on sahid patch as is because i woudl like to see a repoducer test and then test coverage for this | |
| 14:16:44 | sean-k-mooney | but i think exteranlly the beahivor is the same but internally there is now pagination | |
| 14:17:16 | sean-k-mooney | i dont think hardcodeing the pagination to 200 uuids however is a good thing | |
| 14:17:31 | sean-k-mooney | sicne that depend on yoru loadbalnacer config | |
| 14:21:03 | sahid | sorry guys I had to switch to a meeting. | |
| 14:21:13 | sahid | fair enough I will add a configurable | |
| 14:24:40 | gibi | yeah I was wrong as I thought we query for PRs that are in _all_ the aggregates but we query for RPs that are in _any_ of the aggregates. I will drop my -1 | |
| 14:25:28 | sean-k-mooney | oh you tought the in was an and not an or just read that | |
| 14:25:39 | sean-k-mooney | ya its an or of all the uuids in the list | |
| 14:26:14 | sean-k-mooney | in is basically AnyOf rather then AllOf | |
| 14:27:12 | gibi | yeah | |
| 14:27:36 | gibi | and as it is an OR I agree that we can do multiple queries and combine the results | |
| 14:28:03 | sean-k-mooney | im kind of ok in the preodics | |
| 14:28:39 | sean-k-mooney | im not sure that is ok for the schduler but the schduler scales the query based on teh guest request | |
| 14:28:44 | sean-k-mooney | not the number of aggrates that exist | |
| 14:28:49 | sean-k-mooney | so i think tis less of a problem there | |
| 15:19:08 | opendevreview | Will Szumski proposed openstack/nova stable/2023.2: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921686 | |
| 15:19:37 | opendevreview | Will Szumski proposed openstack/nova stable/2023.1: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921687 | |
| 15:50:26 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | |
| 16:07:03 | melwitt | dansmith: here's one of the kernel panics I got recently https://fbf109e10bffd8a657b2-ace28e47b1213bdab56305960a7f8056.ssl.cf5.rackcdn.com/919986/2/check/tempest-integrated-compute/4b51a9c/testr_results.html | |
| 16:07:49 | melwitt | and this https://86989508e8a17de7a050-16e5a6d00ae68a8470fe17c7f38ebfca.ssl.cf2.rackcdn.com/921586/1/check/tempest-integrated-compute-rbac-old-defaults/8980965/testr_results.html | |
| 16:08:06 | dansmith | okay yeah that first one looks like what we had seen before | |
| 16:08:18 | dansmith | and the second | |
| 16:10:19 | melwitt | this one also looks like https://5edd60feb751137d1041-26db3adf591a82ec37c96a7d3180086f.ssl.cf5.rackcdn.com/870932/41/check/nova-next/be498dc/testr_results.html | |
| 16:11:00 | melwitt | this one is different https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/testr_results.html | |
| 16:11:42 | melwitt | (soft lock up ^) | |
| 16:12:43 | melwitt | this one is also different https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d1c/919991/4/check/tempest-integrated-compute-enforce-scope-new-defaults/d1c7ab9/testr_results.html | |
| 16:13:04 | sean-k-mooney | the soft lockup seams ot be related to a memroy allcotion | |
| 16:14:05 | sean-k-mooney | i wonder if that is related to the amount of ram we are usign for the vms but its got 128mb | |
| 16:14:34 | sean-k-mooney | so it shoudl not be a proablem | |
| 16:15:14 | sean-k-mooney | 18.269292] Freeing unused decrypted memory: 2036K | |
| 16:15:16 | sean-k-mooney | [ 18.395056] Freeing unused kernel image (initmem) memory: 3244K | |
| 16:15:18 | sean-k-mooney | [ 18.397470] Write protecting the kernel read-only data: 30720k | |
| 16:15:20 | sean-k-mooney | [ 18.419370] Freeing unused kernel image (text/rodata gap) memory: 2036K | |
| 16:15:22 | sean-k-mooney | [ 18.433426] Freeing unused kernel image (rodata/data gap) memory: 1448K | |
| 16:15:30 | sean-k-mooney | it has already freed the ram disk and early allcoations | |
| 16:19:32 | sean-k-mooney | cirros-0.6.2-x86_64-disk.img so that pretty new | |
| 16:19:56 | sean-k-mooney | or at least its using a ubuntu 22.04 kernel | |
| 16:20:27 | sean-k-mooney | it look like eitehr a real out of memory issue or a kernel bug to me | |
| 16:21:12 | sean-k-mooney | 0.6.2 was released in may so we coudl perhaps ask for a 0 | |
| 16:21:28 | sean-k-mooney | 0.6.3 release with the latest hardwer enabeling kernel package? | |
| 17:00:58 | melwitt | it seems like all of the swift put_object failures I see occur in the nova-lvm job on tests that have nothing to do with volumes. I don't get it Jun 10 04:45:27.576181 np0037692500 devstack@g-api.service[86519]: ERROR glance.common.wsgi Got error from Swift: put_object('glance', 'b1cfd7cf-ff23-4b05-be12-48ce3e14d593-00001', ...) failure and no ability to reset contents for reupload.. | |
| 17:01:39 | sean-k-mooney | hum... we dont need swift in that job | |
| 17:01:50 | sean-k-mooney | the nova-lvm jobs is ment to test nova with the lvm storage backend | |
| 17:01:55 | sean-k-mooney | and nova never talks to swift | |
| 17:01:59 | melwitt | that's where glance is uploading images though | |
| 17:02:11 | sean-k-mooney | right but it could and proably should be using the file backend | |
| 17:02:19 | sean-k-mooney | not the swift backend | |
| 17:02:21 | melwitt | oh, yeah | |
| 17:04:48 | dansmith | why "should" ? | |
| 17:07:21 | sean-k-mooney | as in why it should not be using swift? | |
| 17:07:27 | sean-k-mooney | just to keep the job smaller | |
| 17:07:27 | dansmith | yeah | |
| 17:07:42 | sean-k-mooney | its not really imporant to testing the lvm storag backend | |
| 17:07:47 | dansmith | the default config is to use swift though right? | |
| 17:07:51 | sean-k-mooney | no file | |
| 17:08:02 | sean-k-mooney | although maybe that depend on if swift is deployed | |
| 17:08:21 | dansmith | I think if swift is deployed (which it is in the integrated job, AFAIK) then we'll use it | |
| 17:08:47 | sean-k-mooney | swift is deployed by default in devstack although i always turn it off so im used to glance using file | |
| 17:08:51 | dansmith | I don't see that the LVM job is configuring it specifically.. and point being just that the fewer deltas from the base job the easier it is to reason about failures and why they're different | |
| 17:09:04 | sean-k-mooney | or cinder | |
| 17:09:28 | sean-k-mooney | well i would not expect the base job to use swift for glance | |
| 17:09:39 | dansmith | I would unless we don't deploy swift | |