| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-10 | |||
| 14:28:44 | sean-k-mooney | not the number of aggrates that exist | |
| 14:28:49 | sean-k-mooney | so i think tis less of a problem there | |
| 15:19:08 | opendevreview | Will Szumski proposed openstack/nova stable/2023.2: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921686 | |
| 15:19:37 | opendevreview | Will Szumski proposed openstack/nova stable/2023.1: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921687 | |
| 15:50:26 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | |
| 16:07:03 | melwitt | dansmith: here's one of the kernel panics I got recently https://fbf109e10bffd8a657b2-ace28e47b1213bdab56305960a7f8056.ssl.cf5.rackcdn.com/919986/2/check/tempest-integrated-compute/4b51a9c/testr_results.html | |
| 16:07:49 | melwitt | and this https://86989508e8a17de7a050-16e5a6d00ae68a8470fe17c7f38ebfca.ssl.cf2.rackcdn.com/921586/1/check/tempest-integrated-compute-rbac-old-defaults/8980965/testr_results.html | |
| 16:08:06 | dansmith | okay yeah that first one looks like what we had seen before | |
| 16:08:18 | dansmith | and the second | |
| 16:10:19 | melwitt | this one also looks like https://5edd60feb751137d1041-26db3adf591a82ec37c96a7d3180086f.ssl.cf5.rackcdn.com/870932/41/check/nova-next/be498dc/testr_results.html | |
| 16:11:00 | melwitt | this one is different https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/testr_results.html | |
| 16:11:42 | melwitt | (soft lock up ^) | |
| 16:12:43 | melwitt | this one is also different https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d1c/919991/4/check/tempest-integrated-compute-enforce-scope-new-defaults/d1c7ab9/testr_results.html | |
| 16:13:04 | sean-k-mooney | the soft lockup seams ot be related to a memroy allcotion | |
| 16:14:05 | sean-k-mooney | i wonder if that is related to the amount of ram we are usign for the vms but its got 128mb | |
| 16:14:34 | sean-k-mooney | so it shoudl not be a proablem | |
| 16:15:14 | sean-k-mooney | 18.269292] Freeing unused decrypted memory: 2036K | |
| 16:15:16 | sean-k-mooney | [ 18.395056] Freeing unused kernel image (initmem) memory: 3244K | |
| 16:15:18 | sean-k-mooney | [ 18.397470] Write protecting the kernel read-only data: 30720k | |
| 16:15:20 | sean-k-mooney | [ 18.419370] Freeing unused kernel image (text/rodata gap) memory: 2036K | |
| 16:15:22 | sean-k-mooney | [ 18.433426] Freeing unused kernel image (rodata/data gap) memory: 1448K | |
| 16:15:30 | sean-k-mooney | it has already freed the ram disk and early allcoations | |
| 16:19:32 | sean-k-mooney | cirros-0.6.2-x86_64-disk.img so that pretty new | |
| 16:19:56 | sean-k-mooney | or at least its using a ubuntu 22.04 kernel | |
| 16:20:27 | sean-k-mooney | it look like eitehr a real out of memory issue or a kernel bug to me | |
| 16:21:12 | sean-k-mooney | 0.6.2 was released in may so we coudl perhaps ask for a 0 | |
| 16:21:28 | sean-k-mooney | 0.6.3 release with the latest hardwer enabeling kernel package? | |
| 17:00:58 | melwitt | it seems like all of the swift put_object failures I see occur in the nova-lvm job on tests that have nothing to do with volumes. I don't get it Jun 10 04:45:27.576181 np0037692500 devstack@g-api.service[86519]: ERROR glance.common.wsgi Got error from Swift: put_object('glance', 'b1cfd7cf-ff23-4b05-be12-48ce3e14d593-00001', ...) failure and no ability to reset contents for reupload.. | |
| 17:01:39 | sean-k-mooney | hum... we dont need swift in that job | |
| 17:01:50 | sean-k-mooney | the nova-lvm jobs is ment to test nova with the lvm storage backend | |
| 17:01:55 | sean-k-mooney | and nova never talks to swift | |
| 17:01:59 | melwitt | that's where glance is uploading images though | |
| 17:02:11 | sean-k-mooney | right but it could and proably should be using the file backend | |
| 17:02:19 | sean-k-mooney | not the swift backend | |
| 17:02:21 | melwitt | oh, yeah | |
| 17:04:48 | dansmith | why "should" ? | |
| 17:07:21 | sean-k-mooney | as in why it should not be using swift? | |
| 17:07:27 | sean-k-mooney | just to keep the job smaller | |
| 17:07:27 | dansmith | yeah | |
| 17:07:42 | sean-k-mooney | its not really imporant to testing the lvm storag backend | |
| 17:07:47 | dansmith | the default config is to use swift though right? | |
| 17:07:51 | sean-k-mooney | no file | |
| 17:08:02 | sean-k-mooney | although maybe that depend on if swift is deployed | |
| 17:08:21 | dansmith | I think if swift is deployed (which it is in the integrated job, AFAIK) then we'll use it | |
| 17:08:47 | sean-k-mooney | swift is deployed by default in devstack although i always turn it off so im used to glance using file | |
| 17:08:51 | dansmith | I don't see that the LVM job is configuring it specifically.. and point being just that the fewer deltas from the base job the easier it is to reason about failures and why they're different | |
| 17:09:04 | sean-k-mooney | or cinder | |
| 17:09:28 | sean-k-mooney | well i would not expect the base job to use swift for glance | |
| 17:09:39 | dansmith | I would unless we don't deploy swift | |
| 17:10:15 | dansmith | I don't see an "enable_service" for the nova-lvm job that is opting into swift, is my point | |
| 17:12:12 | dansmith | okay but maybe integrated-compute does disable it? | |
| 17:12:56 | sean-k-mooney | i dont know i might have actully bee looking at the multi store job not lvm | |
| 17:13:17 | sean-k-mooney | the multi store job does turn it off http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/controller/logs/_.localrc_auto.txt | |
| 17:14:37 | sean-k-mooney | maybe i misread the job name but i dont have a nova-lvm tab open so im kind of confuged why i tought it was nova-lvm | |
| 17:14:46 | dansmith | okay, tbh I thought more of our jobs were running with swift-backed glance since it's the default, but perhaps "most" is not quite right | |
| 17:15:21 | dansmith | https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L151 | |
| 17:15:22 | sean-k-mooney | so i would have personaly tough glance is rarely backed by swift and file or cinder backend woudl be more common | |
| 17:15:38 | dansmith | not me :) | |
| 17:16:05 | dansmith | backing it by file requires shared storage and I think doing so by cinder is rather expensive, IIRC | |
| 17:16:10 | sean-k-mooney | i guess im just used to swift not being deployed in the enviornment i look at | |
| 17:17:44 | sean-k-mooney | actully we were talking about timeout an dthe integreated compute job and its deritives | |
| 17:17:47 | sean-k-mooney | https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L141 | |
| 17:17:53 | sean-k-mooney | looks like its already reduced to 4 | |
| 17:18:42 | sean-k-mooney | i guess we clampped it to 4 because of failure when we rased the default for the rest to 6 | |
| 17:22:47 | sean-k-mooney | dansmith: looking at a clople of the tiemout they are all on rax but spread across 3 datacenters rax-iad, rax-dfw and rax-ord | |
| 17:30:31 | sean-k-mooney | dansmith: https://tinyurl.com/3bf6nhkh | |
| 17:31:31 | sean-k-mooney | actull i think i need to grop by somethign | |
| 17:31:37 | sean-k-mooney | so that i only get on ehit per job | |
| 17:32:14 | dansmith | yeah, it wouldn't surprise me at all if rax is the oldest/slowest of the providers | |
| 17:32:25 | dansmith | maybe we could group only smaller jobs to run on rax | |
| 17:32:41 | sean-k-mooney | they also run our vms on xen not kvm i belive | |
| 17:38:08 | sean-k-mooney | ok this is better https://tinyurl.com/35dfcfcy | |
| 17:38:37 | sean-k-mooney | only 72 hits mostly rax | |
| 17:39:10 | sean-k-mooney | what would be useful is if we could break down the average job runtime by provider | |
| #openstack-nova - 2024-06-11 | |||
| 06:05:18 | hamidlotfi_ | An instance whose volume is on CEPH, after abnormally restarting the Compute server where that instance is located, that instance does not come up and displays the following error: | |
| 06:05:18 | hamidlotfi_ | Hi there, | |
| 06:05:22 | hamidlotfi_ | https://www.irccloud.com/pastebin/n5z4qgB7/ | |
| 06:05:27 | hamidlotfi_ | I checked all of those values for the disk_cachemodes parameter, | |
| 06:05:44 | hamidlotfi_ | but in all of the values, the result is the same and I had that error. | |
| 06:05:44 | hamidlotfi_ | network=writethrough | |
| 06:05:44 | hamidlotfi_ | network=none | |
| 06:05:44 | hamidlotfi_ | network=directsync | |
| 08:09:51 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | |
| 11:16:56 | sean-k-mooney | gibi: can you review https://review.opendev.org/c/openstack/nova/+/710848 and the proceedign repoducer again | |
| 11:17:38 | sean-k-mooney | when you have time of course but this has been arrond for a long time and if you agree i would like to back port that to stable branches | |
| 12:09:25 | gibi | added to my queue | |
| 13:15:02 | WJeffs | Hey, I might be being blind, but is there any options for Numa placement when allocating them to a VM. Say we have a HV with 8 numa groups across 2 cpus, and I want to allocate all the cores from 2 groups, thats no issue, but it gives me 1+6, but is there a way we can make it always use numa on same physical cpu at all times? | |
| 13:59:06 | sean-k-mooney | WJeffs: form the same cpu package no. if you have a numa affined guest and ask for 2 numa nodes we will either pack or balance the vm across the numa node depenign on a config option | |
| 13:59:39 | sean-k-mooney | WJeffs: in the past the beahvior was to pack leading to poor performance we now default to spreading | |
| 14:00:12 | sean-k-mooney | WJeffs: we did have a request ot supprot socket level optimiations for amd cpus in the past | |
| 14:00:21 | sean-k-mooney | but we have not had time to implemtn that | |
| 15:00:51 | WJeffs | sean-k-mooney: ah it is for AMD Genoa, I guess that is the same part, we are already creating the numa configuration inside the VM and that works great and gives us good performance, now we are trying to test some extra configurations and found that the numas are spread across sockets | |
| 15:01:20 | WJeffs | I'll have a look for the request/blueprint and see if there is anything we can do to help on this. | |
| 15:05:22 | sean-k-mooney | WJeffs: this is basicaly all contolled by https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2449-L2511 currently | |
| 15:05:47 | sean-k-mooney | to do what you want we would need to add the socket info int the numa toplogy | |
| 15:06:10 | sean-k-mooney | and then add an addtional sort here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2481 | |
| 15:06:18 | sean-k-mooney | to sort the nodes by socket | |
| 15:07:37 | sean-k-mooney | sorry not there just below it https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2488 | |
| 15:07:43 | sean-k-mooney | the les is for pinned cpus | |
| 15:38:27 | WJeffs | sean-k-mooney: Thanks, I'll take a look and see if there is something can do to help | |