| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-10 | |||
| 17:12:56 | sean-k-mooney | i dont know i might have actully bee looking at the multi store job not lvm | |
| 17:13:17 | sean-k-mooney | the multi store job does turn it off http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/controller/logs/_.localrc_auto.txt | |
| 17:14:37 | sean-k-mooney | maybe i misread the job name but i dont have a nova-lvm tab open so im kind of confuged why i tought it was nova-lvm | |
| 17:14:46 | dansmith | okay, tbh I thought more of our jobs were running with swift-backed glance since it's the default, but perhaps "most" is not quite right | |
| 17:15:21 | dansmith | https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L151 | |
| 17:15:22 | sean-k-mooney | so i would have personaly tough glance is rarely backed by swift and file or cinder backend woudl be more common | |
| 17:15:38 | dansmith | not me :) | |
| 17:16:05 | dansmith | backing it by file requires shared storage and I think doing so by cinder is rather expensive, IIRC | |
| 17:16:10 | sean-k-mooney | i guess im just used to swift not being deployed in the enviornment i look at | |
| 17:17:44 | sean-k-mooney | actully we were talking about timeout an dthe integreated compute job and its deritives | |
| 17:17:47 | sean-k-mooney | https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L141 | |
| 17:17:53 | sean-k-mooney | looks like its already reduced to 4 | |
| 17:18:42 | sean-k-mooney | i guess we clampped it to 4 because of failure when we rased the default for the rest to 6 | |
| 17:22:47 | sean-k-mooney | dansmith: looking at a clople of the tiemout they are all on rax but spread across 3 datacenters rax-iad, rax-dfw and rax-ord | |
| 17:30:31 | sean-k-mooney | dansmith: https://tinyurl.com/3bf6nhkh | |
| 17:31:31 | sean-k-mooney | actull i think i need to grop by somethign | |
| 17:31:37 | sean-k-mooney | so that i only get on ehit per job | |
| 17:32:14 | dansmith | yeah, it wouldn't surprise me at all if rax is the oldest/slowest of the providers | |
| 17:32:25 | dansmith | maybe we could group only smaller jobs to run on rax | |
| 17:32:41 | sean-k-mooney | they also run our vms on xen not kvm i belive | |
| 17:38:08 | sean-k-mooney | ok this is better https://tinyurl.com/35dfcfcy | |
| 17:38:37 | sean-k-mooney | only 72 hits mostly rax | |
| 17:39:10 | sean-k-mooney | what would be useful is if we could break down the average job runtime by provider | |
| #openstack-nova - 2024-06-11 | |||
| 06:05:18 | hamidlotfi_ | Hi there, | |
| 06:05:18 | hamidlotfi_ | An instance whose volume is on CEPH, after abnormally restarting the Compute server where that instance is located, that instance does not come up and displays the following error: | |
| 06:05:22 | hamidlotfi_ | https://www.irccloud.com/pastebin/n5z4qgB7/ | |
| 06:05:27 | hamidlotfi_ | I checked all of those values for the disk_cachemodes parameter, | |
| 06:05:44 | hamidlotfi_ | network=directsync | |
| 06:05:44 | hamidlotfi_ | network=none | |
| 06:05:44 | hamidlotfi_ | network=writethrough | |
| 06:05:44 | hamidlotfi_ | but in all of the values, the result is the same and I had that error. | |
| 08:09:51 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | |
| 11:16:56 | sean-k-mooney | gibi: can you review https://review.opendev.org/c/openstack/nova/+/710848 and the proceedign repoducer again | |
| 11:17:38 | sean-k-mooney | when you have time of course but this has been arrond for a long time and if you agree i would like to back port that to stable branches | |
| 12:09:25 | gibi | added to my queue | |
| 13:15:02 | WJeffs | Hey, I might be being blind, but is there any options for Numa placement when allocating them to a VM. Say we have a HV with 8 numa groups across 2 cpus, and I want to allocate all the cores from 2 groups, thats no issue, but it gives me 1+6, but is there a way we can make it always use numa on same physical cpu at all times? | |
| 13:59:06 | sean-k-mooney | WJeffs: form the same cpu package no. if you have a numa affined guest and ask for 2 numa nodes we will either pack or balance the vm across the numa node depenign on a config option | |
| 13:59:39 | sean-k-mooney | WJeffs: in the past the beahvior was to pack leading to poor performance we now default to spreading | |
| 14:00:12 | sean-k-mooney | WJeffs: we did have a request ot supprot socket level optimiations for amd cpus in the past | |
| 14:00:21 | sean-k-mooney | but we have not had time to implemtn that | |
| 15:00:51 | WJeffs | sean-k-mooney: ah it is for AMD Genoa, I guess that is the same part, we are already creating the numa configuration inside the VM and that works great and gives us good performance, now we are trying to test some extra configurations and found that the numas are spread across sockets | |
| 15:01:20 | WJeffs | I'll have a look for the request/blueprint and see if there is anything we can do to help on this. | |
| 15:05:22 | sean-k-mooney | WJeffs: this is basicaly all contolled by https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2449-L2511 currently | |
| 15:05:47 | sean-k-mooney | to do what you want we would need to add the socket info int the numa toplogy | |
| 15:06:10 | sean-k-mooney | and then add an addtional sort here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2481 | |
| 15:06:18 | sean-k-mooney | to sort the nodes by socket | |
| 15:07:37 | sean-k-mooney | sorry not there just below it https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2488 | |
| 15:07:43 | sean-k-mooney | the les is for pinned cpus | |
| 15:38:27 | WJeffs | sean-k-mooney: Thanks, I'll take a look and see if there is something can do to help | |
| 15:40:00 | sean-k-mooney | the main gap right now is we dont track the socket id in the host numa topogy data stucture | |
| 15:40:07 | sean-k-mooney | or the numa distance btween numa ndoes | |
| 15:40:13 | sean-k-mooney | since that really waht you want to minimise | |
| 15:40:34 | sean-k-mooney | libvirt provide both we jsut dont collect it right now | |
| 15:41:00 | WJeffs | yea, thats the main thing we are trying to do. So its collect from libvirt -> use that data that needs to be added. | |
| 15:41:29 | sean-k-mooney | ya so artom looked at this brifly a year or two ago | |
| 15:42:00 | sean-k-mooney | we have a downstream tracker in our backlog to eventually look at this https://issues.redhat.com/browse/OSPRH-36 | |
| 15:42:15 | sean-k-mooney | but we jsut didnt have time or custoemr ask to priories it | |
| 15:42:21 | artom | I even have/had WIP patches up, at some point | |
| 15:42:35 | artom | Just the basics to collect those distance from libvirt | |
| 15:42:35 | sean-k-mooney | https://review.opendev.org/q/topic:%22bp/libvirt-smarter-cpu-placement%22 | |
| 15:42:39 | sean-k-mooney | yep ^ | |
| 15:43:06 | sean-k-mooney | there in merge conflict but its proably pretty trivial | |
| 15:43:28 | sean-k-mooney | this was the bluepritn https://blueprints.launchpad.net/nova/+spec/libvirt-smarter-cpu-placement | |
| 15:43:34 | sean-k-mooney | btu not much detail there | |
| 15:44:53 | WJeffs | ah wow, it was pretty close then, just come work to tidy up. | |
| 15:47:06 | sean-k-mooney | tests, ensuring this works for live mirgation that sort of thing | |
| 15:47:12 | WJeffs | appreciate all the hard work guys | |
| 15:47:36 | sean-k-mooney | i dont recal exactly how far artom got but its a good starting point | |
| 15:47:48 | artom | Those two patches are literally all I had. | |
| 15:48:13 | artom | The hard part is integrating that into the placement of VMs | |
| 15:55:30 | bauzas | nova meeting in 5 mins here | |
| 15:55:41 | bauzas | I could be a bit lagging, don't be afraid | |
| 16:00:22 | bauzas | #startmeeting nova | |
| 16:00:22 | opendevmeet | Meeting started Tue Jun 11 16:00:22 2024 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:22 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:22 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:35 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:00:35 | gibi | o/ | |
| 16:00:56 | elodilles | o/ | |
| 16:01:00 | fwiesel | \o | |
| 16:01:11 | dansmith | o/ | |
| 16:01:27 | auniyal | o/ | |
| 16:01:34 | bauzas | my current time : | |
| 16:01:37 | bauzas | 64 bytes from par21s19-in-f4.1e100.net (142.250.179.68): icmp_seq=27972 ttl=112 time=193 ms | |
| 16:01:49 | bauzas | anyway, let's start | |
| 16:02:05 | bauzas | #topic Bugs (stuck/critical) | |
| 16:02:10 | bauzas | #info No Critical bug | |
| 16:02:15 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:02:22 | bauzas | anything to discuss ? | |
| 16:03:09 | bauzas | looks not | |
| 16:03:19 | bauzas | #topic Gate status | |
| 16:03:25 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:03:30 | bauzas | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:03:36 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:03:56 | bauzas | we had a nova-emulation master job failure https://zuul.openstack.org/build/2f26196745fe4cc8a93b5453317710ee | |
| 16:04:09 | sean-k-mooney | the fix for that job is still not merged | |
| 16:04:23 | sean-k-mooney | it keeps hitting job timeout or unrelated failures in other jobs | |
| 16:04:36 | bauzas | because of when deleting some instance, it was not found | |
| 16:04:53 | sean-k-mooney | i have started to look in to how to gather some data to deteimn if we should bump the timeouts in the integrated-compute jobs and its decendent jobs | |
| 16:05:00 | bauzas | ok | |