| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-05 | |||
| 18:14:09 | sean-k-mooney | that why this didnt progress. its obvious that there is some interest in doing his | |
| 18:14:18 | sean-k-mooney | so i think it could happen next cycle | |
| 18:25:57 | cardoe | okay | |
| 21:56:54 | opendevreview | Callum Dickinson proposed openstack/nova master: Fix image ID in libvirt metadata when unshelving https://review.opendev.org/c/openstack/nova/+/942973 | |
| 21:56:55 | opendevreview | Callum Dickinson proposed openstack/nova master: Add more flavor metadata to libvirt guest XML https://review.opendev.org/c/openstack/nova/+/942974 | |
| 21:56:55 | opendevreview | Callum Dickinson proposed openstack/nova master: Add image meta to libvirt XML metadata https://review.opendev.org/c/openstack/nova/+/942766 | |
| 22:00:32 | Callum027 | sean-k-mooney: Morning, thanks for getting in touch, I've rebased and adopted the suggestions you made to my patches | |
| 22:03:42 | sean-k-mooney | Callum027: thanks, im just wrappiping for today, i proably shoudl have stop about 4 hours ago :) but ill take alook again tomorrow once we get some ci results | |
| 22:03:58 | Callum027 | No worries, have a good evening | |
| 22:04:02 | sean-k-mooney | o/ | |
| 23:06:46 | melwitt | working on code near the server group late affinity check I noticed it looks like there has been a regression in the error handling of the check due to a past bug fix that changed what exception the late check raises https://bugs.launchpad.net/nova/+bug/2119578 | |
| 23:07:10 | melwitt | I just filed a bug to capture it | |
| #openstack-nova - 2025-08-06 | |||
| 06:36:43 | auniyal | hi sean-k-mooney, will you have access to merge this in nova-disgit | |
| 06:36:43 | auniyal | https://review.rdoproject.org/r/c/openstack/nova-distgit/+/58076 | |
| 06:37:46 | auniyal | can you please provide review comments | |
| 06:38:29 | auniyal | we have only 1 nova-disgit core - sahid.ferdjaoui@redhat.com | |
| 06:38:29 | auniyal | https://review.rdoproject.org/r/admin/groups/1abd61acb05653e6333e01582fe4b118a7febc8c,members | |
| 06:38:35 | auniyal | sahid ? | |
| 10:33:53 | sean-k-mooney | auniyal: i did until october whne i asked to be removed form the list rdo cores for nova. artom and gibi have core rights | |
| 10:34:54 | sean-k-mooney | currently the only core right i still have in rdo are for cyborg | |
| 10:40:27 | sean-k-mooney | melwitt: commeted on the bug | |
| 10:41:01 | sean-k-mooney | https://review.rdoproject.org/r/admin/groups/1abd61acb05653e6333e01582fe4b118a7febc8c,members | |
| 10:41:20 | sean-k-mooney | that is not the group we use | |
| 10:41:59 | sean-k-mooney | we use https://review.rdoproject.org/r/admin/groups/d90e2cf3a41ddec7e571c67ce63e10aed1b4e363,members but it looks like artom has not updated that yet to add gibi | |
| 10:42:47 | sean-k-mooney | anyway this is more a downstream or rdo converstaion since its not really in the scope fo the nova project | |
| 12:06:01 | auniyal | sean-k-mooney ack, | |
| 12:06:20 | sean-k-mooney | auniyal: there may be a review open ot update the list | |
| 12:06:32 | auniyal | still can you please have a look patch is okay | |
| 12:08:15 | auniyal | I added sahid, hopefully they will look | |
| 12:08:40 | sean-k-mooney | sahid does not work on this anymore | |
| 12:08:48 | sean-k-mooney | they may but its unlikely | |
| 12:09:00 | sean-k-mooney | so the patch shoudl be propsoed againt master first | |
| 12:09:06 | sean-k-mooney | not direclty to andeploe liek this | |
| 12:09:36 | sean-k-mooney | and you will need to backprot it in rdo | |
| 12:09:51 | auniyal | yeah I did that first https://review.rdoproject.org/r/c/openstack/nova-distgit/+/58075 | |
| 12:10:05 | sean-k-mooney | right but you abandoned it | |
| 12:10:13 | auniyal | bust as we need this in 18 (downstream) I thought antelope is correct one | |
| 12:10:26 | sean-k-mooney | so it was not the wrong branch it ned to land there first and then backprot it to antelope | |
| 12:10:52 | auniyal | also I saw this - https://review.rdoproject.org/r/c/openstack/nova-distgit/+/57752 | |
| 12:11:20 | sean-k-mooney | that is diffent | |
| 12:11:50 | auniyal | oh okay, sure will create the backports then | |
| 12:12:03 | sean-k-mooney | althogh it likely shoudl have been done to master first as well | |
| 12:12:16 | sean-k-mooney | the reason its diffent is the depencies are diffent per branch | |
| 12:16:20 | mohsen__ | hello dear friends. Is there anybody here who has ever tested the vCPU vs pCPU performance? | |
| 12:16:20 | mohsen__ | I tried NUMATopologyFilter on nova scheduler along with cpu_dedicated_set config on the nova compute side. The scheduling process works completely fine. | |
| 12:16:20 | mohsen__ | this is my nova compute config: | |
| 12:16:20 | mohsen__ | [compute] | |
| 12:16:21 | mohsen__ | cpu_shared_set = 18-35,54-71 | |
| 12:16:21 | mohsen__ | cpu_dedicated_set = 0-17,36-53 | |
| 12:16:23 | mohsen__ | consider the fact that the cpu_shared_set value is on numa node 0 and cpu_dedicated_set is on numa node 1. | |
| 12:16:25 | mohsen__ | The issue is that after creating instances and benchmarking vCPU and pCPU instances separately, the result was surprising. The performance of vCPU was better than pCPU. | |
| 12:16:27 | mohsen__ | Is there anybody who has experienced the same issue? | |
| 12:17:39 | sean-k-mooney | that likely because of other effect related to the numa toplogy | |
| 12:17:53 | auniyal | sean-k-mooney, do you think other things are alight in patch - like file path and bin prefix like nova-contrib - do you have any comment on that | |
| 12:18:40 | sean-k-mooney | i skimed it and ot look mostly ok but im not sure if you need to set the mode here https://review.rdoproject.org/r/c/openstack/nova-distgit/+/58076/8/openstack-nova.spec#580 | |
| 12:19:31 | sean-k-mooney | it proably should be 755 or 644 but im not sure if that is required | |
| 12:19:56 | mohsen__ | sean-k-mooney: What do you mean by other effect? did you mean the other instances on the same compute node can affect the result of the test? | |
| 12:20:14 | sean-k-mooney | in general this is not somethign i review often so i alwasy have to look up the manpages for install ectra when i do | |
| 12:20:57 | sean-k-mooney | mohsen__: no i mean if the testing your doing is using cinder volume or relys on networking | |
| 12:21:09 | sean-k-mooney | then using numa node 1 may have higher latency then numa 0 | |
| 12:22:17 | auniyal | I know it should be executable - I use 755 because that rest of the bins/script has set | |
| 12:22:30 | sean-k-mooney | im talking about the directory | |
| 12:22:44 | auniyal | install -d --> that mean mkdir -p | |
| 12:22:44 | sean-k-mooney | not the files | |
| 12:23:05 | sean-k-mooney | right which uses the default umask for the current process for the permsions | |
| 12:23:12 | sean-k-mooney | im not sur eyou shoud be relying on that | |
| 12:23:25 | sean-k-mooney | you likel yshoudl follwo the other exampels where teh mode is alwasy explcitly set | |
| 12:24:21 | sean-k-mooney | auniyal: all other usage of `install` in that file has an explcity -m to set the mode | |
| 12:24:28 | sean-k-mooney | so you are deviating form the conventiosn in the file | |
| 12:24:57 | sean-k-mooney | you proably shoudl be doing install -d -m 755 | |
| 12:25:06 | sean-k-mooney | here https://review.rdoproject.org/r/c/openstack/nova-distgit/+/58076/8/openstack-nova.spec#580 | |
| 12:26:46 | auniyal | yeah above all setting permission to dir, | |
| 12:26:46 | auniyal | I only thought of creating it, then setting chmod to file | |
| 12:30:34 | auniyal | thanks Sean | |
| 12:31:33 | mohsen__ | sean-k-mooney: So how is it possible to have pCPUs with better performance than vCPUs, along with using networking and cinder-volume to create instances? | |
| 12:31:33 | mohsen__ | Note that once I changed the cpu_dedicated_set from one numa node to another, and the benchmarking result stayed the same. | |
| 12:31:52 | sean-k-mooney | mohsen__: by the way on an otherwise idel system you are unlikely to see many of the beniftis of pinning. pinning helps performance for 2 reasons., 1 it enfoces a explict numa toplogy so the guest memroy and cpus are allcoated form teh same numa ndoe, (that why you shoudl alwasy set hw:mem_page_size=<some valid value> i.e. small that give about 70% of the performace boost, | |
| 12:31:54 | sean-k-mooney | the rest comes form reducign cache misses and prevent other vms form consdume cpu time. | |
| 12:33:13 | sean-k-mooney | mohsen__: tha tis very stragne because in the perfomnance testing we did in the past it did impove performance. the main reasoin for CPU pinning is not raw trhoughput however, it benifit is in determinisum | |
| 12:33:37 | sean-k-mooney | it shoudl not reduce the performce vs vcpus becasue the emulation of the vcpu is identical | |
| 12:34:01 | sean-k-mooney | all cpu pinnign does is use cgroups/taskset to prevent the kernel form movinv the cpus | |
| 12:34:23 | sean-k-mooney | and on the nova side we prevent other vms form beign schduled to the same cores | |
| 12:34:38 | sean-k-mooney | its why we call it cpu_policy=dedicated | |
| 12:35:01 | sean-k-mooney | you care giveing an entire host cpu to the vm provide you are isolating other host process form those cores | |
| 12:35:45 | sean-k-mooney | regarding newtwokrign and cidner , if your using sriov you can allcoat vfs form the local numa node or you can use a cross numa bond for the stroage network | |
| 12:36:33 | sean-k-mooney | mohsen__: when you swap the numa nodes did you delete and recreate the vms by the way | |
| 12:48:43 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for using cell-name in cell_v2 commands https://review.opendev.org/c/openstack/nova/+/954460 | |
| 12:50:33 | mohsen__ | sean-k-mooney: Yes. I removed the old instances and after reconfiguration, created new ones. So you mean cpu performance improvement isn't necessarily the benefit of cpu pinning? | |
| 12:50:33 | mohsen__ | having such an expectation isn't always valid? | |
| 13:25:02 | sean-k-mooney | cpu pinning was create to provide predicable perfroamce under load | |
| 13:25:39 | sean-k-mooney | it often provides mroe performce as a side effect of enfoceign numa affintiy between guest cpus ram and pci devices | |
| 13:29:50 | sean-k-mooney | but most of the uplift comes enableing numa affintiy as cpu pinning implictly enables hw:numa_nodes=1 | |
| 13:30:23 | sean-k-mooney | adding hw:numa_nodes=1 hw:mem_page_size=small | |
| 13:30:46 | sean-k-mooney | you will get most of the performace uplift that is expect form cpu pinnin for floating guests | |
| 14:09:46 | opendevreview | Ian Watson proposed openstack/nova master: Add migration conditional for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 14:24:25 | sean-k-mooney | stephenfin: also interesting error in the logs https://paste.opendev.org/show/bxMqpRogUlo3jHCe2trb/ that likely related to your volume bug | |
| 14:24:46 | sean-k-mooney | we shoudl be able to catch that and abort starting the agent and let systemd or whatever restart it | |
| 14:24:53 | stephenfin | ooohh, good spot | |
| 14:25:01 | sean-k-mooney | https://0908e5c3034094a51a1a-ddb50bb0d8d4f5453b9288f160ef0a30.ssl.cf5.rackcdn.com/openstack/70be4003a0ae40b4bfaae667688f694c/controller/logs/screen-n-cpu.txt | |
| 14:25:14 | sean-k-mooney | its right in the startup just after the config is printed | |