| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-06 | |||
| 10:41:20 | sean-k-mooney | that is not the group we use | |
| 10:41:59 | sean-k-mooney | we use https://review.rdoproject.org/r/admin/groups/d90e2cf3a41ddec7e571c67ce63e10aed1b4e363,members but it looks like artom has not updated that yet to add gibi | |
| 10:42:47 | sean-k-mooney | anyway this is more a downstream or rdo converstaion since its not really in the scope fo the nova project | |
| 12:06:01 | auniyal | sean-k-mooney ack, | |
| 12:06:20 | sean-k-mooney | auniyal: there may be a review open ot update the list | |
| 12:06:32 | auniyal | still can you please have a look patch is okay | |
| 12:08:15 | auniyal | I added sahid, hopefully they will look | |
| 12:08:40 | sean-k-mooney | sahid does not work on this anymore | |
| 12:08:48 | sean-k-mooney | they may but its unlikely | |
| 12:09:00 | sean-k-mooney | so the patch shoudl be propsoed againt master first | |
| 12:09:06 | sean-k-mooney | not direclty to andeploe liek this | |
| 12:09:36 | sean-k-mooney | and you will need to backprot it in rdo | |
| 12:09:51 | auniyal | yeah I did that first https://review.rdoproject.org/r/c/openstack/nova-distgit/+/58075 | |
| 12:10:05 | sean-k-mooney | right but you abandoned it | |
| 12:10:13 | auniyal | bust as we need this in 18 (downstream) I thought antelope is correct one | |
| 12:10:26 | sean-k-mooney | so it was not the wrong branch it ned to land there first and then backprot it to antelope | |
| 12:10:52 | auniyal | also I saw this - https://review.rdoproject.org/r/c/openstack/nova-distgit/+/57752 | |
| 12:11:20 | sean-k-mooney | that is diffent | |
| 12:11:50 | auniyal | oh okay, sure will create the backports then | |
| 12:12:03 | sean-k-mooney | althogh it likely shoudl have been done to master first as well | |
| 12:12:16 | sean-k-mooney | the reason its diffent is the depencies are diffent per branch | |
| 12:16:20 | mohsen__ | hello dear friends. Is there anybody here who has ever tested the vCPU vs pCPU performance? | |
| 12:16:20 | mohsen__ | I tried NUMATopologyFilter on nova scheduler along with cpu_dedicated_set config on the nova compute side. The scheduling process works completely fine. | |
| 12:16:20 | mohsen__ | this is my nova compute config: | |
| 12:16:20 | mohsen__ | [compute] | |
| 12:16:21 | mohsen__ | cpu_shared_set = 18-35,54-71 | |
| 12:16:21 | mohsen__ | cpu_dedicated_set = 0-17,36-53 | |
| 12:16:23 | mohsen__ | consider the fact that the cpu_shared_set value is on numa node 0 and cpu_dedicated_set is on numa node 1. | |
| 12:16:25 | mohsen__ | The issue is that after creating instances and benchmarking vCPU and pCPU instances separately, the result was surprising. The performance of vCPU was better than pCPU. | |
| 12:16:27 | mohsen__ | Is there anybody who has experienced the same issue? | |
| 12:17:39 | sean-k-mooney | that likely because of other effect related to the numa toplogy | |
| 12:17:53 | auniyal | sean-k-mooney, do you think other things are alight in patch - like file path and bin prefix like nova-contrib - do you have any comment on that | |
| 12:18:40 | sean-k-mooney | i skimed it and ot look mostly ok but im not sure if you need to set the mode here https://review.rdoproject.org/r/c/openstack/nova-distgit/+/58076/8/openstack-nova.spec#580 | |
| 12:19:31 | sean-k-mooney | it proably should be 755 or 644 but im not sure if that is required | |
| 12:19:56 | mohsen__ | sean-k-mooney: What do you mean by other effect? did you mean the other instances on the same compute node can affect the result of the test? | |
| 12:20:14 | sean-k-mooney | in general this is not somethign i review often so i alwasy have to look up the manpages for install ectra when i do | |
| 12:20:57 | sean-k-mooney | mohsen__: no i mean if the testing your doing is using cinder volume or relys on networking | |
| 12:21:09 | sean-k-mooney | then using numa node 1 may have higher latency then numa 0 | |
| 12:22:17 | auniyal | I know it should be executable - I use 755 because that rest of the bins/script has set | |
| 12:22:30 | sean-k-mooney | im talking about the directory | |
| 12:22:44 | auniyal | install -d --> that mean mkdir -p | |
| 12:22:44 | sean-k-mooney | not the files | |
| 12:23:05 | sean-k-mooney | right which uses the default umask for the current process for the permsions | |
| 12:23:12 | sean-k-mooney | im not sur eyou shoud be relying on that | |
| 12:23:25 | sean-k-mooney | you likel yshoudl follwo the other exampels where teh mode is alwasy explcitly set | |
| 12:24:21 | sean-k-mooney | auniyal: all other usage of `install` in that file has an explcity -m to set the mode | |
| 12:24:28 | sean-k-mooney | so you are deviating form the conventiosn in the file | |
| 12:24:57 | sean-k-mooney | you proably shoudl be doing install -d -m 755 | |
| 12:25:06 | sean-k-mooney | here https://review.rdoproject.org/r/c/openstack/nova-distgit/+/58076/8/openstack-nova.spec#580 | |
| 12:26:46 | auniyal | yeah above all setting permission to dir, | |
| 12:26:46 | auniyal | I only thought of creating it, then setting chmod to file | |
| 12:30:34 | auniyal | thanks Sean | |
| 12:31:33 | mohsen__ | sean-k-mooney: So how is it possible to have pCPUs with better performance than vCPUs, along with using networking and cinder-volume to create instances? | |
| 12:31:33 | mohsen__ | Note that once I changed the cpu_dedicated_set from one numa node to another, and the benchmarking result stayed the same. | |
| 12:31:52 | sean-k-mooney | mohsen__: by the way on an otherwise idel system you are unlikely to see many of the beniftis of pinning. pinning helps performance for 2 reasons., 1 it enfoces a explict numa toplogy so the guest memroy and cpus are allcoated form teh same numa ndoe, (that why you shoudl alwasy set hw:mem_page_size=<some valid value> i.e. small that give about 70% of the performace boost, | |
| 12:31:54 | sean-k-mooney | the rest comes form reducign cache misses and prevent other vms form consdume cpu time. | |
| 12:33:13 | sean-k-mooney | mohsen__: tha tis very stragne because in the perfomnance testing we did in the past it did impove performance. the main reasoin for CPU pinning is not raw trhoughput however, it benifit is in determinisum | |
| 12:33:37 | sean-k-mooney | it shoudl not reduce the performce vs vcpus becasue the emulation of the vcpu is identical | |
| 12:34:01 | sean-k-mooney | all cpu pinnign does is use cgroups/taskset to prevent the kernel form movinv the cpus | |
| 12:34:23 | sean-k-mooney | and on the nova side we prevent other vms form beign schduled to the same cores | |
| 12:34:38 | sean-k-mooney | its why we call it cpu_policy=dedicated | |
| 12:35:01 | sean-k-mooney | you care giveing an entire host cpu to the vm provide you are isolating other host process form those cores | |
| 12:35:45 | sean-k-mooney | regarding newtwokrign and cidner , if your using sriov you can allcoat vfs form the local numa node or you can use a cross numa bond for the stroage network | |
| 12:36:33 | sean-k-mooney | mohsen__: when you swap the numa nodes did you delete and recreate the vms by the way | |
| 12:48:43 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for using cell-name in cell_v2 commands https://review.opendev.org/c/openstack/nova/+/954460 | |
| 12:50:33 | mohsen__ | sean-k-mooney: Yes. I removed the old instances and after reconfiguration, created new ones. So you mean cpu performance improvement isn't necessarily the benefit of cpu pinning? | |
| 12:50:33 | mohsen__ | having such an expectation isn't always valid? | |
| 13:25:02 | sean-k-mooney | cpu pinning was create to provide predicable perfroamce under load | |
| 13:25:39 | sean-k-mooney | it often provides mroe performce as a side effect of enfoceign numa affintiy between guest cpus ram and pci devices | |
| 13:29:50 | sean-k-mooney | but most of the uplift comes enableing numa affintiy as cpu pinning implictly enables hw:numa_nodes=1 | |
| 13:30:23 | sean-k-mooney | adding hw:numa_nodes=1 hw:mem_page_size=small | |
| 13:30:46 | sean-k-mooney | you will get most of the performace uplift that is expect form cpu pinnin for floating guests | |
| 14:09:46 | opendevreview | Ian Watson proposed openstack/nova master: Add migration conditional for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 14:24:25 | sean-k-mooney | stephenfin: also interesting error in the logs https://paste.opendev.org/show/bxMqpRogUlo3jHCe2trb/ that likely related to your volume bug | |
| 14:24:46 | sean-k-mooney | we shoudl be able to catch that and abort starting the agent and let systemd or whatever restart it | |
| 14:24:53 | stephenfin | ooohh, good spot | |
| 14:25:01 | sean-k-mooney | https://0908e5c3034094a51a1a-ddb50bb0d8d4f5453b9288f160ef0a30.ssl.cf5.rackcdn.com/openstack/70be4003a0ae40b4bfaae667688f694c/controller/logs/screen-n-cpu.txt | |
| 14:25:14 | sean-k-mooney | its right in the startup just after the config is printed | |
| 14:25:18 | sean-k-mooney | so it pretyy early | |
| 14:25:31 | sean-k-mooney | but my guess is it never recovers form that if it happens | |
| 14:25:39 | sean-k-mooney | so either we need to abort or retry | |
| 14:26:10 | sean-k-mooney | well or something more indelgent then the 30s tought i have given it | |
| 14:26:32 | sean-k-mooney | can you see if you have the same error in your logs if you have time | |
| 14:28:49 | sean-k-mooney | stephenfin: https://tinyurl.com/yw5ynd7d | |
| 14:29:00 | sean-k-mooney | its happening alot in the usptream jobs | |
| 14:29:18 | stephenfin | yes, I see it in the flamingo VM. I don't see it in the bobcat VM | |
| 14:29:29 | sean-k-mooney | across c9s ubuntu and debian | |
| 14:29:31 | stephenfin | and I've since deleted all the other VMs | |
| 14:29:39 | sean-k-mooney | ack | |
| 14:29:43 | sean-k-mooney | so that likely the problem | |
| 14:29:59 | sean-k-mooney | i dont know what the cause iss but thats a smoking gun | |
| 14:30:49 | sean-k-mooney | i.e. i dont knwo why it does not happen after restarting nova-comptue but apprently it does not always fail | |
| 14:33:06 | stephenfin | sean-k-mooney: dumped it into the bug https://bugs.launchpad.net/nova/+bug/2119114/comments/3 | |
| 14:33:24 | sean-k-mooney | cool i didnt have the link to that | |
| 15:12:55 | noonedeadpunk | ++ | |
| 15:24:08 | cardoe | sean-k-mooney: so not sure if you saw my follow up on the vlan spec. The issue you pointed out in chat about the id field and that dansmith brought up on gerrit itself isn't something that spec can really fix right now. It's just defining the behavior of nova code today (and the behavior that its been since June 2015). | |
| 15:25:04 | cardoe | So I'm really not sure what to fix cause that spec isn't about changing the behavior of existing "phy" interfaces. It's about including "vlan" interfaces. The "vlan" interface references to "phy" interface its running on. | |
| 15:26:22 | sean-k-mooney | the spec si free to extend or modify the schema and change the data that is stored in specific fileds | |
| 15:26:40 | sean-k-mooney | it is not possibel for nova ot know what the name of the Phy interface will be | |
| 15:26:50 | sean-k-mooney | so the value that are provdie are and alwasy were informational | |