Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-13
14:56:56 f0o vnc shows guest hasnt initialized display and view logs is only showing that reboot message
15:00:53 f0o During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (3). Updating power_state in the DB to match the hypervisor / Instance is paused unexpectedly. Ignore.
15:00:55 f0o again :<
15:02:19 f0o process is still running and eating up 30% cpu tho, not sure what it's actually doing there
15:49:26 sean-k-mooney f0o: that is sounding more like a qemu bug then an openstack one
15:49:41 sean-k-mooney it sound like its not actully restating properly
15:50:58 f0o cool :D
15:51:06 openstackgerrit Ghanshyam Mann proposed openstack/nova master: DNM: Testing system scope in tempest https://review.opendev.org/740124
15:51:11 f0o just my luck
15:52:09 tacco hey everyone. Anyone knows why i only get 64VCPUs on a HV with 256 CPU Cores? over commiting ratio is 1.0 :(
15:52:28 tacco AMD EPYC 7742 64-Core Processor
15:53:24 gibi dansmith: hi! Do I understand correctly that the service version check at https://review.opendev.org/#/c/729563/17/nova/compute/api.py@4162 can see version 54 and allowing the shelve call with accelerators while the RPC is still can be manually pinned to < 5.13 and therefore the compute will not the accel_uuids param and therefore not handle the acceleratos properly?
15:53:53 gibi * will not get the accel_uuids
15:54:23 dansmith I'll have to look at that decorator, I think that just got added, right?
15:54:38 tacco i see processor: 255 and cpu cores: 64 in /proc/cpuinfo is this something like HT on Intel CPUs? but only can provide the "real" cores to the VM?
15:54:54 tacco cause i have 250CPUs in my flavor
15:55:03 tacco the VM then spawns with 64CPus
15:55:10 f0o Uhm
15:55:18 f0o 7742 shows as 64 cores
15:55:26 f0o HT stuff doesnt really count afaik
15:55:53 dansmith gibi: yikes, that makes an uncached cross-cell db lookup for every single call of that method :(
15:56:05 gibi dansmith: not too long ago, but it basically calls get_minimum_version_all_cells
15:56:27 f0o tacco: inside the vm you dont see the HT threads?
15:57:06 tacco in the VM at proc/cpuinfo i see only 64 but the vm was spawned with a flavor with 250 vcpus
15:57:47 tacco thats kinda strange and cli hypervisor list also shows 256 CPUs
15:59:20 dansmith gibi: commented on that patch, see if that helps
15:59:34 gibi thanks
16:05:36 tacco i see this seems to be two physical CPUs with 64 cores and 128 threads. that makes sense. But no clue why the VM only got 64Cpus if in the flavor are 250 specified. anyway. will have to dive deeper :D
16:08:10 f0o I'm happy to swap issues with you tacoc :D
16:08:25 f0o other than my dyslexia today lol
16:09:47 tacco always nice to be a usefull person :D
16:12:30 gibi dansmith: you confirmed my fears, thanks
16:12:36 dansmith gibi: ack
16:12:54 dansmith gibi: honestly that decorator seems like a bad idea to me.. that's a lot of overhead hidden in a decorator
16:13:22 dansmith it should at least use a cached value, but I'd prefer we did things like I described, which is rely on the rpc version and a raise from rpcapi
16:13:30 sean-k-mooney tacco: there was a libvirt bug realted to numa reporting and amd but perhaps there are others
16:13:51 sean-k-mooney tacco: what do you see on the host if you do nprc
16:14:04 gibi dansmith: agree. I missed the heavy weightness of that decorator impl in previous reviews.
16:14:19 sean-k-mooney tacco: can you provide the output of virsh capabilities for me in a paste and ill quickly take a look
16:14:21 dansmith gibi: oh actually, I forgot that we cache in the service object itself, so I guess not as bad, but still, it leaks the rpc problem so it'd be better to use that
16:14:55 tacco nproc shows me 256 as expected.
16:15:18 sean-k-mooney tacco: also the vm xml. if possible. i know some kernel have a process limit built in so just want to check if the xml has 64 or 250 cores
16:15:26 tacco one sec. will do so.
16:16:11 gibi dansmith: ack
16:19:45 tacco sean-k-mooney: http://paste.openstack.org/show/xs49lPLHXUPh43AeceZs/
16:20:05 tacco thats the dumpxml from the VM
16:20:07 sean-k-mooney <nova:vcpus>250</nova:vcpus>
16:20:13 tacco virsh capas on the way. :)
16:20:14 sean-k-mooney <vcpu placement='static'>250</vcpu>
16:20:26 sean-k-mooney so nova is telling libvirt/qemu to use 250
16:20:54 tacco ok, but inside the VM i only see 64 when i do cat /proc/cpuinfo hm.. maybe some problems with the image
16:20:56 sean-k-mooney <topology sockets='250' cores='1' threads='1'/> in a really dumb way but this is the default
16:21:25 tacco yes would enable numa pinning later
16:21:27 sean-k-mooney try adding hw_cpu_sockets=2 hw_cpu_theads=2 to the image
16:21:59 tacco would keep this in the flavor, because i would like to keep this as a seperate aggregate only available for "some" users
16:22:07 sean-k-mooney actull with 250 that wont work
16:22:22 sean-k-mooney tacco: try 248 cores
16:22:28 sean-k-mooney with those too set
16:22:35 tacco will do so. one sec.
16:22:59 sean-k-mooney im guessing the kernel is comiled to only support 64 sockets
16:23:00 gibi dansmith: I feel we have other places in the code where we rely purely on the service version to see if something is supported and we ignore the fact that rpc might be pinned.
16:23:25 dansmith gibi: those *should* be places where there isn't a specific rpc version in play,
16:23:36 dansmith in which case the service version is specifically what we want
16:23:50 gibi dansmith: yeah, that would be the proper usage of the service version only checks
16:23:55 dansmith service version alone should be used for cases where "I'm not saying anything new over rpc, but I depend on some action being done on the compute"
16:24:06 gibi dansmith: I will look through these checks to be sure
16:27:27 tacco sean-k-mooney: i guess this should be debian-cloud-image most things kernel related should be default
16:28:34 tacco ok flavor has now 248vcpus and | properties | hw:cpu_sockets='2', hw:cpu_theads='2' |
16:29:32 sean-k-mooney tacco: ya im not sure what the default sockets is upstream but 64 sound like a number peopel would choose as a default
16:29:49 tacco sean-k-mooney: yes this was also in my mind as first :D
16:30:08 tacco thats why i asked here.. because if this is known.. you should know it. :)
16:30:21 tacco this is the first time i have a HV with so many CPUs
16:30:43 tacco and i know some people here should have way larger setups and way more experience than i have. :)
16:31:07 sean-k-mooney i havent gone over 128 but i alway make my flavor mirror the host toplogy in terms or threads and sockets
16:31:18 tacco anyway. Thanks for your initial help. Know in know this could be related to the image. Will digg aroung and see what i can find.
16:31:29 gibi dansmith: does an ovo always travels through RPC with all its data and only backlevelled on the receiving side? So if a new field is added to an o.vo that is sent via RPC then we don't need to bump RPC version and the reciving side gets the new field if the code on the reciving side has the new field definition in its own ovo class independently of the RPC api version?
16:31:37 sean-k-mooney tacco: did that work by the way
16:31:41 sean-k-mooney the updated flavor
16:31:43 tacco ususaly you also don't want such huge flavors.
16:31:53 tacco nope updated flavor also has only 64cpus in the cm
16:31:57 tacco s/cm/vm/
16:32:07 sean-k-mooney really
16:32:12 sean-k-mooney that is odd
16:32:31 sean-k-mooney am can you quickly check the qemu string jut to triple check
16:32:33 tacco and i double checked the xml if that change affected the xml and reflects my change to the flavor
16:32:47 dansmith gibi: we always send the version of the object we have. If the receiving side determines it's too new, it calls to conductor and asks for conductor to backlevel it. Conductor can either return an object that has an older version (i.e. if it _can_ backlevel it) or it can refuse
16:32:52 sean-k-mooney it likely a qemu or guest kernel limiation however
16:33:13 tacco yes. Thanks thats where i would like to digg more deeper
16:33:35 gibi dansmith: good to know, that in this case there is an extra call back to the conductor
16:34:15 dansmith gibi: but that's just for the object(s), not the rpc signature itself, and the idea was that during an upgrade you have extra conductor load to handle all the backports, but as you upgrade everything that just disappears
16:35:50 gibi dansmith: yes, it is clear that this is only possible for o.vos itself, not for the whole RPC method signature.
16:36:05 gibi dansmith: but then in a new-field-in-an-ovo case the service version check is enough
16:36:29 gibi as if the compute service version is new enough then it will understand a new ovo version with the extra field
16:36:30 sean-k-mooney tacco: https://github.com/torvalds/linux/blob/master/arch/x86/Kconfig#L994-L1005
16:36:53 dansmith yeah, and in some cases, it's possible to backlevel the object so we can just deal with it on the receiving end, but not if you require specific behavior
16:37:15 sean-k-mooney tacco: so it should be 512 so proably a qemu issue
16:38:41 tacco i see. Thanks.
16:39:08 gibi dansmith: thanks again, this make sense now
16:39:41 dansmith cool
16:40:14 sean-k-mooney tacco: it look like the max cpus depends on the machine type you enable

Earlier   Later