Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-13
16:13:51 sean-k-mooney tacco: what do you see on the host if you do nprc
16:14:04 gibi dansmith: agree. I missed the heavy weightness of that decorator impl in previous reviews.
16:14:19 sean-k-mooney tacco: can you provide the output of virsh capabilities for me in a paste and ill quickly take a look
16:14:21 dansmith gibi: oh actually, I forgot that we cache in the service object itself, so I guess not as bad, but still, it leaks the rpc problem so it'd be better to use that
16:14:55 tacco nproc shows me 256 as expected.
16:15:18 sean-k-mooney tacco: also the vm xml. if possible. i know some kernel have a process limit built in so just want to check if the xml has 64 or 250 cores
16:15:26 tacco one sec. will do so.
16:16:11 gibi dansmith: ack
16:19:45 tacco sean-k-mooney: http://paste.openstack.org/show/xs49lPLHXUPh43AeceZs/
16:20:05 tacco thats the dumpxml from the VM
16:20:07 sean-k-mooney <nova:vcpus>250</nova:vcpus>
16:20:13 tacco virsh capas on the way. :)
16:20:14 sean-k-mooney <vcpu placement='static'>250</vcpu>
16:20:26 sean-k-mooney so nova is telling libvirt/qemu to use 250
16:20:54 tacco ok, but inside the VM i only see 64 when i do cat /proc/cpuinfo hm.. maybe some problems with the image
16:20:56 sean-k-mooney <topology sockets='250' cores='1' threads='1'/> in a really dumb way but this is the default
16:21:25 tacco yes would enable numa pinning later
16:21:27 sean-k-mooney try adding hw_cpu_sockets=2 hw_cpu_theads=2 to the image
16:21:59 tacco would keep this in the flavor, because i would like to keep this as a seperate aggregate only available for "some" users
16:22:07 sean-k-mooney actull with 250 that wont work
16:22:22 sean-k-mooney tacco: try 248 cores
16:22:28 sean-k-mooney with those too set
16:22:35 tacco will do so. one sec.
16:22:59 sean-k-mooney im guessing the kernel is comiled to only support 64 sockets
16:23:00 gibi dansmith: I feel we have other places in the code where we rely purely on the service version to see if something is supported and we ignore the fact that rpc might be pinned.
16:23:25 dansmith gibi: those *should* be places where there isn't a specific rpc version in play,
16:23:36 dansmith in which case the service version is specifically what we want
16:23:50 gibi dansmith: yeah, that would be the proper usage of the service version only checks
16:23:55 dansmith service version alone should be used for cases where "I'm not saying anything new over rpc, but I depend on some action being done on the compute"
16:24:06 gibi dansmith: I will look through these checks to be sure
16:27:27 tacco sean-k-mooney: i guess this should be debian-cloud-image most things kernel related should be default
16:28:34 tacco ok flavor has now 248vcpus and | properties | hw:cpu_sockets='2', hw:cpu_theads='2' |
16:29:32 sean-k-mooney tacco: ya im not sure what the default sockets is upstream but 64 sound like a number peopel would choose as a default
16:29:49 tacco sean-k-mooney: yes this was also in my mind as first :D
16:30:08 tacco thats why i asked here.. because if this is known.. you should know it. :)
16:30:21 tacco this is the first time i have a HV with so many CPUs
16:30:43 tacco and i know some people here should have way larger setups and way more experience than i have. :)
16:31:07 sean-k-mooney i havent gone over 128 but i alway make my flavor mirror the host toplogy in terms or threads and sockets
16:31:18 tacco anyway. Thanks for your initial help. Know in know this could be related to the image. Will digg aroung and see what i can find.
16:31:29 gibi dansmith: does an ovo always travels through RPC with all its data and only backlevelled on the receiving side? So if a new field is added to an o.vo that is sent via RPC then we don't need to bump RPC version and the reciving side gets the new field if the code on the reciving side has the new field definition in its own ovo class independently of the RPC api version?
16:31:37 sean-k-mooney tacco: did that work by the way
16:31:41 sean-k-mooney the updated flavor
16:31:43 tacco ususaly you also don't want such huge flavors.
16:31:53 tacco nope updated flavor also has only 64cpus in the cm
16:31:57 tacco s/cm/vm/
16:32:07 sean-k-mooney really
16:32:12 sean-k-mooney that is odd
16:32:31 sean-k-mooney am can you quickly check the qemu string jut to triple check
16:32:33 tacco and i double checked the xml if that change affected the xml and reflects my change to the flavor
16:32:47 dansmith gibi: we always send the version of the object we have. If the receiving side determines it's too new, it calls to conductor and asks for conductor to backlevel it. Conductor can either return an object that has an older version (i.e. if it _can_ backlevel it) or it can refuse
16:32:52 sean-k-mooney it likely a qemu or guest kernel limiation however
16:33:13 tacco yes. Thanks thats where i would like to digg more deeper
16:33:35 gibi dansmith: good to know, that in this case there is an extra call back to the conductor
16:34:15 dansmith gibi: but that's just for the object(s), not the rpc signature itself, and the idea was that during an upgrade you have extra conductor load to handle all the backports, but as you upgrade everything that just disappears
16:35:50 gibi dansmith: yes, it is clear that this is only possible for o.vos itself, not for the whole RPC method signature.
16:36:05 gibi dansmith: but then in a new-field-in-an-ovo case the service version check is enough
16:36:29 gibi as if the compute service version is new enough then it will understand a new ovo version with the extra field
16:36:30 sean-k-mooney tacco: https://github.com/torvalds/linux/blob/master/arch/x86/Kconfig#L994-L1005
16:36:53 dansmith yeah, and in some cases, it's possible to backlevel the object so we can just deal with it on the receiving end, but not if you require specific behavior
16:37:15 sean-k-mooney tacco: so it should be 512 so proably a qemu issue
16:38:41 tacco i see. Thanks.
16:39:08 gibi dansmith: thanks again, this make sense now
16:39:41 dansmith cool
16:40:14 sean-k-mooney tacco: it look like the max cpus depends on the machine type you enable
16:41:01 gibi I think this is a good time to finish my week and let the new understanding solidifies :)
16:41:14 gibi have a nice weekend folks o/
16:41:24 lyarwood \o
16:44:11 tacco sean-k-mooney: here is also the capa list of virsh http://paste.openstack.org/show/nxpSZUUrICfvvRJBgosL/
16:45:20 tacco this machine type? <type arch='x86_64' machine='pc-i440fx-4.0'>hvm</type>
16:45:47 sean-k-mooney ya you are using the pc machine type but it should in theory support up to 256
16:47:00 sean-k-mooney tacco: if you look in teh output it has the limits
16:47:11 sean-k-mooney line 1193
16:47:18 sean-k-mooney <machine maxCpus='255'>pc-i440fx-4.0</machine>
16:47:28 tacco yes i also found this in there.
16:47:45 tacco ok. so no "real"limitations more of a missconfiguration or bug. :)
16:48:01 sean-k-mooney so this is looking like a guest issue or a bug ya
16:48:13 sean-k-mooney you could try with another image
16:48:26 tacco i already struggled about that limitations that you can only have 8 disks inside a qemu vm. :)
16:48:48 tacco yes will do so but in general this was the debian cloud image with only minimal changes. but will test next week.
16:48:49 sean-k-mooney like a fedora or tubleweed image
16:49:01 tacco ok. will do so. but for today im done.
16:49:09 sean-k-mooney cool
16:49:13 sean-k-mooney enjoy your weekend
16:49:31 tacco thanks a lot for your help so far. Will get back to you next week. have a nice weekend as well
17:04:27 openstackgerrit Balazs Gibizer proposed openstack/nova master: Restore retrying the RPC connection to conductor https://review.opendev.org/762633
17:17:54 dopereira Hi, everyone. I'm a new contributor and just joined the OpenStack community.
17:18:08 dopereira I recently started to work on a project that develops a Openstack distribuition with comercial support. Hopefully I will be contributing to the upstream Openstack as well.
17:18:19 dopereira I recently started to work on a project that develops a Openstack distribuition with comercial support. Hopefully I will be contributing to the upstream Openstack as well.
17:18:40 dopereira As part of my ramp up process, I'm learning how to contribute with Openstack, and I was asked to take care of a low-hanging-fruit bug.
17:18:58 dopereira I choose this one: https://bugs.launchpad.net/nova/+bug/1888927 and already have a patch for it: https://review.opendev.org/#/c/762433/4
17:18:58 openstack Launchpad bug 1888927 in OpenStack Compute (nova) "cell_v2 update_cell cell0 get transport_url from config file" [Low,In progress] - Assigned to Daniel de Oliveira Pereira (danielpereira01)
17:19:10 dopereira Could you guys please take a look and help to review it?
17:21:51 dopereira Also, the VMware NSX CI check failed. How can I started recheck for it? It does not provide this information
17:23:18 sean-k-mooney dopereira: have you submitted the patch to gerrit
17:23:29 sean-k-mooney ah you have
17:24:23 sean-k-mooney dopereira: when a third party ci fails it leaves a comment telling you how to recheck that specific ci
17:24:47 sean-k-mooney dopereira: you have to click the toggle extra ci button to see it
17:24:59 sean-k-mooney oh but they dont
17:25:24 sean-k-mooney it should be listed here
17:25:25 dopereira it seems that's not the case for VMware NSX CI

Earlier   Later