| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-13 | |||
| 14:33:00 | sean-k-mooney | <on_reboot>restart</on_reboot> is what i was wondering about | |
| 14:33:16 | f0o | so that seems alright then? | |
| 14:33:55 | f0o | nova-compute-kvm version 2:20.0.0~rc1-0ubuntu3~cloud0 and qemu-kvm version 1:4.0+dfsg-0ubuntu9~cloud0 (just in case) | |
| 14:33:55 | sean-k-mooney | yes so what could be happening is a race between the periodic task and the vm reboot | |
| 14:34:10 | sean-k-mooney | is it happeing everytime | |
| 14:34:16 | sean-k-mooney | or jsut some times | |
| 14:34:29 | f0o | it's happening the majority of times | |
| 14:34:48 | f0o | there are some exceptions to it, but by now I'd wager that most times it gets stuck in paused | |
| 14:35:13 | sean-k-mooney | hum ok the interval is 10 minutes for the update by default | |
| 14:35:14 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.sync_power_state_interval | |
| 14:35:38 | sean-k-mooney | i assume you have not made that run faster | |
| 14:35:48 | f0o | nope | |
| 14:35:56 | f0o | config is very trivial, most are left as defaults | |
| 14:36:10 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.handle_virt_lifecycle_events | |
| 14:36:32 | sean-k-mooney | so you might want to set that to false | |
| 14:36:38 | sean-k-mooney | it look liek this is a know race | |
| 14:37:29 | f0o | ok let's give it a shot | |
| 14:37:35 | sean-k-mooney | if you have the interval at its default then setting that to false should be fine | |
| 14:38:21 | sean-k-mooney | that would have to be set on the compute nodes fyi | |
| 14:38:22 | f0o | set and restarting nova-compute | |
| 14:38:28 | sean-k-mooney | cool | |
| 14:38:45 | f0o | let's give it a shot | |
| 14:41:36 | sean-k-mooney | f0o: by the way the reason we default to haneling event i belive is for ironic | |
| 14:43:57 | f0o | makes sense | |
| 14:44:20 | f0o | at first I thought I was going insane but now that it became more frequent I figured I ask | |
| 14:47:04 | sean-k-mooney | hopefully that option will help | |
| 14:47:47 | sean-k-mooney | the main sideefct is that if you do poweroff in the guest then it wont be refected in the api/db until the interval expires | |
| 14:48:02 | sean-k-mooney | e.g. up to 10 mins form power off by efault | |
| 14:48:30 | f0o | that should be fine | |
| 14:48:37 | sean-k-mooney | that is genreally and ok tradeoff and you can adjust the interval if you want too | |
| 14:48:38 | f0o | worst case I lower the update interval | |
| 14:48:45 | sean-k-mooney | yep | |
| 14:53:00 | f0o | [ 652.025665] reboot: Restarting system .... let's hope it comes back up :D | |
| 14:56:10 | f0o | I can see the qemu process running and consuming 30% cpu but nothing seems alive in it | |
| 14:56:56 | f0o | vnc shows guest hasnt initialized display and view logs is only showing that reboot message | |
| 15:00:53 | f0o | During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (3). Updating power_state in the DB to match the hypervisor / Instance is paused unexpectedly. Ignore. | |
| 15:00:55 | f0o | again :< | |
| 15:02:19 | f0o | process is still running and eating up 30% cpu tho, not sure what it's actually doing there | |
| 15:49:26 | sean-k-mooney | f0o: that is sounding more like a qemu bug then an openstack one | |
| 15:49:41 | sean-k-mooney | it sound like its not actully restating properly | |
| 15:50:58 | f0o | cool :D | |
| 15:51:06 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: Testing system scope in tempest https://review.opendev.org/740124 | |
| 15:51:11 | f0o | just my luck | |
| 15:52:09 | tacco | hey everyone. Anyone knows why i only get 64VCPUs on a HV with 256 CPU Cores? over commiting ratio is 1.0 :( | |
| 15:52:28 | tacco | AMD EPYC 7742 64-Core Processor | |
| 15:53:24 | gibi | dansmith: hi! Do I understand correctly that the service version check at https://review.opendev.org/#/c/729563/17/nova/compute/api.py@4162 can see version 54 and allowing the shelve call with accelerators while the RPC is still can be manually pinned to < 5.13 and therefore the compute will not the accel_uuids param and therefore not handle the acceleratos properly? | |
| 15:53:53 | gibi | * will not get the accel_uuids | |
| 15:54:23 | dansmith | I'll have to look at that decorator, I think that just got added, right? | |
| 15:54:38 | tacco | i see processor: 255 and cpu cores: 64 in /proc/cpuinfo is this something like HT on Intel CPUs? but only can provide the "real" cores to the VM? | |
| 15:54:54 | tacco | cause i have 250CPUs in my flavor | |
| 15:55:03 | tacco | the VM then spawns with 64CPus | |
| 15:55:10 | f0o | Uhm | |
| 15:55:18 | f0o | 7742 shows as 64 cores | |
| 15:55:26 | f0o | HT stuff doesnt really count afaik | |
| 15:55:53 | dansmith | gibi: yikes, that makes an uncached cross-cell db lookup for every single call of that method :( | |
| 15:56:05 | gibi | dansmith: not too long ago, but it basically calls get_minimum_version_all_cells | |
| 15:56:27 | f0o | tacco: inside the vm you dont see the HT threads? | |
| 15:57:06 | tacco | in the VM at proc/cpuinfo i see only 64 but the vm was spawned with a flavor with 250 vcpus | |
| 15:57:47 | tacco | thats kinda strange and cli hypervisor list also shows 256 CPUs | |
| 15:59:20 | dansmith | gibi: commented on that patch, see if that helps | |
| 15:59:34 | gibi | thanks | |
| 16:05:36 | tacco | i see this seems to be two physical CPUs with 64 cores and 128 threads. that makes sense. But no clue why the VM only got 64Cpus if in the flavor are 250 specified. anyway. will have to dive deeper :D | |
| 16:08:10 | f0o | I'm happy to swap issues with you tacoc :D | |
| 16:08:25 | f0o | other than my dyslexia today lol | |
| 16:09:47 | tacco | always nice to be a usefull person :D | |
| 16:12:30 | gibi | dansmith: you confirmed my fears, thanks | |
| 16:12:36 | dansmith | gibi: ack | |
| 16:12:54 | dansmith | gibi: honestly that decorator seems like a bad idea to me.. that's a lot of overhead hidden in a decorator | |
| 16:13:22 | dansmith | it should at least use a cached value, but I'd prefer we did things like I described, which is rely on the rpc version and a raise from rpcapi | |
| 16:13:30 | sean-k-mooney | tacco: there was a libvirt bug realted to numa reporting and amd but perhaps there are others | |
| 16:13:51 | sean-k-mooney | tacco: what do you see on the host if you do nprc | |
| 16:14:04 | gibi | dansmith: agree. I missed the heavy weightness of that decorator impl in previous reviews. | |
| 16:14:19 | sean-k-mooney | tacco: can you provide the output of virsh capabilities for me in a paste and ill quickly take a look | |
| 16:14:21 | dansmith | gibi: oh actually, I forgot that we cache in the service object itself, so I guess not as bad, but still, it leaks the rpc problem so it'd be better to use that | |
| 16:14:55 | tacco | nproc shows me 256 as expected. | |
| 16:15:18 | sean-k-mooney | tacco: also the vm xml. if possible. i know some kernel have a process limit built in so just want to check if the xml has 64 or 250 cores | |
| 16:15:26 | tacco | one sec. will do so. | |
| 16:16:11 | gibi | dansmith: ack | |
| 16:19:45 | tacco | sean-k-mooney: http://paste.openstack.org/show/xs49lPLHXUPh43AeceZs/ | |
| 16:20:05 | tacco | thats the dumpxml from the VM | |
| 16:20:07 | sean-k-mooney | <nova:vcpus>250</nova:vcpus> | |
| 16:20:13 | tacco | virsh capas on the way. :) | |
| 16:20:14 | sean-k-mooney | <vcpu placement='static'>250</vcpu> | |
| 16:20:26 | sean-k-mooney | so nova is telling libvirt/qemu to use 250 | |
| 16:20:54 | tacco | ok, but inside the VM i only see 64 when i do cat /proc/cpuinfo hm.. maybe some problems with the image | |
| 16:20:56 | sean-k-mooney | <topology sockets='250' cores='1' threads='1'/> in a really dumb way but this is the default | |
| 16:21:25 | tacco | yes would enable numa pinning later | |
| 16:21:27 | sean-k-mooney | try adding hw_cpu_sockets=2 hw_cpu_theads=2 to the image | |
| 16:21:59 | tacco | would keep this in the flavor, because i would like to keep this as a seperate aggregate only available for "some" users | |
| 16:22:07 | sean-k-mooney | actull with 250 that wont work | |
| 16:22:22 | sean-k-mooney | tacco: try 248 cores | |
| 16:22:28 | sean-k-mooney | with those too set | |
| 16:22:35 | tacco | will do so. one sec. | |
| 16:22:59 | sean-k-mooney | im guessing the kernel is comiled to only support 64 sockets | |
| 16:23:00 | gibi | dansmith: I feel we have other places in the code where we rely purely on the service version to see if something is supported and we ignore the fact that rpc might be pinned. | |
| 16:23:25 | dansmith | gibi: those *should* be places where there isn't a specific rpc version in play, | |
| 16:23:36 | dansmith | in which case the service version is specifically what we want | |
| 16:23:50 | gibi | dansmith: yeah, that would be the proper usage of the service version only checks | |
| 16:23:55 | dansmith | service version alone should be used for cases where "I'm not saying anything new over rpc, but I depend on some action being done on the compute" | |
| 16:24:06 | gibi | dansmith: I will look through these checks to be sure | |