| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-13 | |||
| 13:46:05 | sean-k-mooney | did you check the instance event log | |
| 13:46:28 | f0o | the way I can reproduce it is: create instance (ubuntu lts for instance), log in and do sudo reboot, vm is now stuck in paused | |
| 13:46:46 | f0o | no api-calls made | |
| 13:47:12 | sean-k-mooney | im not directly aware of anythin that would cause that | |
| 13:47:24 | sean-k-mooney | i have not seen it on ussuri at least | |
| 13:47:29 | sean-k-mooney | what release are you using | |
| 13:47:41 | f0o | it becomes very annoying for OS that run updates as part of their first boot (like coreos/flatcar) then the vm never becomes alive... | |
| 13:47:44 | f0o | I think I'm still on stein | |
| 13:48:10 | kashyap | sean-k-mooney: Yeah; it's a non-starter for other reasons too | |
| 13:48:19 | kashyap | sean-k-mooney: I've just only gave a cursory look :-) Please comment there | |
| 13:49:01 | sean-k-mooney | kashyap: i did | |
| 13:49:07 | sean-k-mooney | i was summerising for you :) | |
| 13:49:45 | kashyap | Hehe; thanks | |
| 13:56:48 | f0o | how would I go about debugging this weird paused issue? | |
| 13:56:51 | f0o | where to start looking? | |
| 13:57:24 | sean-k-mooney | f0o: you should be looking at the nova-compute logs for the instance in question and seeing what actions cause the instance to be paused | |
| 13:57:50 | sean-k-mooney | e.g. did the compute agent recived an event form libvirt saying the guest is now paused | |
| 13:57:57 | sean-k-mooney | or did it call libvirt to pause it | |
| 13:58:09 | f0o | ok, will do | |
| 13:58:15 | f0o | thnkas :) | |
| 13:58:22 | f0o | thanks* | |
| 14:02:34 | f0o | I have a few entries that have VM Paused (Lifecycle Event) in them. just gonna spawn a new instance and reboot it to see all the logs it generates | |
| 14:19:02 | f0o | [instance: 4d131088-4ca6-47bd-ab5c-47b9e0a7c996] VM Paused (Lifecycle Event) & During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (3). Updating power_state in the DB to match the hypervisor. | |
| 14:19:13 | f0o | [instance: 4d131088-4ca6-47bd-ab5c-47b9e0a7c996] Instance is paused unexpectedly. Ignore. | |
| 14:19:28 | f0o | so those 3 lines happen when I issue reboot inside the instance | |
| 14:20:16 | f0o | and now horizon shows it as Active/Paused and there's no resume action. I need to issue Hard-Reboot to kick it back alive | |
| 14:20:30 | sean-k-mooney | that look like libvirt is moving it to paused so | |
| 14:20:42 | sean-k-mooney | and then the compute agent is just updating the db to reflect that | |
| 14:22:41 | f0o | well /var/log/libvirt/qemu/instance-00000115.log sure is useless lol | |
| 14:26:50 | openstackgerrit | Radosław Piliszek proposed openstack/nova master: [docs] Fix a placement client's command https://review.opendev.org/762663 | |
| 14:29:54 | f0o | not sure where to go from here | |
| 14:30:14 | sean-k-mooney | can you paste the xml for the instace somewhere | |
| 14:30:19 | f0o | sure | |
| 14:30:47 | sean-k-mooney | there is an option to contol the reboot action but we dont set it but im just wondering if anything else is there that was generated by libvirt | |
| 14:31:11 | sean-k-mooney | libvirt modifies the xml we give it an fills in things like pci address automatically | |
| 14:31:56 | f0o | http://paste.openstack.org/show/oGhApPwoy0lNe5cKmV0G/ | |
| 14:32:35 | sean-k-mooney | <on_poweroff>destroy</on_poweroff> | |
| 14:32:36 | sean-k-mooney | <on_reboot>restart</on_reboot> | |
| 14:32:38 | sean-k-mooney | <on_crash>destroy</on_crash> | |
| 14:32:41 | sean-k-mooney | so those are what i expect | |
| 14:33:00 | sean-k-mooney | <on_reboot>restart</on_reboot> is what i was wondering about | |
| 14:33:16 | f0o | so that seems alright then? | |
| 14:33:55 | f0o | nova-compute-kvm version 2:20.0.0~rc1-0ubuntu3~cloud0 and qemu-kvm version 1:4.0+dfsg-0ubuntu9~cloud0 (just in case) | |
| 14:33:55 | sean-k-mooney | yes so what could be happening is a race between the periodic task and the vm reboot | |
| 14:34:10 | sean-k-mooney | is it happeing everytime | |
| 14:34:16 | sean-k-mooney | or jsut some times | |
| 14:34:29 | f0o | it's happening the majority of times | |
| 14:34:48 | f0o | there are some exceptions to it, but by now I'd wager that most times it gets stuck in paused | |
| 14:35:13 | sean-k-mooney | hum ok the interval is 10 minutes for the update by default | |
| 14:35:14 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.sync_power_state_interval | |
| 14:35:38 | sean-k-mooney | i assume you have not made that run faster | |
| 14:35:48 | f0o | nope | |
| 14:35:56 | f0o | config is very trivial, most are left as defaults | |
| 14:36:10 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.handle_virt_lifecycle_events | |
| 14:36:32 | sean-k-mooney | so you might want to set that to false | |
| 14:36:38 | sean-k-mooney | it look liek this is a know race | |
| 14:37:29 | f0o | ok let's give it a shot | |
| 14:37:35 | sean-k-mooney | if you have the interval at its default then setting that to false should be fine | |
| 14:38:21 | sean-k-mooney | that would have to be set on the compute nodes fyi | |
| 14:38:22 | f0o | set and restarting nova-compute | |
| 14:38:28 | sean-k-mooney | cool | |
| 14:38:45 | f0o | let's give it a shot | |
| 14:41:36 | sean-k-mooney | f0o: by the way the reason we default to haneling event i belive is for ironic | |
| 14:43:57 | f0o | makes sense | |
| 14:44:20 | f0o | at first I thought I was going insane but now that it became more frequent I figured I ask | |
| 14:47:04 | sean-k-mooney | hopefully that option will help | |
| 14:47:47 | sean-k-mooney | the main sideefct is that if you do poweroff in the guest then it wont be refected in the api/db until the interval expires | |
| 14:48:02 | sean-k-mooney | e.g. up to 10 mins form power off by efault | |
| 14:48:30 | f0o | that should be fine | |
| 14:48:37 | sean-k-mooney | that is genreally and ok tradeoff and you can adjust the interval if you want too | |
| 14:48:38 | f0o | worst case I lower the update interval | |
| 14:48:45 | sean-k-mooney | yep | |
| 14:53:00 | f0o | [ 652.025665] reboot: Restarting system .... let's hope it comes back up :D | |
| 14:56:10 | f0o | I can see the qemu process running and consuming 30% cpu but nothing seems alive in it | |
| 14:56:56 | f0o | vnc shows guest hasnt initialized display and view logs is only showing that reboot message | |
| 15:00:53 | f0o | During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (3). Updating power_state in the DB to match the hypervisor / Instance is paused unexpectedly. Ignore. | |
| 15:00:55 | f0o | again :< | |
| 15:02:19 | f0o | process is still running and eating up 30% cpu tho, not sure what it's actually doing there | |
| 15:49:26 | sean-k-mooney | f0o: that is sounding more like a qemu bug then an openstack one | |
| 15:49:41 | sean-k-mooney | it sound like its not actully restating properly | |
| 15:50:58 | f0o | cool :D | |
| 15:51:06 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: Testing system scope in tempest https://review.opendev.org/740124 | |
| 15:51:11 | f0o | just my luck | |
| 15:52:09 | tacco | hey everyone. Anyone knows why i only get 64VCPUs on a HV with 256 CPU Cores? over commiting ratio is 1.0 :( | |
| 15:52:28 | tacco | AMD EPYC 7742 64-Core Processor | |
| 15:53:24 | gibi | dansmith: hi! Do I understand correctly that the service version check at https://review.opendev.org/#/c/729563/17/nova/compute/api.py@4162 can see version 54 and allowing the shelve call with accelerators while the RPC is still can be manually pinned to < 5.13 and therefore the compute will not the accel_uuids param and therefore not handle the acceleratos properly? | |
| 15:53:53 | gibi | * will not get the accel_uuids | |
| 15:54:23 | dansmith | I'll have to look at that decorator, I think that just got added, right? | |
| 15:54:38 | tacco | i see processor: 255 and cpu cores: 64 in /proc/cpuinfo is this something like HT on Intel CPUs? but only can provide the "real" cores to the VM? | |
| 15:54:54 | tacco | cause i have 250CPUs in my flavor | |
| 15:55:03 | tacco | the VM then spawns with 64CPus | |
| 15:55:10 | f0o | Uhm | |
| 15:55:18 | f0o | 7742 shows as 64 cores | |
| 15:55:26 | f0o | HT stuff doesnt really count afaik | |
| 15:55:53 | dansmith | gibi: yikes, that makes an uncached cross-cell db lookup for every single call of that method :( | |
| 15:56:05 | gibi | dansmith: not too long ago, but it basically calls get_minimum_version_all_cells | |
| 15:56:27 | f0o | tacco: inside the vm you dont see the HT threads? | |
| 15:57:06 | tacco | in the VM at proc/cpuinfo i see only 64 but the vm was spawned with a flavor with 250 vcpus | |
| 15:57:47 | tacco | thats kinda strange and cli hypervisor list also shows 256 CPUs | |
| 15:59:20 | dansmith | gibi: commented on that patch, see if that helps | |