| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-13 | |||
| 10:56:17 | gibi | and thanks | |
| 10:59:02 | bauzas | gibi: +W'd stephenfin's one | |
| 10:59:07 | gibi | awesome, thanks | |
| 10:59:28 | bauzas | gibi: for your own change, I have a concern, where are we looking at the conductor ? | |
| 11:00:11 | gibi | bauzas: https://github.com/openstack/nova/blob/eb279e9a5676f4142cce4700c3097ecc14161895/nova/service.py#L115 | |
| 11:00:27 | gibi | it is well hidden gem of the Service object | |
| 11:01:22 | bauzas | thanks | |
| 11:01:23 | bauzas | looking | |
| 11:01:50 | bauzas | a-ha, now I remember | |
| 11:02:13 | bauzas | https://github.com/openstack/nova/blob/eb279e9a5676f4142cce4700c3097ecc14161895/nova/service.py#L113 is for the computes | |
| 11:02:50 | gibi | yes | |
| 11:03:34 | bauzas | gibi: okay, then waiting for your test | |
| 11:03:43 | bauzas | gibi: just do a functional test honestly | |
| 11:03:44 | gibi | OK, working on it | |
| 11:03:53 | gibi | yes, it will be a functional one | |
| 11:03:55 | bauzas | would be simplier and better | |
| 11:04:07 | bauzas | cool | |
| 11:04:31 | bauzas | gibi: fwiw, I'll be lunching, but ping me when you're done and I'll look at it when I'm back | |
| 11:04:40 | gibi | sure. have a nice one! | |
| 11:24:43 | gibi | bauzas: bah, in the func test env the compute service does not need a running conductor to start up, I guess something is mocked in the indirection API | |
| 11:31:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Restore retrying the RCP connection to conductor https://review.opendev.org/762633 | |
| 11:32:43 | gibi | with a less good but still covering unit test ^^ | |
| 12:09:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Disable *all* virt services during negative tests https://review.opendev.org/762623 | |
| 12:15:48 | openstackgerrit | chengsheng proposed openstack/nova master: Add hypervisor CPU feature check during live migration https://review.opendev.org/762330 | |
| 12:25:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Doc that [database]connection is not for nova-compute https://review.opendev.org/762647 | |
| 12:35:20 | kashyap | Anyone else with CPU and live migration interest, have a look at this review, from above, chengsheng ... I'm not comfortable parsing /proc/cpuinfo straight-up (and a couple of other issues in there): https://review.opendev.org/#/c/762330/6 | |
| 12:35:26 | kashyap | Err, wrong link | |
| 12:35:36 | kashyap | Correct one: https://review.opendev.org/#/c/762330/ | |
| 12:52:57 | sean-k-mooney | we have said no to that in the past | |
| 12:53:08 | sean-k-mooney | and required it the info to be exposed in libvirt | |
| 13:06:13 | sean-k-mooney | kashyap: as written that patch will break upgrades and prevent migrating a vm to newwer hardware so its a nonstarter | |
| 13:07:13 | sean-k-mooney | its also not filtering out no virutaliseable feature that dont affact the cpu flags exposed to the vm | |
| 13:40:15 | openstackgerrit | Merged openstack/nova master: functional: Wait for revert resize to complete https://review.opendev.org/762543 | |
| 13:44:11 | f0o | I'm a bit confused... More and more linux guests cant reboot. their instances go from running to paused for some reason and need to be hard-reboot from horizon/api to be able to become alive again. Is a guest no longer allowed to reboot by itself? Am I missing a setting somewhere? (using kvm/qemu) | |
| 13:44:48 | sean-k-mooney | f0o: they are allowed too yes | |
| 13:45:11 | f0o | why are they ending up in "paused" then? | |
| 13:45:30 | sean-k-mooney | they would only be moved to paused if the state in the db for the vm was paused and you had the sync power state option set | |
| 13:45:50 | f0o | what would cause the db to think it's paused? | |
| 13:45:58 | sean-k-mooney | only an api request | |
| 13:46:05 | sean-k-mooney | did you check the instance event log | |
| 13:46:28 | f0o | the way I can reproduce it is: create instance (ubuntu lts for instance), log in and do sudo reboot, vm is now stuck in paused | |
| 13:46:46 | f0o | no api-calls made | |
| 13:47:12 | sean-k-mooney | im not directly aware of anythin that would cause that | |
| 13:47:24 | sean-k-mooney | i have not seen it on ussuri at least | |
| 13:47:29 | sean-k-mooney | what release are you using | |
| 13:47:41 | f0o | it becomes very annoying for OS that run updates as part of their first boot (like coreos/flatcar) then the vm never becomes alive... | |
| 13:47:44 | f0o | I think I'm still on stein | |
| 13:48:10 | kashyap | sean-k-mooney: Yeah; it's a non-starter for other reasons too | |
| 13:48:19 | kashyap | sean-k-mooney: I've just only gave a cursory look :-) Please comment there | |
| 13:49:01 | sean-k-mooney | kashyap: i did | |
| 13:49:07 | sean-k-mooney | i was summerising for you :) | |
| 13:49:45 | kashyap | Hehe; thanks | |
| 13:56:48 | f0o | how would I go about debugging this weird paused issue? | |
| 13:56:51 | f0o | where to start looking? | |
| 13:57:24 | sean-k-mooney | f0o: you should be looking at the nova-compute logs for the instance in question and seeing what actions cause the instance to be paused | |
| 13:57:50 | sean-k-mooney | e.g. did the compute agent recived an event form libvirt saying the guest is now paused | |
| 13:57:57 | sean-k-mooney | or did it call libvirt to pause it | |
| 13:58:09 | f0o | ok, will do | |
| 13:58:15 | f0o | thnkas :) | |
| 13:58:22 | f0o | thanks* | |
| 14:02:34 | f0o | I have a few entries that have VM Paused (Lifecycle Event) in them. just gonna spawn a new instance and reboot it to see all the logs it generates | |
| 14:19:02 | f0o | [instance: 4d131088-4ca6-47bd-ab5c-47b9e0a7c996] VM Paused (Lifecycle Event) & During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (3). Updating power_state in the DB to match the hypervisor. | |
| 14:19:13 | f0o | [instance: 4d131088-4ca6-47bd-ab5c-47b9e0a7c996] Instance is paused unexpectedly. Ignore. | |
| 14:19:28 | f0o | so those 3 lines happen when I issue reboot inside the instance | |
| 14:20:16 | f0o | and now horizon shows it as Active/Paused and there's no resume action. I need to issue Hard-Reboot to kick it back alive | |
| 14:20:30 | sean-k-mooney | that look like libvirt is moving it to paused so | |
| 14:20:42 | sean-k-mooney | and then the compute agent is just updating the db to reflect that | |
| 14:22:41 | f0o | well /var/log/libvirt/qemu/instance-00000115.log sure is useless lol | |
| 14:26:50 | openstackgerrit | Radosław Piliszek proposed openstack/nova master: [docs] Fix a placement client's command https://review.opendev.org/762663 | |
| 14:29:54 | f0o | not sure where to go from here | |
| 14:30:14 | sean-k-mooney | can you paste the xml for the instace somewhere | |
| 14:30:19 | f0o | sure | |
| 14:30:47 | sean-k-mooney | there is an option to contol the reboot action but we dont set it but im just wondering if anything else is there that was generated by libvirt | |
| 14:31:11 | sean-k-mooney | libvirt modifies the xml we give it an fills in things like pci address automatically | |
| 14:31:56 | f0o | http://paste.openstack.org/show/oGhApPwoy0lNe5cKmV0G/ | |
| 14:32:35 | sean-k-mooney | <on_poweroff>destroy</on_poweroff> | |
| 14:32:36 | sean-k-mooney | <on_reboot>restart</on_reboot> | |
| 14:32:38 | sean-k-mooney | <on_crash>destroy</on_crash> | |
| 14:32:41 | sean-k-mooney | so those are what i expect | |
| 14:33:00 | sean-k-mooney | <on_reboot>restart</on_reboot> is what i was wondering about | |
| 14:33:16 | f0o | so that seems alright then? | |
| 14:33:55 | sean-k-mooney | yes so what could be happening is a race between the periodic task and the vm reboot | |
| 14:33:55 | f0o | nova-compute-kvm version 2:20.0.0~rc1-0ubuntu3~cloud0 and qemu-kvm version 1:4.0+dfsg-0ubuntu9~cloud0 (just in case) | |
| 14:34:10 | sean-k-mooney | is it happeing everytime | |
| 14:34:16 | sean-k-mooney | or jsut some times | |
| 14:34:29 | f0o | it's happening the majority of times | |
| 14:34:48 | f0o | there are some exceptions to it, but by now I'd wager that most times it gets stuck in paused | |
| 14:35:13 | sean-k-mooney | hum ok the interval is 10 minutes for the update by default | |
| 14:35:14 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.sync_power_state_interval | |
| 14:35:38 | sean-k-mooney | i assume you have not made that run faster | |
| 14:35:48 | f0o | nope | |
| 14:35:56 | f0o | config is very trivial, most are left as defaults | |
| 14:36:10 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.handle_virt_lifecycle_events | |
| 14:36:32 | sean-k-mooney | so you might want to set that to false | |
| 14:36:38 | sean-k-mooney | it look liek this is a know race | |
| 14:37:29 | f0o | ok let's give it a shot | |
| 14:37:35 | sean-k-mooney | if you have the interval at its default then setting that to false should be fine | |
| 14:38:21 | sean-k-mooney | that would have to be set on the compute nodes fyi | |
| 14:38:22 | f0o | set and restarting nova-compute | |
| 14:38:28 | sean-k-mooney | cool | |