| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-13 | |||
| 09:45:17 | kashyap | bauzas: "There is no Gate" | |
| 09:50:48 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Filter instances by tenant_id https://review.opendev.org/737241 | |
| 09:55:57 | gibi | bauzas: I use this to see if there any total brokenness on the gate https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fnova&pipeline=gate | |
| 09:56:57 | gibi | bauzas: also we have 83 unclassified failures on the gate by http://status.openstack.org/elastic-recheck/data/integrated_gate.html | |
| 09:57:07 | gibi | so the answer is we don't know | |
| 09:58:24 | openstack | Launchpad bug 1903979 in OpenStack Compute (nova) "nova-live-migration job fails during evacuate negative test" [High,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 09:58:24 | gibi | bauzas: I did filed https://bugs.launchpad.net/nova/+bug/1903979 and lyarwood is trying to find the root of it | |
| 10:01:56 | openstack | Launchpad bug 1904179 in OpenStack Compute (nova) "[doc] define the minimal mandatory configuration for each nova service" [Medium,Confirmed] | |
| 10:01:56 | gibi | dansmith, stephenfin, gmann: filed a doc bug to document the minimal config for each nova services https://bugs.launchpad.net/nova/+bug/1904179 | |
| 10:12:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Warn when starting services with older than N-1 computes https://review.opendev.org/761923 | |
| 10:13:52 | bauzas | gibi: ack thanks | |
| 10:14:14 | gibi | bauzas: this is also new for me on master https://1d1ac8c4d7a38514d020-4dcc482f43713c13ecd75f64a0eb3df3.ssl.cf1.rackcdn.com/762319/1/check/tempest-integrated-compute/0b7fa72/controller/logs/screen-n-cpu.txt | |
| 10:14:31 | gibi | bauzas: filing a bug now for that too | |
| 10:14:43 | bauzas | gibi: http://status.openstack.org/elastic-recheck/#1686542 | |
| 10:15:25 | bauzas | looks like we have problems with the queues | |
| 10:15:35 | gibi | the we have multiple problems :) | |
| 10:15:35 | bauzas | anyway;, /me needs to taxi his kids | |
| 10:15:37 | gibi | yey | |
| 10:15:51 | bauzas | looks like a super fancy Friday | |
| 10:15:55 | bauzas | \o/ | |
| 10:16:08 | gibi | it is 13th so I'm not surprised | |
| 10:24:19 | xinranwang | gibi: Hi gibi, I have updated the smarnic spec, please review it when you got time :) | |
| 10:24:29 | gibi | xinranwang: thanks | |
| 10:24:49 | gibi | I will try to jump on reviews after I put out the fire on the gate | |
| 10:26:36 | xinranwang | gibi: Ah ok, it's friday evening in my timezone, I will check it next monday. Take your time. | |
| 10:26:40 | xinranwang | Thanks in advance | |
| 10:27:22 | gibi | xinranwang: there is a good chance that I will not reach your spec today. Have a nice weekend. | |
| 10:30:02 | openstack | Launchpad bug 1904181 in OpenStack Compute (nova) "nova-compute fails to start is cell conductor is not running." [High,Confirmed] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 10:30:02 | gibi | FYI another gate failure https://bugs.launchpad.net/nova/+bug/1904181 | |
| 10:32:04 | xinranwang | gibi: thanks :) | |
| 10:53:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Restore retrying the RCP connection to conductor https://review.opendev.org/762633 | |
| 10:54:58 | gibi | bauzas: when you are back there are two nova gate fixing patch. One from me ^^ and one from stephenfin https://review.opendev.org/#/c/762543 | |
| 10:55:04 | bauzas | I'm here | |
| 10:55:14 | bauzas | ack, looking then | |
| 10:55:37 | gibi | bauzas: look stephenfin's first I will try to add a test to mine | |
| 10:56:17 | gibi | and thanks | |
| 10:59:02 | bauzas | gibi: +W'd stephenfin's one | |
| 10:59:07 | gibi | awesome, thanks | |
| 10:59:28 | bauzas | gibi: for your own change, I have a concern, where are we looking at the conductor ? | |
| 11:00:11 | gibi | bauzas: https://github.com/openstack/nova/blob/eb279e9a5676f4142cce4700c3097ecc14161895/nova/service.py#L115 | |
| 11:00:27 | gibi | it is well hidden gem of the Service object | |
| 11:01:22 | bauzas | thanks | |
| 11:01:23 | bauzas | looking | |
| 11:01:50 | bauzas | a-ha, now I remember | |
| 11:02:13 | bauzas | https://github.com/openstack/nova/blob/eb279e9a5676f4142cce4700c3097ecc14161895/nova/service.py#L113 is for the computes | |
| 11:02:50 | gibi | yes | |
| 11:03:34 | bauzas | gibi: okay, then waiting for your test | |
| 11:03:43 | bauzas | gibi: just do a functional test honestly | |
| 11:03:44 | gibi | OK, working on it | |
| 11:03:53 | gibi | yes, it will be a functional one | |
| 11:03:55 | bauzas | would be simplier and better | |
| 11:04:07 | bauzas | cool | |
| 11:04:31 | bauzas | gibi: fwiw, I'll be lunching, but ping me when you're done and I'll look at it when I'm back | |
| 11:04:40 | gibi | sure. have a nice one! | |
| 11:24:43 | gibi | bauzas: bah, in the func test env the compute service does not need a running conductor to start up, I guess something is mocked in the indirection API | |
| 11:31:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Restore retrying the RCP connection to conductor https://review.opendev.org/762633 | |
| 11:32:43 | gibi | with a less good but still covering unit test ^^ | |
| 12:09:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Disable *all* virt services during negative tests https://review.opendev.org/762623 | |
| 12:15:48 | openstackgerrit | chengsheng proposed openstack/nova master: Add hypervisor CPU feature check during live migration https://review.opendev.org/762330 | |
| 12:25:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Doc that [database]connection is not for nova-compute https://review.opendev.org/762647 | |
| 12:35:20 | kashyap | Anyone else with CPU and live migration interest, have a look at this review, from above, chengsheng ... I'm not comfortable parsing /proc/cpuinfo straight-up (and a couple of other issues in there): https://review.opendev.org/#/c/762330/6 | |
| 12:35:26 | kashyap | Err, wrong link | |
| 12:35:36 | kashyap | Correct one: https://review.opendev.org/#/c/762330/ | |
| 12:52:57 | sean-k-mooney | we have said no to that in the past | |
| 12:53:08 | sean-k-mooney | and required it the info to be exposed in libvirt | |
| 13:06:13 | sean-k-mooney | kashyap: as written that patch will break upgrades and prevent migrating a vm to newwer hardware so its a nonstarter | |
| 13:07:13 | sean-k-mooney | its also not filtering out no virutaliseable feature that dont affact the cpu flags exposed to the vm | |
| 13:40:15 | openstackgerrit | Merged openstack/nova master: functional: Wait for revert resize to complete https://review.opendev.org/762543 | |
| 13:44:11 | f0o | I'm a bit confused... More and more linux guests cant reboot. their instances go from running to paused for some reason and need to be hard-reboot from horizon/api to be able to become alive again. Is a guest no longer allowed to reboot by itself? Am I missing a setting somewhere? (using kvm/qemu) | |
| 13:44:48 | sean-k-mooney | f0o: they are allowed too yes | |
| 13:45:11 | f0o | why are they ending up in "paused" then? | |
| 13:45:30 | sean-k-mooney | they would only be moved to paused if the state in the db for the vm was paused and you had the sync power state option set | |
| 13:45:50 | f0o | what would cause the db to think it's paused? | |
| 13:45:58 | sean-k-mooney | only an api request | |
| 13:46:05 | sean-k-mooney | did you check the instance event log | |
| 13:46:28 | f0o | the way I can reproduce it is: create instance (ubuntu lts for instance), log in and do sudo reboot, vm is now stuck in paused | |
| 13:46:46 | f0o | no api-calls made | |
| 13:47:12 | sean-k-mooney | im not directly aware of anythin that would cause that | |
| 13:47:24 | sean-k-mooney | i have not seen it on ussuri at least | |
| 13:47:29 | sean-k-mooney | what release are you using | |
| 13:47:41 | f0o | it becomes very annoying for OS that run updates as part of their first boot (like coreos/flatcar) then the vm never becomes alive... | |
| 13:47:44 | f0o | I think I'm still on stein | |
| 13:48:10 | kashyap | sean-k-mooney: Yeah; it's a non-starter for other reasons too | |
| 13:48:19 | kashyap | sean-k-mooney: I've just only gave a cursory look :-) Please comment there | |
| 13:49:01 | sean-k-mooney | kashyap: i did | |
| 13:49:07 | sean-k-mooney | i was summerising for you :) | |
| 13:49:45 | kashyap | Hehe; thanks | |
| 13:56:48 | f0o | how would I go about debugging this weird paused issue? | |
| 13:56:51 | f0o | where to start looking? | |
| 13:57:24 | sean-k-mooney | f0o: you should be looking at the nova-compute logs for the instance in question and seeing what actions cause the instance to be paused | |
| 13:57:50 | sean-k-mooney | e.g. did the compute agent recived an event form libvirt saying the guest is now paused | |
| 13:57:57 | sean-k-mooney | or did it call libvirt to pause it | |
| 13:58:09 | f0o | ok, will do | |
| 13:58:15 | f0o | thnkas :) | |
| 13:58:22 | f0o | thanks* | |
| 14:02:34 | f0o | I have a few entries that have VM Paused (Lifecycle Event) in them. just gonna spawn a new instance and reboot it to see all the logs it generates | |
| 14:19:02 | f0o | [instance: 4d131088-4ca6-47bd-ab5c-47b9e0a7c996] VM Paused (Lifecycle Event) & During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (3). Updating power_state in the DB to match the hypervisor. | |
| 14:19:13 | f0o | [instance: 4d131088-4ca6-47bd-ab5c-47b9e0a7c996] Instance is paused unexpectedly. Ignore. | |
| 14:19:28 | f0o | so those 3 lines happen when I issue reboot inside the instance | |
| 14:20:16 | f0o | and now horizon shows it as Active/Paused and there's no resume action. I need to issue Hard-Reboot to kick it back alive | |