| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-05 | |||
| 16:25:35 | sean-k-mooney | u->v would (posssible) be intermitent | |
| 16:25:42 | sean-k-mooney | so yes | |
| 16:25:54 | dansmith | right, very many fewer things running that configuration | |
| 16:26:44 | sean-k-mooney | i have just been seeign some intermitent grendade job failure before the ptg so was wondering if this was the issue or if there are others | |
| 16:27:06 | sean-k-mooney | most of the issue seam to be realted to volumes however rather then rebuild | |
| 16:27:06 | dansmith | could be.. so many CI fails lately, I expect people are doing a lot of recheck grinding | |
| 16:32:17 | gibi | this is a recent grenade multinode run from stable/victoria https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/index.html | |
| 16:32:27 | gibi | I see two test reports | |
| 16:32:28 | gibi | https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/old/testr_results.html | |
| 16:32:34 | gibi | and | |
| 16:32:35 | gibi | https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/testr_results.html | |
| 16:32:50 | gibi | is there a 3rd report somewhere in the tree? | |
| 16:33:17 | sean-k-mooney | nope | |
| 16:33:22 | sean-k-mooney | just those two | |
| 16:33:35 | sean-k-mooney | so we are not running rebuild in the grenade job | |
| 16:34:04 | sean-k-mooney | i tought we were but i guess not | |
| 16:35:19 | sean-k-mooney | the grenade failures i was seeing were likely something else so | |
| 16:36:16 | sean-k-mooney | its been like 2 weeks so all that is left in my brain on the topic is "i have seen more grenade failures lately then i normally do" | |
| 16:48:04 | openstackgerrit | Sylvain Bauza proposed openstack/nova stable/victoria: Add a regression test for 5.12 compute API issue https://review.opendev.org/761638 | |
| 16:48:05 | openstackgerrit | Sylvain Bauza proposed openstack/nova stable/victoria: Fix the compute RPC 5.12 issue https://review.opendev.org/761639 | |
| 16:50:27 | bauzas | elod: stable changes are up there ^ | |
| 16:51:01 | bauzas | hopefully master changes will be merged tonight so we could move on tomorrow | |
| 16:51:09 | bauzas | and ideally release subsequently | |
| 16:51:20 | bauzas | (release stable/victoria) | |
| 16:51:58 | elod | bauzas: thx, looking :) | |
| 16:52:03 | bauzas | elod: don't | |
| 16:52:11 | bauzas | the master change isn't merged yet so I -2 it | |
| 16:52:28 | elod | don't worry I'll wait with the +2 until master is merged ;) | |
| 16:53:23 | elod | (if I don't find any mistake with the backport, ofc) | |
| 16:53:27 | elod | :] | |
| 17:00:12 | openstackgerrit | Merged openstack/nova master: Add a regression test for 5.12 compute API issue https://review.opendev.org/761457 | |
| 17:03:40 | bauzas | elod: heh ^ | |
| 17:11:28 | elod | bauzas: ok, so the regression test part is ready and looks OK. +2'd | |
| 17:11:37 | bauzas | <3 | |
| 17:11:38 | elod | one more to go :) | |
| 17:16:31 | elod | the backport of the fix also looks good to me and the fix is on the gate in master, so we just have to wait. | |
| 17:17:38 | elod | i'll prepare a release patch tomorrow for victoria if the fix gets merged | |
| 21:55:30 | openstackgerrit | Merged openstack/nova master: Fix the compute RPC 5.12 issue https://review.opendev.org/761458 | |
| 23:20:54 | openstackgerrit | Merged openstack/nova stable/victoria: Add a regression test for 5.12 compute API issue https://review.opendev.org/761638 | |
| #openstack-nova - 2020-11-06 | |||
| 05:23:47 | openstackgerrit | Merged openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | |
| 05:44:40 | openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/victoria: Update pci stat pools based on PCI device changes https://review.opendev.org/761700 | |
| 06:23:28 | openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/ussuri: Update pci stat pools based on PCI device changes https://review.opendev.org/761701 | |
| 07:47:55 | openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | |
| 09:54:36 | gibi | bauzas: you can remove your -2 on https://review.opendev.org/#/c/761639/ the fix merged to master | |
| 09:59:41 | bauzas | gibi: sure okay | |
| 10:00:47 | bauzas | done | |
| 10:07:15 | gibi | bauzas: https://zuul.opendev.org/t/openstack/build/aefd1852c72345148e6f62b65b6b49fd/log/logs/screen-n-cpu.txt?severity=4#17226 | |
| 10:07:37 | bauzas | gibi: I was on it :) | |
| 10:07:52 | gibi | https://bugs.launchpad.net/nova/+bug/1901739 | |
| 10:07:52 | openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,Confirmed] | |
| 10:07:52 | bauzas | but I was looking at e-r to see whether we also had a bug | |
| 10:08:00 | bauzas | ack, thanks | |
| 10:08:42 | bauzas | ok, I declared it | |
| 10:08:56 | bauzas | http://status.openstack.org/elastic-recheck/#1901739 | |
| 10:09:36 | gibi | yep | |
| 10:15:16 | lyarwood | `Delay in Elastic Search: Indexing behind by 100 hours` | |
| 10:15:36 | lyarwood | I know melwitt was looking into this a while ago with infra but I think we need to flag that again | |
| 10:15:56 | lyarwood | I hit 1901739 yesterday FWIW, might bump the bug a little | |
| 10:16:27 | gibi | lyarwood: there is a bug that produces 100MBs of libvirt log maybe that causes logstash to get swamped https://bugs.launchpad.net/nova/+bug/1902276 | |
| 10:16:27 | openstack | Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New] | |
| 10:16:41 | lyarwood | kashyap: https://bugs.launchpad.net/nova/+bug/1901739 - I know you've been under water with other things but did you find anything more about https://bugs.launchpad.net/nova/+bug/1901739 | |
| 10:16:42 | openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,Confirmed] | |
| 10:17:00 | lyarwood | gibi: yeah true are we hitting that often? I've not seen it outside of the example job in the bug tbh | |
| 10:17:11 | gibi | let me look | |
| 10:17:18 | kashyap | lyarwood: Hi, yeah; I'm under a "fun" deadline to do an LWN write-up of a couple of KVM Forum talks (including one involving Block Layer) | |
| 10:17:21 | kashyap | lyarwood: Looking... | |
| 10:18:20 | lyarwood | kashyap: ack np | |
| 10:18:27 | lyarwood | kashyap: I can try to push it forward today | |
| 10:18:36 | kashyap | lyarwood: Let me ping Peter Krempa once more to see if he can have a look at - https://kashyapc.fedorapeople.org/LP-1901739/ | |
| 10:18:47 | lyarwood | kashyap: awesome thanks | |
| 10:19:17 | bauzas | lyarwood: the e-r indexing allows to set the hits automatically | |
| 10:19:21 | kashyap | It's the interaction of libvirt's interface with QEMU's NBD | |
| 10:19:32 | bauzas | lyarwood: but when you recheck with a bug number, it does increase the counter too | |
| 10:19:37 | bauzas | just manually | |
| 10:19:51 | lyarwood | bauzas: right my point is that we are likely hitting this more than the counter shows anyway | |
| 10:20:10 | bauzas | correct, but fwiw like I said, i incremented the counter at least this morning | |
| 10:20:38 | bauzas | but we can look at logstash to see the occurences if you do feel we could have more | |
| 10:20:56 | bauzas | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message%3A%5C%22Live%20Migration%20failure%3A%20internal%20error%3A%20missing%20block%20job%20data%20for%20disk%20%27vda%27%3A%20libvirt.libvirtError%3A%20internal%20error%3A%20missing%20block%20job%20data%20for%20disk%20%27vda%27%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A1&from=864000s | |
| 10:21:37 | lyarwood | right but if the logs aren't indexed they don't show up in logstash right? | |
| 10:21:58 | lyarwood | so again to my point, due to the delay we are likely missing hits of this | |
| 10:22:09 | lyarwood | that's all I was suggesting | |
| 10:22:10 | bauzas | oh shit, you're right, my bad | |
| 10:22:55 | bauzas | I was just thinking e-r indexing was broken, not the whole logstash munging | |
| 10:23:12 | bauzas | (because we had specific e-r issues in the past) | |
| 10:29:28 | kashyap | lyarwood: Peter points out the version 5.4.0. But weren't we supposed to be using libvirt-6.0.0, from Focal? | |
| 10:38:30 | lyarwood | kashyap: huh | |
| 10:38:57 | gibi | lyarwood: regarding https://bugs.launchpad.net/nova/+bug/1902276 and the big libvirtd.log. I don't found any meaningful log I can use as a signature for this bug. so i don't know how frequently octavia hit this. Either my logstash knowledge is not enough or logstash does not index libvirtd.log | |
| 10:38:58 | openstack | Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New] | |
| 10:39:55 | lyarwood | gibi: kk, I'm going to finish looking at 1901739 then I'll switch back to that | |
| 10:40:10 | kashyap | lyarwood: What do you mean? Yeah, that was also sorta my reaction ... 5.4 is only about a year-ish old | |
| 10:40:22 | gibi | now I go and ask infra about how we can help with the indexing delay of logstash | |
| 10:42:31 | lyarwood | kashyap: gah so our nova-live-migration job is still bionic | |
| 10:42:38 | lyarwood | kashyap: not sure how I missed this | |
| 10:42:53 | kashyap | lyarwood: Either way; Peter says 6.0.0, which uses the -blockdev infra should fix it | |
| 10:43:11 | lyarwood | kashyap: kk well the new zuulv3 live migration job is using that and passing | |
| 10:43:12 | kashyap | lyarwood: No problem; you do a lot of CI job work too; easy to miss | |
| 10:43:23 | kashyap | Ah-ha; good to know! Got a link? | |
| 10:43:57 | lyarwood | kashyap: is the fix generic or can he point to something specific just so we can document why we need to move to focal? | |
| 10:44:10 | lyarwood | kashyap: I'm thinking of marking the current job as non-voting while we move | |
| 10:46:08 | kashyap | lyarwood: Yeah, that makes sense; anything that gets us away from the "legacy" (-drive) | |