| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-05 | |||
| 16:17:22 | gibi | really? I only found a smoke result | |
| 16:17:26 | sean-k-mooney | im thinging that if the vm that landed on the unupgraded node was rebuilt it would fail | |
| 16:17:27 | gibi | + the live migration | |
| 16:17:34 | sean-k-mooney | maybe im wrong | |
| 16:20:21 | sean-k-mooney | https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-grenade-multinode/run.yaml#L40 | |
| 16:20:37 | sean-k-mooney | its running the compute api tests and senario tests | |
| 16:21:09 | sean-k-mooney | oh just the smoke subset of those? | |
| 16:21:24 | dansmith | just smoke before, not sure about full after t hough | |
| 16:22:41 | sean-k-mooney | if its runnign rebuild after then if it booted on the upgraded node we would get teh type error | |
| 16:22:57 | sean-k-mooney | if it booted on the un upgraded node it would have rebuilt fine | |
| 16:23:21 | sean-k-mooney | which would have made the test failure intermitent | |
| 16:23:23 | dansmith | I dunno why you say that, | |
| 16:23:29 | dansmith | the control plane would be upgraded, | |
| 16:23:55 | dansmith | oh you mean because the pin is set to auto and the presence of an old compute would keep it pinned I guess? | |
| 16:24:02 | sean-k-mooney | yes | |
| 16:24:20 | sean-k-mooney | it would be pinned but the old nova code would not expect the parmater and the new code would | |
| 16:24:27 | dansmith | that only works for U->V jobs, since V supported it, it'll be using the new version | |
| 16:24:40 | dansmith | you need to be looking at U->V grenade multinode jobs I'd expect right? | |
| 16:25:06 | dansmith | also, as bad as the gate has been lately, it wouldn't surprise me if people have just been rechecking past that occasional fail | |
| 16:25:20 | sean-k-mooney | well v->master woudl work since they woudl both use 5.12+ | |
| 16:25:29 | dansmith | that's my point | |
| 16:25:35 | sean-k-mooney | u->v would (posssible) be intermitent | |
| 16:25:42 | sean-k-mooney | so yes | |
| 16:25:54 | dansmith | right, very many fewer things running that configuration | |
| 16:26:44 | sean-k-mooney | i have just been seeign some intermitent grendade job failure before the ptg so was wondering if this was the issue or if there are others | |
| 16:27:06 | sean-k-mooney | most of the issue seam to be realted to volumes however rather then rebuild | |
| 16:27:06 | dansmith | could be.. so many CI fails lately, I expect people are doing a lot of recheck grinding | |
| 16:32:17 | gibi | this is a recent grenade multinode run from stable/victoria https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/index.html | |
| 16:32:27 | gibi | I see two test reports | |
| 16:32:28 | gibi | https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/old/testr_results.html | |
| 16:32:34 | gibi | and | |
| 16:32:35 | gibi | https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/testr_results.html | |
| 16:32:50 | gibi | is there a 3rd report somewhere in the tree? | |
| 16:33:17 | sean-k-mooney | nope | |
| 16:33:22 | sean-k-mooney | just those two | |
| 16:33:35 | sean-k-mooney | so we are not running rebuild in the grenade job | |
| 16:34:04 | sean-k-mooney | i tought we were but i guess not | |
| 16:35:19 | sean-k-mooney | the grenade failures i was seeing were likely something else so | |
| 16:36:16 | sean-k-mooney | its been like 2 weeks so all that is left in my brain on the topic is "i have seen more grenade failures lately then i normally do" | |
| 16:48:04 | openstackgerrit | Sylvain Bauza proposed openstack/nova stable/victoria: Add a regression test for 5.12 compute API issue https://review.opendev.org/761638 | |
| 16:48:05 | openstackgerrit | Sylvain Bauza proposed openstack/nova stable/victoria: Fix the compute RPC 5.12 issue https://review.opendev.org/761639 | |
| 16:50:27 | bauzas | elod: stable changes are up there ^ | |
| 16:51:01 | bauzas | hopefully master changes will be merged tonight so we could move on tomorrow | |
| 16:51:09 | bauzas | and ideally release subsequently | |
| 16:51:20 | bauzas | (release stable/victoria) | |
| 16:51:58 | elod | bauzas: thx, looking :) | |
| 16:52:03 | bauzas | elod: don't | |
| 16:52:11 | bauzas | the master change isn't merged yet so I -2 it | |
| 16:52:28 | elod | don't worry I'll wait with the +2 until master is merged ;) | |
| 16:53:23 | elod | (if I don't find any mistake with the backport, ofc) | |
| 16:53:27 | elod | :] | |
| 17:00:12 | openstackgerrit | Merged openstack/nova master: Add a regression test for 5.12 compute API issue https://review.opendev.org/761457 | |
| 17:03:40 | bauzas | elod: heh ^ | |
| 17:11:28 | elod | bauzas: ok, so the regression test part is ready and looks OK. +2'd | |
| 17:11:37 | bauzas | <3 | |
| 17:11:38 | elod | one more to go :) | |
| 17:16:31 | elod | the backport of the fix also looks good to me and the fix is on the gate in master, so we just have to wait. | |
| 17:17:38 | elod | i'll prepare a release patch tomorrow for victoria if the fix gets merged | |
| 21:55:30 | openstackgerrit | Merged openstack/nova master: Fix the compute RPC 5.12 issue https://review.opendev.org/761458 | |
| 23:20:54 | openstackgerrit | Merged openstack/nova stable/victoria: Add a regression test for 5.12 compute API issue https://review.opendev.org/761638 | |
| #openstack-nova - 2020-11-06 | |||
| 05:23:47 | openstackgerrit | Merged openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | |
| 05:44:40 | openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/victoria: Update pci stat pools based on PCI device changes https://review.opendev.org/761700 | |
| 06:23:28 | openstackgerrit | Hemanth Nakkina proposed openstack/nova stable/ussuri: Update pci stat pools based on PCI device changes https://review.opendev.org/761701 | |
| 07:47:55 | openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | |
| 09:54:36 | gibi | bauzas: you can remove your -2 on https://review.opendev.org/#/c/761639/ the fix merged to master | |
| 09:59:41 | bauzas | gibi: sure okay | |
| 10:00:47 | bauzas | done | |
| 10:07:15 | gibi | bauzas: https://zuul.opendev.org/t/openstack/build/aefd1852c72345148e6f62b65b6b49fd/log/logs/screen-n-cpu.txt?severity=4#17226 | |
| 10:07:37 | bauzas | gibi: I was on it :) | |
| 10:07:52 | gibi | https://bugs.launchpad.net/nova/+bug/1901739 | |
| 10:07:52 | openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,Confirmed] | |
| 10:07:52 | bauzas | but I was looking at e-r to see whether we also had a bug | |
| 10:08:00 | bauzas | ack, thanks | |
| 10:08:42 | bauzas | ok, I declared it | |
| 10:08:56 | bauzas | http://status.openstack.org/elastic-recheck/#1901739 | |
| 10:09:36 | gibi | yep | |
| 10:15:16 | lyarwood | `Delay in Elastic Search: Indexing behind by 100 hours` | |
| 10:15:36 | lyarwood | I know melwitt was looking into this a while ago with infra but I think we need to flag that again | |
| 10:15:56 | lyarwood | I hit 1901739 yesterday FWIW, might bump the bug a little | |
| 10:16:27 | gibi | lyarwood: there is a bug that produces 100MBs of libvirt log maybe that causes logstash to get swamped https://bugs.launchpad.net/nova/+bug/1902276 | |
| 10:16:27 | openstack | Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New] | |
| 10:16:41 | lyarwood | kashyap: https://bugs.launchpad.net/nova/+bug/1901739 - I know you've been under water with other things but did you find anything more about https://bugs.launchpad.net/nova/+bug/1901739 | |
| 10:16:42 | openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,Confirmed] | |
| 10:17:00 | lyarwood | gibi: yeah true are we hitting that often? I've not seen it outside of the example job in the bug tbh | |
| 10:17:11 | gibi | let me look | |
| 10:17:18 | kashyap | lyarwood: Hi, yeah; I'm under a "fun" deadline to do an LWN write-up of a couple of KVM Forum talks (including one involving Block Layer) | |
| 10:17:21 | kashyap | lyarwood: Looking... | |
| 10:18:20 | lyarwood | kashyap: ack np | |
| 10:18:27 | lyarwood | kashyap: I can try to push it forward today | |
| 10:18:36 | kashyap | lyarwood: Let me ping Peter Krempa once more to see if he can have a look at - https://kashyapc.fedorapeople.org/LP-1901739/ | |
| 10:18:47 | lyarwood | kashyap: awesome thanks | |
| 10:19:17 | bauzas | lyarwood: the e-r indexing allows to set the hits automatically | |
| 10:19:21 | kashyap | It's the interaction of libvirt's interface with QEMU's NBD | |
| 10:19:32 | bauzas | lyarwood: but when you recheck with a bug number, it does increase the counter too | |
| 10:19:37 | bauzas | just manually | |
| 10:19:51 | lyarwood | bauzas: right my point is that we are likely hitting this more than the counter shows anyway | |
| 10:20:10 | bauzas | correct, but fwiw like I said, i incremented the counter at least this morning | |
| 10:20:38 | bauzas | but we can look at logstash to see the occurences if you do feel we could have more | |
| 10:20:56 | bauzas | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message%3A%5C%22Live%20Migration%20failure%3A%20internal%20error%3A%20missing%20block%20job%20data%20for%20disk%20%27vda%27%3A%20libvirt.libvirtError%3A%20internal%20error%3A%20missing%20block%20job%20data%20for%20disk%20%27vda%27%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A1&from=864000s | |
| 10:21:37 | lyarwood | right but if the logs aren't indexed they don't show up in logstash right? | |