| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-06 | |||
| 18:58:37 | gmann | bauzas: gibi, yes, after tempest, sdks broke it and with this it fixed and green https://review.opendev.org/c/openstack/nova/+/871798 | |
| 18:58:55 | gmann | it is good to recheck now | |
| 21:24:42 | tobias-urdin | found a little nasty spelling mistake in compute rpc api aliasing introduced here https://review.opendev.org/c/openstack/nova/+/858383/30/nova/compute/rpcapi.py#428 | |
| 21:24:49 | tobias-urdin | unless there is a patch pending for it | |
| 21:25:03 | tobias-urdin | alias for 6.2 rpc version says antilope and not antelope | |
| 21:26:40 | opendevreview | Tobias Urdin proposed openstack/nova master: Fix 6.2 compute RPC version alias https://review.opendev.org/c/openstack/nova/+/872804 | |
| 21:26:45 | tobias-urdin | ^ patch | |
| #openstack-nova - 2023-02-07 | |||
| 02:17:43 | opendevreview | Merged openstack/nova master: Protect against a deleted node id file https://review.opendev.org/c/openstack/nova/+/872204 | |
| 07:56:47 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 07:56:47 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 07:56:48 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | |
| 07:56:48 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 07:56:49 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870933 | |
| 07:56:49 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 07:56:50 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 07:56:51 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 07:56:51 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 07:56:53 | opendevreview | melanie witt proposed openstack/nova master: Add reset_encryption_fields() and save_all() to BlockDeviceMappingList https://review.opendev.org/c/openstack/nova/+/870938 | |
| 07:56:53 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 07:56:55 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 08:13:18 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 08:13:18 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 08:13:19 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | |
| 08:13:19 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 08:13:20 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870933 | |
| 08:13:20 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 08:13:21 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 08:13:21 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 08:13:22 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 08:13:23 | opendevreview | melanie witt proposed openstack/nova master: Add reset_encryption_fields() and save_all() to BlockDeviceMappingList https://review.opendev.org/c/openstack/nova/+/870938 | |
| 08:13:23 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 08:13:25 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 09:04:17 | bauzas | tobias-urdin: excellent catch on the RPC pin alias, many thanks | |
| 09:04:31 | bauzas | it would have broken seriously our users if we were having it released | |
| 09:27:36 | bauzas | yet another strange DB issue https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d00/868237/9/check/nova-tox-functional-py38/d00d1ff/testr_results.html | |
| 09:27:50 | bauzas | gibi: I'm writing the etherpad | |
| 09:28:03 | bauzas | and I'm reporting a new gate failure | |
| 09:29:15 | gibi | bauzas: the failure you linked is probably https://bugs.launchpad.net/nova/+bug/1946339 | |
| 09:29:51 | gibi | so no need for a new bug report | |
| 09:30:45 | bauzas | gibi: seems unrelated | |
| 09:30:56 | bauzas | I get a DB issue due to a missing table | |
| 09:31:09 | bauzas | oh | |
| 09:31:29 | gibi | both fails with | |
| 09:31:29 | gibi | sqlite3.OperationalError: no such table: instance_faults | |
| 09:31:41 | gibi | after a message timeout | |
| 09:31:51 | bauzas | yeah, not the same table but quite the same behaviour | |
| 09:32:03 | bauzas | well, the bug report title is too specific | |
| 09:33:06 | bauzas | gibi: agreed, seems the same | |
| 09:47:20 | bauzas | gibi: https://etherpad.opendev.org/p/nova-ci-failures | |
| 09:49:49 | bauzas | damn shit, opensearch urls aren't shareable | |
| 09:52:40 | gibi | bauzas: feel free to update the title of https://bugs.launchpad.net/nova/+bug/1946339 | |
| 09:52:58 | bauzas | gibi: I'm not good at naming | |
| 09:53:10 | bauzas | and I'm not sure of the rootcause | |
| 09:53:26 | bauzas | I guess this is due to long-running threads | |
| 09:53:40 | bauzas | so, due to the length it takes, then we delete the DB | |
| 09:54:06 | gibi | bauzas: changed the title then | |
| 09:54:08 | bauzas | and so, when the thread stops, then we try to lookup the table which no longer exists | |
| 09:54:22 | bauzas | amirite on the root cause ? | |
| 09:54:52 | gibi | bauzas: one of the root cause is described in the commit message here https://review.opendev.org/c/openstack/nova/+/814036 | |
| 09:56:24 | gibi | what I we need to figure out what is the new sequence of events that leads to a similar leak | |
| 09:56:25 | bauzas | yeah ok, so we agree | |
| 09:56:35 | bauzas | we have spawned greenlets | |
| 09:56:42 | bauzas | that finish after the db is cleaned up | |
| 09:56:59 | bauzas | we should hold the test until those greenlets finish | |
| 09:58:16 | gibi | 1) we need to figure out which eventlet leaks 2) then we need to figure out how that leaked eventlet affects the later test case, i.e. where it the global variable that supports the leak 3) then we can figure out how to fix it | |
| 09:59:12 | tobias-urdin | bauzas: no worries, funny coincidence that i was digging around in the RPC layer and saw it :) | |
| 09:59:53 | gibi | in the fixed case it was RPC handling eventlet was leaked that after 60 sec waked up do to timeout and used nova.rpc.get_versioned_notifier() to get a fresh notifier to the actually running test case | |
| 10:00:21 | opendevreview | Jorge San Emeterio proposed openstack/nova master: Dividing global privsep profile https://review.opendev.org/c/openstack/nova/+/871729 | |
| 10:00:51 | gibi | the fix there was to ignore notifications if it is comming from a test case that is different from the currently running one | |
| 10:03:53 | bauzas | gibi: grabbing a coffee, this is a hard issue to dig into | |
| 10:04:11 | gibi | bauzas: ack I will add a logsearch query for it as soon as I have one | |
| 10:04:27 | bauzas | gibi: thanks, I was just doing it but I'm not expert of the tool | |
| 10:04:51 | bauzas | I'm very disappointed we can't longer share our ex-logstack urls | |
| 10:04:54 | bauzas | logstash* | |
| 10:05:17 | gibi | the SQL cursor one https://bugs.launchpad.net/nova/+bug/2002782 is not that frequent, we hit it 9 times in 20 days so I would ignore it for now | |
| 10:05:19 | bauzas | not very handy for finding how much we're doomed | |
| 10:05:29 | bauzas | gibi: yup, I was about to tell you | |
| 10:05:47 | bauzas | gibi: that's why I wanted to add the opensearch one | |
| 10:09:18 | gibi | that 1525 hits in 7 days seem way to much for https://bugs.launchpad.net/nova/+bug/1946339 I suspect some false positives there | |
| 10:10:00 | bauzas | probably | |
| 10:11:29 | bauzas | gibi: fancy to share your logsearch config files ? | |
| 10:12:05 | gibi | https://github.com/gibizer/zuul-log-search-config | |
| 10:14:54 | gibi | so we have sort of false positives. For example this https://zuul.opendev.org/t/openstack/build/31e9de9e4a574df6a2f45546927954fe/log/job-output.txt#23436 partially reproduced https://bugs.launchpad.net/nova/+bug/1946339 but in this case the test case did not fail due the the missing DB table just the stack trace was logged. | |
| 10:15:27 | gibi | so we probably have a lot of hits that has the stack trace but no failed tests | |
| 10:16:04 | bauzas | gibi: I can amend the opensearch query to ensure the outcome is FAILURE | |
| 10:16:51 | gibi | yepp, this is an example of a passed tox run with the missing DB table test case https://zuul.opendev.org/t/openstack/build/f389ba6c4d9f4a04a5b6f09e253d864b/log/job-output.txt#23501 | |
| 10:17:05 | bauzas | 191 hits :) | |
| 10:17:10 | gibi | s/missing DB table test case/missing DB table stack trace/ | |
| 10:17:11 | bauzas | in the last 7 days | |
| 10:18:20 | bauzas | gibi: sorry for this dumb question but I wanna rush | |
| 10:18:34 | bauzas | gibi: how to use the configs from the other repo into the main one ? | |
| 10:18:43 | bauzas | just all the files ? | |
| 10:18:50 | bauzas | or just checkouting some of them ? | |
| 10:19:53 | gibi | bauzas: what I do is: 1) create a venv and install the tool with pip install git+http://github.com/gibizer/zuul-log-search 2) next to the .venv clone the config repo | |
| 10:20:01 | bauzas | did 1) | |
| 10:20:13 | bauzas | did 2) | |
| 10:20:31 | gibi | It uses .logsearch.conf.d/ in the current directory if exists. Otherwise, uses $XDG_CONFIG_HOME/logsearch/ if XDG_CONFIG_HOME is defined. Otherwise, uses ~/.config/logsearch/. | |
| 10:20:44 | bauzas | yeah so mv the whole dir ? | |
| 10:20:51 | bauzas | that was my question | |
| 10:21:20 | bauzas | I see a config subdir in the zuul-log-search | |