| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-08-23 | |||
| 10:21:22 | sean-k-mooney | it failed on teh vdpa patch | |
| 10:21:30 | sean-k-mooney | but i have seen it on several others recnetly | |
| 10:21:53 | sean-k-mooney | do you know where the new elastic search deployment is hosted | |
| 10:22:08 | sean-k-mooney | im kind of wonderign how often this happens | |
| 10:23:24 | gibi | interesting | |
| 10:23:29 | gibi | https://zuul.opendev.org/t/openstack/build/5e64ad2283a24653a424ad91d99a3004/log/compute1/logs/screen-n-cpu.txt#52040 | |
| 10:23:38 | gibi | libvirt.libvirtError: invalid argument: disk '/dev/sda' was not found in the domain config | |
| 10:23:44 | sean-k-mooney | ya | |
| 10:23:48 | sean-k-mooney | thats the same failure | |
| 10:24:03 | gibi | sean-k-mooney: https://opensearch.logs.openstack.org/_dashboards/app/login?nextUrl=%2F_dashboards%2Fapp%2Fdiscover%2F%3Fsecurity_tenant%3Dglobal this is the logstash | |
| 10:24:04 | sean-k-mooney | we dont guarentee the adress of th disk | |
| 10:24:26 | sean-k-mooney | ack i was trying http://logstash.openstack.org/ | |
| 10:24:40 | sean-k-mooney | which is obviously dead but maybe infra coudl add a redirect to the new location | |
| 10:25:52 | gibi | yah a redirect would be nice | |
| 10:27:23 | gibi | hm from the last 10 novan-next failure 9 was due to the above error | |
| 10:27:29 | sean-k-mooney | do you know the login info? | |
| 10:27:50 | sean-k-mooney | i think they sent one ot the mail list but i cant find it in my email | |
| 10:28:39 | sean-k-mooney | gibi: ya so we have a log message that says "the libvirt driver cannot honner the request path" or somethign like that when we attach so im supresed we care when detaching | |
| 10:28:40 | gibi | openstack openstack | |
| 10:29:13 | sean-k-mooney | thanks | |
| 10:30:13 | gibi | it is almost like it started failing on 08.19. | |
| 10:30:51 | gibi | and we merged https://review.opendev.org/c/openstack/nova/+/843700/ in 19th | |
| 10:31:44 | sean-k-mooney | yep | |
| 10:31:44 | gibi | so I think we enabled an unstable test | |
| 10:31:58 | gibi | I'm proposing a revert | |
| 10:32:02 | sean-k-mooney | i can revert that i guess | |
| 10:32:04 | sean-k-mooney | ah go for it | |
| 10:33:24 | sean-k-mooney | so its calling | |
| 10:33:25 | sean-k-mooney | def attach_detach_volume(self, server, volume): | |
| 10:33:27 | sean-k-mooney | attached_volume = self.nova_volume_attach(server, volume) | |
| 10:33:29 | sean-k-mooney | self.nova_volume_detach(server, attached_volume) | |
| 10:33:43 | sean-k-mooney | if that is not waiting for sshable then that is l ikely the issue | |
| 10:34:03 | sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/836065/4/tempest/scenario/test_encrypted_cinder_volumes.py | |
| 10:34:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Revert "Test attached volume extend actions in the nova-next job" https://review.opendev.org/c/openstack/nova/+/854132 | |
| 10:34:57 | sean-k-mooney | i wonder is this another case of that type of error | |
| 10:35:01 | gibi | yeah that can be one issue yes | |
| 10:35:13 | sean-k-mooney | we might have removed it form teh persitent tomain but not the active domain | |
| 10:35:21 | sean-k-mooney | and fail with not found on a retry or something | |
| 10:35:37 | sean-k-mooney | i wonder if the logs shows anything like that | |
| 10:36:20 | sean-k-mooney | hum no its the resize that is failing | |
| 10:36:33 | sean-k-mooney | but perhaps the attach has not completed | |
| 10:38:09 | gibi | I'm table this for now. we can merge the revert or somebody can propose a fix to the tempest test but I let that be done by others :) | |
| 10:38:31 | sean-k-mooney | sure :) | |
| 10:41:16 | gibi | (we actually did a naked recheck that showed this error to merge the original patche enabling this test case) | |
| 10:42:33 | sean-k-mooney | thats unfortunet | |
| 10:43:43 | opendevreview | Merged openstack/nova master: Alphabetizes objects https://review.opendev.org/c/openstack/nova/+/853986 | |
| 10:44:48 | gibi | that shows why we are against naked rechecks :D | |
| 10:46:21 | sean-k-mooney | i left a comment on the tempest chagne https://review.opendev.org/c/openstack/tempest/+/836065/4#message-a8c9c9794c75555728517230fcab3b7beb8427de | |
| 10:46:26 | gibi | thanks | |
| 10:48:37 | sean-k-mooney | is logsearch a cli for logstash/opensearch ? | |
| 10:49:10 | sean-k-mooney | that looks like its quite nice to use vs the ui | |
| 10:49:26 | sean-k-mooney | the ui is not bad bu the cli output is very short and too the point | |
| 10:50:04 | gibi | nope it is my own tool grepping logs locally | |
| 10:50:22 | gibi | https://github.com/gibizer/zuul-log-search | |
| 10:50:37 | sean-k-mooney | oh ok | |
| 10:50:51 | sean-k-mooney | so its pulling the logs locally | |
| 10:50:55 | gibi | that is one of my hack and hustle target | |
| 10:51:25 | gibi | yes it pulls the logs locally based on the zuul query you make | |
| 10:51:30 | gibi | caches it locally | |
| 10:51:34 | gibi | and can grep in it | |
| 10:51:45 | gibi | also you can define bug signatures | |
| 10:51:45 | sean-k-mooney | ah nice | |
| 10:51:57 | gibi | and it can try to classify failures by stored signatures | |
| 10:52:25 | gibi | it was my answer to shutting down the old logsearch instance | |
| 10:52:38 | sean-k-mooney | ya | |
| 10:52:39 | gibi | *logstash | |
| 10:52:46 | sean-k-mooney | only downside is disk space really | |
| 10:53:00 | sean-k-mooney | but that is relitivly cheap | |
| 10:53:37 | sean-k-mooney | when you do --file compute1/logs/screen-n-cpu.txt --file controller/logs/screen-n-cpu.txt | |
| 10:53:43 | sean-k-mooney | is that going to just pull those files | |
| 10:53:54 | gibi | yes | |
| 10:54:06 | gibi | by default it only pulls the job-output.txt | |
| 10:54:16 | gibi | to limit disk space usage :) | |
| 10:54:16 | sean-k-mooney | ok so as logn as you knwo what file you want it wont take that mauch space at all | |
| 10:54:39 | sean-k-mooney | ill have to find some time to set it up and try it out | |
| 10:55:04 | gibi | my query for the above failure in nova-next in the last 25 days takes ~1G local disk space | |
| 10:56:52 | gibi | the compute service logs are pretty big | |
| 10:57:13 | gibi | 53 compute service logs == 1G | |
| 10:57:38 | sean-k-mooney | ya i guess they are in debug | |
| 10:57:38 | gibi | but I can put the cache dir to a compressed volume and that can safe a lot of space | |
| 10:57:47 | gibi | yes fortunately gate use debug :D | |
| 10:57:57 | gibi | s/safe/save/ | |
| 10:58:28 | sean-k-mooney | i rememebr the intel nfv ci used to use a 300G log share that we sotred gziped logs too | |
| 10:58:38 | sean-k-mooney | that was enough for about 60 days fo ci logs | |
| 10:58:56 | sean-k-mooney | once you compressed everything | |
| 10:59:25 | gibi | yes these logs are highl compressable | |
| 11:00:09 | sean-k-mooney | like db dumps. the db dump i was working with for the downstream escalation was about 900MB raw or about 60 mb compressed | |
| 11:44:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: Retry /reshape at provider generation conflict https://review.opendev.org/c/openstack/nova/+/851358 | |
| 11:44:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: Heal missing simple PCI allocation in the resource tracker https://review.opendev.org/c/openstack/nova/+/851359 | |
| 11:44:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Heal PCI allocation during resize https://review.opendev.org/c/openstack/nova/+/852396 | |
| 11:44:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Handle PCI dev reconf with allocations https://review.opendev.org/c/openstack/nova/+/852397 | |
| 11:44:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement https://review.opendev.org/c/openstack/nova/+/850468 | |
| 11:44:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: Generate request_id for Flavor based InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/853835 | |
| 11:44:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Request filter for PCI in placement https://review.opendev.org/c/openstack/nova/+/852771 | |
| 11:44:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias https://review.opendev.org/c/openstack/nova/+/853316 | |
| 11:44:18 | opendevreview | Balazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs https://review.opendev.org/c/openstack/nova/+/854118 | |
| 11:44:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters https://review.opendev.org/c/openstack/nova/+/854119 | |
| 11:44:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation https://review.opendev.org/c/openstack/nova/+/854120 | |
| 11:44:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/854121 | |
| 11:44:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling https://review.opendev.org/c/openstack/nova/+/854122 | |