| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-08-23 | |||
| 10:27:23 | gibi | hm from the last 10 novan-next failure 9 was due to the above error | |
| 10:27:29 | sean-k-mooney | do you know the login info? | |
| 10:27:50 | sean-k-mooney | i think they sent one ot the mail list but i cant find it in my email | |
| 10:28:39 | sean-k-mooney | gibi: ya so we have a log message that says "the libvirt driver cannot honner the request path" or somethign like that when we attach so im supresed we care when detaching | |
| 10:28:40 | gibi | openstack openstack | |
| 10:29:13 | sean-k-mooney | thanks | |
| 10:30:13 | gibi | it is almost like it started failing on 08.19. | |
| 10:30:51 | gibi | and we merged https://review.opendev.org/c/openstack/nova/+/843700/ in 19th | |
| 10:31:44 | gibi | so I think we enabled an unstable test | |
| 10:31:44 | sean-k-mooney | yep | |
| 10:31:58 | gibi | I'm proposing a revert | |
| 10:32:02 | sean-k-mooney | i can revert that i guess | |
| 10:32:04 | sean-k-mooney | ah go for it | |
| 10:33:24 | sean-k-mooney | so its calling | |
| 10:33:25 | sean-k-mooney | def attach_detach_volume(self, server, volume): | |
| 10:33:27 | sean-k-mooney | attached_volume = self.nova_volume_attach(server, volume) | |
| 10:33:29 | sean-k-mooney | self.nova_volume_detach(server, attached_volume) | |
| 10:33:43 | sean-k-mooney | if that is not waiting for sshable then that is l ikely the issue | |
| 10:34:03 | sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/836065/4/tempest/scenario/test_encrypted_cinder_volumes.py | |
| 10:34:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Revert "Test attached volume extend actions in the nova-next job" https://review.opendev.org/c/openstack/nova/+/854132 | |
| 10:34:57 | sean-k-mooney | i wonder is this another case of that type of error | |
| 10:35:01 | gibi | yeah that can be one issue yes | |
| 10:35:13 | sean-k-mooney | we might have removed it form teh persitent tomain but not the active domain | |
| 10:35:21 | sean-k-mooney | and fail with not found on a retry or something | |
| 10:35:37 | sean-k-mooney | i wonder if the logs shows anything like that | |
| 10:36:20 | sean-k-mooney | hum no its the resize that is failing | |
| 10:36:33 | sean-k-mooney | but perhaps the attach has not completed | |
| 10:38:09 | gibi | I'm table this for now. we can merge the revert or somebody can propose a fix to the tempest test but I let that be done by others :) | |
| 10:38:31 | sean-k-mooney | sure :) | |
| 10:41:16 | gibi | (we actually did a naked recheck that showed this error to merge the original patche enabling this test case) | |
| 10:42:33 | sean-k-mooney | thats unfortunet | |
| 10:43:43 | opendevreview | Merged openstack/nova master: Alphabetizes objects https://review.opendev.org/c/openstack/nova/+/853986 | |
| 10:44:48 | gibi | that shows why we are against naked rechecks :D | |
| 10:46:21 | sean-k-mooney | i left a comment on the tempest chagne https://review.opendev.org/c/openstack/tempest/+/836065/4#message-a8c9c9794c75555728517230fcab3b7beb8427de | |
| 10:46:26 | gibi | thanks | |
| 10:48:37 | sean-k-mooney | is logsearch a cli for logstash/opensearch ? | |
| 10:49:10 | sean-k-mooney | that looks like its quite nice to use vs the ui | |
| 10:49:26 | sean-k-mooney | the ui is not bad bu the cli output is very short and too the point | |
| 10:50:04 | gibi | nope it is my own tool grepping logs locally | |
| 10:50:22 | gibi | https://github.com/gibizer/zuul-log-search | |
| 10:50:37 | sean-k-mooney | oh ok | |
| 10:50:51 | sean-k-mooney | so its pulling the logs locally | |
| 10:50:55 | gibi | that is one of my hack and hustle target | |
| 10:51:25 | gibi | yes it pulls the logs locally based on the zuul query you make | |
| 10:51:30 | gibi | caches it locally | |
| 10:51:34 | gibi | and can grep in it | |
| 10:51:45 | sean-k-mooney | ah nice | |
| 10:51:45 | gibi | also you can define bug signatures | |
| 10:51:57 | gibi | and it can try to classify failures by stored signatures | |
| 10:52:25 | gibi | it was my answer to shutting down the old logsearch instance | |
| 10:52:38 | sean-k-mooney | ya | |
| 10:52:39 | gibi | *logstash | |
| 10:52:46 | sean-k-mooney | only downside is disk space really | |
| 10:53:00 | sean-k-mooney | but that is relitivly cheap | |
| 10:53:37 | sean-k-mooney | when you do --file compute1/logs/screen-n-cpu.txt --file controller/logs/screen-n-cpu.txt | |
| 10:53:43 | sean-k-mooney | is that going to just pull those files | |
| 10:53:54 | gibi | yes | |
| 10:54:06 | gibi | by default it only pulls the job-output.txt | |
| 10:54:16 | sean-k-mooney | ok so as logn as you knwo what file you want it wont take that mauch space at all | |
| 10:54:16 | gibi | to limit disk space usage :) | |
| 10:54:39 | sean-k-mooney | ill have to find some time to set it up and try it out | |
| 10:55:04 | gibi | my query for the above failure in nova-next in the last 25 days takes ~1G local disk space | |
| 10:56:52 | gibi | the compute service logs are pretty big | |
| 10:57:13 | gibi | 53 compute service logs == 1G | |
| 10:57:38 | gibi | but I can put the cache dir to a compressed volume and that can safe a lot of space | |
| 10:57:38 | sean-k-mooney | ya i guess they are in debug | |
| 10:57:47 | gibi | yes fortunately gate use debug :D | |
| 10:57:57 | gibi | s/safe/save/ | |
| 10:58:28 | sean-k-mooney | i rememebr the intel nfv ci used to use a 300G log share that we sotred gziped logs too | |
| 10:58:38 | sean-k-mooney | that was enough for about 60 days fo ci logs | |
| 10:58:56 | sean-k-mooney | once you compressed everything | |
| 10:59:25 | gibi | yes these logs are highl compressable | |
| 11:00:09 | sean-k-mooney | like db dumps. the db dump i was working with for the downstream escalation was about 900MB raw or about 60 mb compressed | |
| 11:44:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: Heal missing simple PCI allocation in the resource tracker https://review.opendev.org/c/openstack/nova/+/851359 | |
| 11:44:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: Retry /reshape at provider generation conflict https://review.opendev.org/c/openstack/nova/+/851358 | |
| 11:44:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Handle PCI dev reconf with allocations https://review.opendev.org/c/openstack/nova/+/852397 | |
| 11:44:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Heal PCI allocation during resize https://review.opendev.org/c/openstack/nova/+/852396 | |
| 11:44:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: Generate request_id for Flavor based InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/853835 | |
| 11:44:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement https://review.opendev.org/c/openstack/nova/+/850468 | |
| 11:44:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias https://review.opendev.org/c/openstack/nova/+/853316 | |
| 11:44:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Request filter for PCI in placement https://review.opendev.org/c/openstack/nova/+/852771 | |
| 11:44:18 | opendevreview | Balazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs https://review.opendev.org/c/openstack/nova/+/854118 | |
| 11:44:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation https://review.opendev.org/c/openstack/nova/+/854120 | |
| 11:44:19 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters https://review.opendev.org/c/openstack/nova/+/854119 | |
| 11:44:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling https://review.opendev.org/c/openstack/nova/+/854122 | |
| 11:44:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/854121 | |
| 11:46:32 | gibi | sean-k-mooney, stephenfin: solved the todo in https://review.opendev.org/c/openstack/nova/+/851358 so now the pci series is ready for review up until https://review.opendev.org/c/openstack/nova/+/853835/ the rest of the series only ready for high level architectural review, as those patches does not have proper test coverage. The last patch of the series adds some functional tests that proves that | |
| 11:46:38 | gibi | the scheduling logic is sane though | |
| 11:48:31 | sean-k-mooney | ack | |
| 11:48:49 | sean-k-mooney | i was looking at some other stuff but ill start on it soon | |
| 11:48:57 | gibi | ack | |
| 11:49:48 | gibi | stephenfin, sean-k-mooney: I'm really thankful for all these reviews. I know that this is a lot of code and logic | |
| 11:50:02 | gibi | ... and now a quick lunch | |
| 13:02:07 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds check for VM snapshot fail while quiesce https://review.opendev.org/c/openstack/nova/+/852171 | |
| 13:12:23 | opendevreview | Elod Illes proposed openstack/nova stable/stein: DNM: check CI health https://review.opendev.org/c/openstack/nova/+/854184 | |
| 14:40:44 | sean-k-mooney | stephenfin: o/ | |
| 14:41:34 | sean-k-mooney | stephenfin: just an fyi i figured out the issue with the live migration test and that is now passing | |
| 14:41:42 | sean-k-mooney | melwitt: are you around | |
| 14:41:57 | sean-k-mooney | melwitt: if so can you take a look at this revert https://review.opendev.org/c/openstack/nova/+/854132 | |
| 14:42:14 | sean-k-mooney | its failing pretty consitently | |