| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-11 | |||
| 12:51:07 | gibi | /o\ | |
| 12:51:12 | lyarwood | sean-k-mooney: right but the change I've pointed to is reading console output | |
| 12:51:23 | sean-k-mooney | ya which i think is wrong | |
| 12:51:48 | lyarwood | Wonderful, just making sure | |
| 12:52:30 | sean-k-mooney | so the old validation code support waithign for the server to be reacable via ping or ssh via config | |
| 12:52:40 | sean-k-mooney | i was expecting it to jsut use that | |
| 12:52:54 | sean-k-mooney | which did not have any console interaction if i understand correctly | |
| 12:53:28 | lyarwood | There's nothing I can see in the create code that did this previously | |
| 12:53:43 | lyarwood | AFAICT this was added as a step prior to the tests attempting to SSH into the instance | |
| 12:53:55 | lyarwood | to ensure the instance had booted up and was ssh'able itself | |
| 12:57:07 | sean-k-mooney | ack so i tought you can use https://github.com/openstack/tempest/blob/master/tempest/config.py#L897-L970 to configure vlaidations for all test | |
| 12:57:22 | sean-k-mooney | can we use that to adress this in the job config | |
| 12:58:13 | lyarwood | that doesn't actually do anything for most tests | |
| 12:58:27 | lyarwood | brb need to jump on a call | |
| 12:59:16 | sean-k-mooney | ... i tought this was ment to run on any test that created a server as part of the server create automatically | |
| 13:06:30 | gibi | sean-k-mooney: https://github.com/openstack/tempest/blob/master/tempest/scenario/manager.py#L229-L236 I think this describes the situation | |
| 13:06:55 | gibi | sean-k-mooney: so there was an intention to allow running validation for each server but it was never introduced globally | |
| 13:11:38 | sean-k-mooney | right i remember that being the intent | |
| 13:11:57 | sean-k-mooney | and in the past i think you could even confiure that validation metion to be either ping or ssh | |
| 13:12:09 | sean-k-mooney | there was a spec for this somewhere | |
| 13:15:14 | sean-k-mooney | this https://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/ssh-auth-strategy.html | |
| 13:15:27 | sean-k-mooney | """ it extends the valid value for wait_until with new types of wait abilities: PINGABLE and SSHABLE. """ | |
| 13:17:27 | sean-k-mooney | ... https://github.com/openstack/tempest/blob/ed89c77222917235290c8cc51974835528ed4cfa/tempest/common/compute.py#L101 | |
| 13:17:55 | sean-k-mooney | so ya i twas not actully implemeted | |
| 13:18:05 | gibi | yeah | |
| 13:18:19 | sean-k-mooney | this is also not the first time i have wanted to use this and discoverd this | |
| 13:18:26 | gibi | :) | |
| 13:19:44 | sean-k-mooney | maybe we shoudl jsut implement it | |
| 13:19:50 | sean-k-mooney | at least the pingable version | |
| 13:20:16 | sean-k-mooney | ssh would be nice but if it has an ip the os should be live enough for hotplug | |
| 13:20:25 | gibi | I think the comment also states that pingability means some level of network setup is in place and from the create_server perspective this cannot be ensured | |
| 13:20:29 | lyarwood | Yup I can hack on this | |
| 13:21:01 | sean-k-mooney | gibi: well we woudl wait for active and then waith for pingable | |
| 13:21:10 | sean-k-mooney | only if the server has a network | |
| 13:21:24 | gibi | do we need floating ip for pingability? | |
| 13:21:30 | lyarwood | has a network, fip etc yeah | |
| 13:21:31 | gibi | and securty group setuo? | |
| 13:21:45 | lyarwood | yeah we'd need to pass in and setup the validation resources | |
| 13:21:58 | lyarwood | it's entirely possible in the base server creation method | |
| 13:22:33 | gibi | OK then we are on the same page about what is required | |
| 13:22:51 | lyarwood | I'll revert the console stuff and try to hack on this in the background of calls this afternoon | |
| 13:23:08 | gibi | thank you lyarwood | |
| 13:23:30 | sean-k-mooney | by the way if we just put a sleep(300) in the test will the issue go away | |
| 13:23:55 | gibi | sean-k-mooney: that can be tried too | |
| 13:24:09 | sean-k-mooney | e.g. before you do all that work which is good, are we confident it will help | |
| 13:24:18 | lyarwood | yup that's a fair test | |
| 13:24:21 | lyarwood | maybe not 300 | |
| 13:24:23 | sean-k-mooney | i think it might if its an issue with the guest not being ready | |
| 13:24:29 | sean-k-mooney | well ya mayb like 30 | |
| 13:24:30 | lyarwood | as other things will likely timeout | |
| 13:24:47 | lyarwood | okay if someone can test that it would be great | |
| 13:25:09 | gibi | I will push a tempest patch and a nova depends-on for that | |
| 13:25:17 | gibi | * for the sleep casse | |
| 13:25:18 | gibi | case | |
| 13:25:53 | sean-k-mooney | how is it already half 1 | |
| 13:26:19 | sean-k-mooney | not that the conversation is not engagin but i keep getting distracted today | |
| 13:26:41 | sean-k-mooney | i ment ot start with the off path acclerator spec this morning | |
| 13:27:58 | gibi | I feel your pain sean-k-mooney I had a good day on tuesday but wednesday was a loss and today doesn't look good either :) | |
| 13:40:55 | gibi | lyarwood sean-k-mooney: so what I see is that test_live_block_migration_with_attached_volume causing the most kernel panic (if not all) and the panic happens when tempest runs the resource cleanup after the whole test class. So I will add the extra sleep at the top of the volume detach code to see if that helps | |
| 13:41:20 | gibi | does it sounds good to you? | |
| 13:42:01 | gibi | I can even dump console before and after the sleep to see if there was activity in the guest | |
| 14:01:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817562 | |
| 14:01:37 | gibi | lyarwood, sean-k-mooney that is my trial ^^ | |
| 14:01:57 | gibi | ohh I pushed it to master /p\ | |
| 14:02:13 | sean-k-mooney | well we see the issue on master too right? | |
| 14:02:25 | sean-k-mooney | just less often or is that now resoved | |
| 14:02:27 | gibi | sean-k-mooney: not really | |
| 14:02:28 | sean-k-mooney | on master | |
| 14:02:42 | gibi | or it is so infrequent that I don't see it | |
| 14:03:24 | sean-k-mooney | oh ok well i guess just cherry pick it | |
| 14:03:38 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:03:41 | gibi | sean-k-mooney: here are the recent hits https://paste.opendev.org/show/810943/ | |
| 14:04:20 | sean-k-mooney | look like there are 2 failture there on master | |
| 14:04:25 | sean-k-mooney | | 8fc6afb0549b43d7a4aa7179dfc12700 | 2021-11-05T21:25:53 | openstack/tempest | check | https://review.opendev.org/810808 | master | | |
| 14:04:27 | sean-k-mooney | | 5695c2c90c7e4d579e7f412cd6c60ef8 | 2021-11-05T17:44:50 | openstack/tempest | check | https://review.opendev.org/810809 | master | |
| 14:04:46 | gibi | nope does are just build failures without the log match | |
| 14:04:47 | sean-k-mooney | oh those are fips | |
| 14:04:52 | sean-k-mooney | ya | |
| 14:04:54 | sean-k-mooney | ok | |
| 14:08:38 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:20:52 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:29:38 | lyarwood | gibi: during the volume detach right? I thought I was already dumping the console on failure then? | |
| 14:31:24 | lyarwood | sean-k-mooney: random one, do you know the configurable to switch back to ml2/ovs in devstack? | |
| 14:31:34 | lyarwood | OVN is being a total PITA on F34 | |
| 14:32:02 | lyarwood | ah found an example, sorry | |
| 14:37:35 | opendevreview | Merged openstack/nova master: Log failed metadata retrievals https://review.opendev.org/c/openstack/nova/+/815859 | |
| 14:37:42 | opendevreview | Merged openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | |
| 14:37:58 | opendevreview | Merged openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359 | |
| 14:48:20 | sean-k-mooney | lyarwood: its multiple things but ya you can look at os-vif | |
| 14:48:58 | sean-k-mooney | https://github.com/openstack/os-vif/commit/b75af3ae1303cf449e881f14191bb10a3fdc373d#diff-108978819c05ae183d88ec87959c2341a94cfc3f9465e3aeee82d554217b4f58 | |
| 14:49:59 | sean-k-mooney | also OVN beign a PITA no way how could that be :P | |
| 14:50:37 | gibi | lyarwood: I use these in my local conf https://paste.opendev.org/show/810945/ | |
| 14:51:09 | sean-k-mooney | i dont think thats actully sufficent now | |
| 14:51:26 | sean-k-mooney | unless they have fixed it recently | |
| 14:51:37 | gibi | sean-k-mooney: I build devstack with this last week I think | |
| 14:51:41 | sean-k-mooney | Q_AGENT=openvswitch did not disable qemu last time i check | |
| 14:51:49 | sean-k-mooney | ok so them might have fixed it then | |
| 14:52:14 | gibi | lyarwood: I added console dumping before and after the sleep that happens before detach https://review.opendev.org/c/openstack/tempest/+/817560/1/tempest/api/compute/base.py | |
| 14:52:30 | lyarwood | gibi: ack cool thanks | |
| 14:53:03 | lyarwood | sean-k-mooney: yeah thanks, I think the local issue was with an older version of ovn as the f34 jobs are passing upstream | |