| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-11 | |||
| 13:24:09 | sean-k-mooney | e.g. before you do all that work which is good, are we confident it will help | |
| 13:24:18 | lyarwood | yup that's a fair test | |
| 13:24:21 | lyarwood | maybe not 300 | |
| 13:24:23 | sean-k-mooney | i think it might if its an issue with the guest not being ready | |
| 13:24:29 | sean-k-mooney | well ya mayb like 30 | |
| 13:24:30 | lyarwood | as other things will likely timeout | |
| 13:24:47 | lyarwood | okay if someone can test that it would be great | |
| 13:25:09 | gibi | I will push a tempest patch and a nova depends-on for that | |
| 13:25:17 | gibi | * for the sleep casse | |
| 13:25:18 | gibi | case | |
| 13:25:53 | sean-k-mooney | how is it already half 1 | |
| 13:26:19 | sean-k-mooney | not that the conversation is not engagin but i keep getting distracted today | |
| 13:26:41 | sean-k-mooney | i ment ot start with the off path acclerator spec this morning | |
| 13:27:58 | gibi | I feel your pain sean-k-mooney I had a good day on tuesday but wednesday was a loss and today doesn't look good either :) | |
| 13:40:55 | gibi | lyarwood sean-k-mooney: so what I see is that test_live_block_migration_with_attached_volume causing the most kernel panic (if not all) and the panic happens when tempest runs the resource cleanup after the whole test class. So I will add the extra sleep at the top of the volume detach code to see if that helps | |
| 13:41:20 | gibi | does it sounds good to you? | |
| 13:42:01 | gibi | I can even dump console before and after the sleep to see if there was activity in the guest | |
| 14:01:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817562 | |
| 14:01:37 | gibi | lyarwood, sean-k-mooney that is my trial ^^ | |
| 14:01:57 | gibi | ohh I pushed it to master /p\ | |
| 14:02:13 | sean-k-mooney | well we see the issue on master too right? | |
| 14:02:25 | sean-k-mooney | just less often or is that now resoved | |
| 14:02:27 | gibi | sean-k-mooney: not really | |
| 14:02:28 | sean-k-mooney | on master | |
| 14:02:42 | gibi | or it is so infrequent that I don't see it | |
| 14:03:24 | sean-k-mooney | oh ok well i guess just cherry pick it | |
| 14:03:38 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:03:41 | gibi | sean-k-mooney: here are the recent hits https://paste.opendev.org/show/810943/ | |
| 14:04:20 | sean-k-mooney | look like there are 2 failture there on master | |
| 14:04:25 | sean-k-mooney | | 8fc6afb0549b43d7a4aa7179dfc12700 | 2021-11-05T21:25:53 | openstack/tempest | check | https://review.opendev.org/810808 | master | | |
| 14:04:27 | sean-k-mooney | | 5695c2c90c7e4d579e7f412cd6c60ef8 | 2021-11-05T17:44:50 | openstack/tempest | check | https://review.opendev.org/810809 | master | |
| 14:04:46 | gibi | nope does are just build failures without the log match | |
| 14:04:47 | sean-k-mooney | oh those are fips | |
| 14:04:52 | sean-k-mooney | ya | |
| 14:04:54 | sean-k-mooney | ok | |
| 14:08:38 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:20:52 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:29:38 | lyarwood | gibi: during the volume detach right? I thought I was already dumping the console on failure then? | |
| 14:31:24 | lyarwood | sean-k-mooney: random one, do you know the configurable to switch back to ml2/ovs in devstack? | |
| 14:31:34 | lyarwood | OVN is being a total PITA on F34 | |
| 14:32:02 | lyarwood | ah found an example, sorry | |
| 14:37:35 | opendevreview | Merged openstack/nova master: Log failed metadata retrievals https://review.opendev.org/c/openstack/nova/+/815859 | |
| 14:37:42 | opendevreview | Merged openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | |
| 14:37:58 | opendevreview | Merged openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359 | |
| 14:48:20 | sean-k-mooney | lyarwood: its multiple things but ya you can look at os-vif | |
| 14:48:58 | sean-k-mooney | https://github.com/openstack/os-vif/commit/b75af3ae1303cf449e881f14191bb10a3fdc373d#diff-108978819c05ae183d88ec87959c2341a94cfc3f9465e3aeee82d554217b4f58 | |
| 14:49:59 | sean-k-mooney | also OVN beign a PITA no way how could that be :P | |
| 14:50:37 | gibi | lyarwood: I use these in my local conf https://paste.opendev.org/show/810945/ | |
| 14:51:09 | sean-k-mooney | i dont think thats actully sufficent now | |
| 14:51:26 | sean-k-mooney | unless they have fixed it recently | |
| 14:51:37 | gibi | sean-k-mooney: I build devstack with this last week I think | |
| 14:51:41 | sean-k-mooney | Q_AGENT=openvswitch did not disable qemu last time i check | |
| 14:51:49 | sean-k-mooney | ok so them might have fixed it then | |
| 14:52:14 | gibi | lyarwood: I added console dumping before and after the sleep that happens before detach https://review.opendev.org/c/openstack/tempest/+/817560/1/tempest/api/compute/base.py | |
| 14:52:30 | lyarwood | gibi: ack cool thanks | |
| 14:53:03 | lyarwood | sean-k-mooney: yeah thanks, I think the local issue was with an older version of ovn as the f34 jobs are passing upstream | |
| 14:53:12 | lyarwood | die 174 'Socket /var/run/openvswitch/ovnnb_db.sock not found' | |
| 14:53:14 | lyarwood | maybe not | |
| 14:53:16 | lyarwood | urgh | |
| 14:53:35 | sean-k-mooney | it might be selinux | |
| 14:53:57 | sean-k-mooney | devstack used to complie form source in the past to make ovn work | |
| 14:54:09 | lyarwood | Permissive | |
| 14:54:09 | lyarwood | $ getenforce | |
| 14:55:06 | lyarwood | https://paste.centos.org/view/2cd83b46 - I get the feeling devstack is just looking for the wrong socket somehow | |
| 14:55:54 | sean-k-mooney | am if i remmeber they create a symlink | |
| 14:56:05 | sean-k-mooney | they bing devstack | |
| 14:56:36 | sean-k-mooney | devstack was simlink /run/ovn to /run/openvswtch | |
| 14:56:43 | sean-k-mooney | so that they were all in one folder | |
| 14:56:50 | sean-k-mooney | so ys that proably is broken | |
| 14:58:03 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/lib/neutron_plugins/ovn_agent#L362 | |
| 14:59:22 | sean-k-mooney | lyarwood: is /run a symlink of /var/run on your system or the other way around | |
| 14:59:35 | sean-k-mooney | it shold be but if not that would be the issue | |
| 14:59:58 | sean-k-mooney | devstack is assuming that /run and /var/run are the same location | |
| 15:00:08 | lyarwood | /var/run is a symlink to /run | |
| 15:00:43 | lyarwood | I'll just nuke this env and start again | |
| 15:00:49 | lyarwood | but thanks for the pointers | |
| 15:01:02 | sean-k-mooney | ok then you shoudl see. /run/ovn/ovnnb_db.sock at /run/openvswitch/ovnnb_db.sock | |
| 15:01:23 | sean-k-mooney | this was broken on one of my setup | |
| 15:01:39 | lyarwood | yeah it wasn't there | |
| 15:01:41 | sean-k-mooney | i had to fully uninstall openvswtich and ovn | |
| 15:01:47 | lyarwood | the symlink was missing | |
| 15:02:02 | lyarwood | I'm just going to reimage the vm and start again | |
| 15:02:07 | sean-k-mooney | then remove the /run/openvswtch and /run/ovn dirs and the /var/run versions | |
| 15:02:09 | sean-k-mooney | then it worked | |
| 15:02:48 | sean-k-mooney | well you can but i would jsut remove the directores and packages and try stacking one more time first | |
| 15:04:02 | sean-k-mooney | i had to do this to get vdpa working on my home server on ubuntu about 3 weeks ago but its been fine since then | |
| 15:25:27 | EugenMayer | sean-k-mooney i made ESXi work finally, interestingly, one cannot install it on openstack but i can install it under my local libvirt and thus use a qcow of the prininstalled variant on openstack, see https://stackoverflow.com/questions/69924901/how-to-run-esxi-on-openstack-as-kvm-vm - what wonders me is, that the installer should theoratically | |
| 15:25:28 | EugenMayer | work under openstack too. I mean i use the same e1000/e1000e nics and is use libvirt qemu/kvm locally as for the hv in nove. Odd | |
| 15:26:21 | EugenMayer | Kind of unexpected "i cannot explain" why e1000e works for 7.x when booting the non installer, but not for the installer under openstack. | |
| 15:43:14 | sean-k-mooney | e1000e and e1000 are actully different nics | |
| 15:43:25 | sean-k-mooney | on is a pci nic and the other is pcie | |
| 15:43:40 | sean-k-mooney | but i have never had much luck using e1000e with openstack | |
| 15:43:48 | sean-k-mooney | you need to use the q35 manchine type | |
| 15:43:53 | sean-k-mooney | to get a pcie bus | |
| 15:43:59 | sean-k-mooney | perhaps that is the problem | |
| 15:44:09 | sean-k-mooney | the default machine type is pc-ix440 | |
| 15:44:19 | sean-k-mooney | which doen not have pcie only pci | |
| 15:48:40 | EugenMayer | that is an interesting point | |
| 15:48:59 | EugenMayer | but since i do not set this via meta-data it does not make sense that the qcow2 variant works and the other does not, right | |
| 15:49:53 | kevko | sean-k-mooney: hi, maybe I found something weird in my last discussed problem if you remember ... | |