| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-11 | |||
| 13:26:41 | sean-k-mooney | i ment ot start with the off path acclerator spec this morning | |
| 13:27:58 | gibi | I feel your pain sean-k-mooney I had a good day on tuesday but wednesday was a loss and today doesn't look good either :) | |
| 13:40:55 | gibi | lyarwood sean-k-mooney: so what I see is that test_live_block_migration_with_attached_volume causing the most kernel panic (if not all) and the panic happens when tempest runs the resource cleanup after the whole test class. So I will add the extra sleep at the top of the volume detach code to see if that helps | |
| 13:41:20 | gibi | does it sounds good to you? | |
| 13:42:01 | gibi | I can even dump console before and after the sleep to see if there was activity in the guest | |
| 14:01:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817562 | |
| 14:01:37 | gibi | lyarwood, sean-k-mooney that is my trial ^^ | |
| 14:01:57 | gibi | ohh I pushed it to master /p\ | |
| 14:02:13 | sean-k-mooney | well we see the issue on master too right? | |
| 14:02:25 | sean-k-mooney | just less often or is that now resoved | |
| 14:02:27 | gibi | sean-k-mooney: not really | |
| 14:02:28 | sean-k-mooney | on master | |
| 14:02:42 | gibi | or it is so infrequent that I don't see it | |
| 14:03:24 | sean-k-mooney | oh ok well i guess just cherry pick it | |
| 14:03:38 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:03:41 | gibi | sean-k-mooney: here are the recent hits https://paste.opendev.org/show/810943/ | |
| 14:04:20 | sean-k-mooney | look like there are 2 failture there on master | |
| 14:04:25 | sean-k-mooney | | 8fc6afb0549b43d7a4aa7179dfc12700 | 2021-11-05T21:25:53 | openstack/tempest | check | https://review.opendev.org/810808 | master | | |
| 14:04:27 | sean-k-mooney | | 5695c2c90c7e4d579e7f412cd6c60ef8 | 2021-11-05T17:44:50 | openstack/tempest | check | https://review.opendev.org/810809 | master | |
| 14:04:46 | gibi | nope does are just build failures without the log match | |
| 14:04:47 | sean-k-mooney | oh those are fips | |
| 14:04:52 | sean-k-mooney | ya | |
| 14:04:54 | sean-k-mooney | ok | |
| 14:08:38 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:20:52 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:29:38 | lyarwood | gibi: during the volume detach right? I thought I was already dumping the console on failure then? | |
| 14:31:24 | lyarwood | sean-k-mooney: random one, do you know the configurable to switch back to ml2/ovs in devstack? | |
| 14:31:34 | lyarwood | OVN is being a total PITA on F34 | |
| 14:32:02 | lyarwood | ah found an example, sorry | |
| 14:37:35 | opendevreview | Merged openstack/nova master: Log failed metadata retrievals https://review.opendev.org/c/openstack/nova/+/815859 | |
| 14:37:42 | opendevreview | Merged openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | |
| 14:37:58 | opendevreview | Merged openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359 | |
| 14:48:20 | sean-k-mooney | lyarwood: its multiple things but ya you can look at os-vif | |
| 14:48:58 | sean-k-mooney | https://github.com/openstack/os-vif/commit/b75af3ae1303cf449e881f14191bb10a3fdc373d#diff-108978819c05ae183d88ec87959c2341a94cfc3f9465e3aeee82d554217b4f58 | |
| 14:49:59 | sean-k-mooney | also OVN beign a PITA no way how could that be :P | |
| 14:50:37 | gibi | lyarwood: I use these in my local conf https://paste.opendev.org/show/810945/ | |
| 14:51:09 | sean-k-mooney | i dont think thats actully sufficent now | |
| 14:51:26 | sean-k-mooney | unless they have fixed it recently | |
| 14:51:37 | gibi | sean-k-mooney: I build devstack with this last week I think | |
| 14:51:41 | sean-k-mooney | Q_AGENT=openvswitch did not disable qemu last time i check | |
| 14:51:49 | sean-k-mooney | ok so them might have fixed it then | |
| 14:52:14 | gibi | lyarwood: I added console dumping before and after the sleep that happens before detach https://review.opendev.org/c/openstack/tempest/+/817560/1/tempest/api/compute/base.py | |
| 14:52:30 | lyarwood | gibi: ack cool thanks | |
| 14:53:03 | lyarwood | sean-k-mooney: yeah thanks, I think the local issue was with an older version of ovn as the f34 jobs are passing upstream | |
| 14:53:12 | lyarwood | die 174 'Socket /var/run/openvswitch/ovnnb_db.sock not found' | |
| 14:53:14 | lyarwood | maybe not | |
| 14:53:16 | lyarwood | urgh | |
| 14:53:35 | sean-k-mooney | it might be selinux | |
| 14:53:57 | sean-k-mooney | devstack used to complie form source in the past to make ovn work | |
| 14:54:09 | lyarwood | Permissive | |
| 14:54:09 | lyarwood | $ getenforce | |
| 14:55:06 | lyarwood | https://paste.centos.org/view/2cd83b46 - I get the feeling devstack is just looking for the wrong socket somehow | |
| 14:55:54 | sean-k-mooney | am if i remmeber they create a symlink | |
| 14:56:05 | sean-k-mooney | they bing devstack | |
| 14:56:36 | sean-k-mooney | devstack was simlink /run/ovn to /run/openvswtch | |
| 14:56:43 | sean-k-mooney | so that they were all in one folder | |
| 14:56:50 | sean-k-mooney | so ys that proably is broken | |
| 14:58:03 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/lib/neutron_plugins/ovn_agent#L362 | |
| 14:59:22 | sean-k-mooney | lyarwood: is /run a symlink of /var/run on your system or the other way around | |
| 14:59:35 | sean-k-mooney | it shold be but if not that would be the issue | |
| 14:59:58 | sean-k-mooney | devstack is assuming that /run and /var/run are the same location | |
| 15:00:08 | lyarwood | /var/run is a symlink to /run | |
| 15:00:43 | lyarwood | I'll just nuke this env and start again | |
| 15:00:49 | lyarwood | but thanks for the pointers | |
| 15:01:02 | sean-k-mooney | ok then you shoudl see. /run/ovn/ovnnb_db.sock at /run/openvswitch/ovnnb_db.sock | |
| 15:01:23 | sean-k-mooney | this was broken on one of my setup | |
| 15:01:39 | lyarwood | yeah it wasn't there | |
| 15:01:41 | sean-k-mooney | i had to fully uninstall openvswtich and ovn | |
| 15:01:47 | lyarwood | the symlink was missing | |
| 15:02:02 | lyarwood | I'm just going to reimage the vm and start again | |
| 15:02:07 | sean-k-mooney | then remove the /run/openvswtch and /run/ovn dirs and the /var/run versions | |
| 15:02:09 | sean-k-mooney | then it worked | |
| 15:02:48 | sean-k-mooney | well you can but i would jsut remove the directores and packages and try stacking one more time first | |
| 15:04:02 | sean-k-mooney | i had to do this to get vdpa working on my home server on ubuntu about 3 weeks ago but its been fine since then | |
| 15:25:27 | EugenMayer | sean-k-mooney i made ESXi work finally, interestingly, one cannot install it on openstack but i can install it under my local libvirt and thus use a qcow of the prininstalled variant on openstack, see https://stackoverflow.com/questions/69924901/how-to-run-esxi-on-openstack-as-kvm-vm - what wonders me is, that the installer should theoratically | |
| 15:25:28 | EugenMayer | work under openstack too. I mean i use the same e1000/e1000e nics and is use libvirt qemu/kvm locally as for the hv in nove. Odd | |
| 15:26:21 | EugenMayer | Kind of unexpected "i cannot explain" why e1000e works for 7.x when booting the non installer, but not for the installer under openstack. | |
| 15:43:14 | sean-k-mooney | e1000e and e1000 are actully different nics | |
| 15:43:25 | sean-k-mooney | on is a pci nic and the other is pcie | |
| 15:43:40 | sean-k-mooney | but i have never had much luck using e1000e with openstack | |
| 15:43:48 | sean-k-mooney | you need to use the q35 manchine type | |
| 15:43:53 | sean-k-mooney | to get a pcie bus | |
| 15:43:59 | sean-k-mooney | perhaps that is the problem | |
| 15:44:09 | sean-k-mooney | the default machine type is pc-ix440 | |
| 15:44:19 | sean-k-mooney | which doen not have pcie only pci | |
| 15:48:40 | EugenMayer | that is an interesting point | |
| 15:48:59 | EugenMayer | but since i do not set this via meta-data it does not make sense that the qcow2 variant works and the other does not, right | |
| 15:49:53 | kevko | sean-k-mooney: hi, maybe I found something weird in my last discussed problem if you remember ... | |
| 15:50:10 | EugenMayer | --property hw_disk_bus=sata --property hw_cpu_policy=dedicated --property img_config_drive=mandatory --property hw_cdrom_bus=ide --property hw_vif_model=e1000 --property hw_boot_menu=true --property hw_qemu_guest_agent=no | |
| 15:50:10 | EugenMayer | if using q35 fixes it, i must have used it on my local as on the openstack libvirt. For the latter i did not. The properties i used for running the qcow or iso image where the same: | |
| 15:50:20 | EugenMayer | or for 7.x | |
| 15:50:25 | EugenMayer | --property hw_disk_bus=sata --property hw_cpu_policy=dedicated --property img_config_drive=mandatory --property hw_cdrom_bus=ide --property hw_vif_model=e1000e --property hw_boot_menu=true --property hw_qemu_guest_agent=no | |
| 15:52:57 | opendevreview | mitya-eremeev-2 proposed openstack/nova master: Close Glance image if downloading failed. https://review.opendev.org/c/openstack/nova/+/815347 | |
| 16:20:03 | gmann | lyarwood: gibi sean-k-mooney on tempest things to ssh things. we have two things here 1. all API tests are configurable to pause SSHing via run_validation config option 2. all scenario tests run SSH validation and does not care of run_validation config as most of scenario tests need to ssh | |
| 16:21:24 | lyarwood | gmann: I'm not sure what you mean by 1. do you have an example? | |
| 16:22:13 | lyarwood | what we are looking for here is a check during creation before any additional actions or validations actually happen | |
| 16:22:42 | lyarwood | I've got a WIP locally that I can finish up this evening to show what I mean | |
| 16:24:36 | gmann | lyarwood: we do access remote client only if run_validation is true in API tests | |
| 16:24:41 | gmann | lyarwood: but I think this does not check https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/common/compute.py#L270 | |
| 16:25:08 | gmann | lyarwood: we should have this with condition CONF.validation.run_validation | |