| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-11 | |||
| 14:08:38 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:20:52 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: DNM: troubleshoot kernel panic https://review.opendev.org/c/openstack/nova/+/817564 | |
| 14:29:38 | lyarwood | gibi: during the volume detach right? I thought I was already dumping the console on failure then? | |
| 14:31:24 | lyarwood | sean-k-mooney: random one, do you know the configurable to switch back to ml2/ovs in devstack? | |
| 14:31:34 | lyarwood | OVN is being a total PITA on F34 | |
| 14:32:02 | lyarwood | ah found an example, sorry | |
| 14:37:35 | opendevreview | Merged openstack/nova master: Log failed metadata retrievals https://review.opendev.org/c/openstack/nova/+/815859 | |
| 14:37:42 | opendevreview | Merged openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | |
| 14:37:58 | opendevreview | Merged openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359 | |
| 14:48:20 | sean-k-mooney | lyarwood: its multiple things but ya you can look at os-vif | |
| 14:48:58 | sean-k-mooney | https://github.com/openstack/os-vif/commit/b75af3ae1303cf449e881f14191bb10a3fdc373d#diff-108978819c05ae183d88ec87959c2341a94cfc3f9465e3aeee82d554217b4f58 | |
| 14:49:59 | sean-k-mooney | also OVN beign a PITA no way how could that be :P | |
| 14:50:37 | gibi | lyarwood: I use these in my local conf https://paste.opendev.org/show/810945/ | |
| 14:51:09 | sean-k-mooney | i dont think thats actully sufficent now | |
| 14:51:26 | sean-k-mooney | unless they have fixed it recently | |
| 14:51:37 | gibi | sean-k-mooney: I build devstack with this last week I think | |
| 14:51:41 | sean-k-mooney | Q_AGENT=openvswitch did not disable qemu last time i check | |
| 14:51:49 | sean-k-mooney | ok so them might have fixed it then | |
| 14:52:14 | gibi | lyarwood: I added console dumping before and after the sleep that happens before detach https://review.opendev.org/c/openstack/tempest/+/817560/1/tempest/api/compute/base.py | |
| 14:52:30 | lyarwood | gibi: ack cool thanks | |
| 14:53:03 | lyarwood | sean-k-mooney: yeah thanks, I think the local issue was with an older version of ovn as the f34 jobs are passing upstream | |
| 14:53:12 | lyarwood | die 174 'Socket /var/run/openvswitch/ovnnb_db.sock not found' | |
| 14:53:14 | lyarwood | maybe not | |
| 14:53:16 | lyarwood | urgh | |
| 14:53:35 | sean-k-mooney | it might be selinux | |
| 14:53:57 | sean-k-mooney | devstack used to complie form source in the past to make ovn work | |
| 14:54:09 | lyarwood | Permissive | |
| 14:54:09 | lyarwood | $ getenforce | |
| 14:55:06 | lyarwood | https://paste.centos.org/view/2cd83b46 - I get the feeling devstack is just looking for the wrong socket somehow | |
| 14:55:54 | sean-k-mooney | am if i remmeber they create a symlink | |
| 14:56:05 | sean-k-mooney | they bing devstack | |
| 14:56:36 | sean-k-mooney | devstack was simlink /run/ovn to /run/openvswtch | |
| 14:56:43 | sean-k-mooney | so that they were all in one folder | |
| 14:56:50 | sean-k-mooney | so ys that proably is broken | |
| 14:58:03 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/lib/neutron_plugins/ovn_agent#L362 | |
| 14:59:22 | sean-k-mooney | lyarwood: is /run a symlink of /var/run on your system or the other way around | |
| 14:59:35 | sean-k-mooney | it shold be but if not that would be the issue | |
| 14:59:58 | sean-k-mooney | devstack is assuming that /run and /var/run are the same location | |
| 15:00:08 | lyarwood | /var/run is a symlink to /run | |
| 15:00:43 | lyarwood | I'll just nuke this env and start again | |
| 15:00:49 | lyarwood | but thanks for the pointers | |
| 15:01:02 | sean-k-mooney | ok then you shoudl see. /run/ovn/ovnnb_db.sock at /run/openvswitch/ovnnb_db.sock | |
| 15:01:23 | sean-k-mooney | this was broken on one of my setup | |
| 15:01:39 | lyarwood | yeah it wasn't there | |
| 15:01:41 | sean-k-mooney | i had to fully uninstall openvswtich and ovn | |
| 15:01:47 | lyarwood | the symlink was missing | |
| 15:02:02 | lyarwood | I'm just going to reimage the vm and start again | |
| 15:02:07 | sean-k-mooney | then remove the /run/openvswtch and /run/ovn dirs and the /var/run versions | |
| 15:02:09 | sean-k-mooney | then it worked | |
| 15:02:48 | sean-k-mooney | well you can but i would jsut remove the directores and packages and try stacking one more time first | |
| 15:04:02 | sean-k-mooney | i had to do this to get vdpa working on my home server on ubuntu about 3 weeks ago but its been fine since then | |
| 15:25:27 | EugenMayer | sean-k-mooney i made ESXi work finally, interestingly, one cannot install it on openstack but i can install it under my local libvirt and thus use a qcow of the prininstalled variant on openstack, see https://stackoverflow.com/questions/69924901/how-to-run-esxi-on-openstack-as-kvm-vm - what wonders me is, that the installer should theoratically | |
| 15:25:28 | EugenMayer | work under openstack too. I mean i use the same e1000/e1000e nics and is use libvirt qemu/kvm locally as for the hv in nove. Odd | |
| 15:26:21 | EugenMayer | Kind of unexpected "i cannot explain" why e1000e works for 7.x when booting the non installer, but not for the installer under openstack. | |
| 15:43:14 | sean-k-mooney | e1000e and e1000 are actully different nics | |
| 15:43:25 | sean-k-mooney | on is a pci nic and the other is pcie | |
| 15:43:40 | sean-k-mooney | but i have never had much luck using e1000e with openstack | |
| 15:43:48 | sean-k-mooney | you need to use the q35 manchine type | |
| 15:43:53 | sean-k-mooney | to get a pcie bus | |
| 15:43:59 | sean-k-mooney | perhaps that is the problem | |
| 15:44:09 | sean-k-mooney | the default machine type is pc-ix440 | |
| 15:44:19 | sean-k-mooney | which doen not have pcie only pci | |
| 15:48:40 | EugenMayer | that is an interesting point | |
| 15:48:59 | EugenMayer | but since i do not set this via meta-data it does not make sense that the qcow2 variant works and the other does not, right | |
| 15:49:53 | kevko | sean-k-mooney: hi, maybe I found something weird in my last discussed problem if you remember ... | |
| 15:50:10 | EugenMayer | --property hw_disk_bus=sata --property hw_cpu_policy=dedicated --property img_config_drive=mandatory --property hw_cdrom_bus=ide --property hw_vif_model=e1000 --property hw_boot_menu=true --property hw_qemu_guest_agent=no | |
| 15:50:10 | EugenMayer | if using q35 fixes it, i must have used it on my local as on the openstack libvirt. For the latter i did not. The properties i used for running the qcow or iso image where the same: | |
| 15:50:20 | EugenMayer | or for 7.x | |
| 15:50:25 | EugenMayer | --property hw_disk_bus=sata --property hw_cpu_policy=dedicated --property img_config_drive=mandatory --property hw_cdrom_bus=ide --property hw_vif_model=e1000e --property hw_boot_menu=true --property hw_qemu_guest_agent=no | |
| 15:52:57 | opendevreview | mitya-eremeev-2 proposed openstack/nova master: Close Glance image if downloading failed. https://review.opendev.org/c/openstack/nova/+/815347 | |
| 16:20:03 | gmann | lyarwood: gibi sean-k-mooney on tempest things to ssh things. we have two things here 1. all API tests are configurable to pause SSHing via run_validation config option 2. all scenario tests run SSH validation and does not care of run_validation config as most of scenario tests need to ssh | |
| 16:21:24 | lyarwood | gmann: I'm not sure what you mean by 1. do you have an example? | |
| 16:22:13 | lyarwood | what we are looking for here is a check during creation before any additional actions or validations actually happen | |
| 16:22:42 | lyarwood | I've got a WIP locally that I can finish up this evening to show what I mean | |
| 16:24:36 | gmann | lyarwood: we do access remote client only if run_validation is true in API tests | |
| 16:24:41 | gmann | lyarwood: but I think this does not check https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/common/compute.py#L270 | |
| 16:25:08 | gmann | lyarwood: we should have this with condition CONF.validation.run_validation | |
| 16:25:12 | lyarwood | yeah I'm nuking that implementation FWIW | |
| 16:25:27 | lyarwood | parsing the console it's valid IMHO | |
| 16:25:46 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Revert project-specific APIs for servers https://review.opendev.org/c/openstack/nova/+/816206 | |
| 16:26:09 | dansmith | gmann: johnthetubaguy[m] ^ wasn't too bad to add that new scenario, and it helps to be able to see what works and how in that case | |
| 16:26:48 | gmann | lyarwood: +1 thanks, ping me once you are ready with that or even we can add new config option also in case we do not want all tests doing ssh things and do not want to wait for console ? | |
| 16:27:06 | gmann | dansmith: thanks, I will check it after lunch today. | |
| 16:27:39 | lyarwood | gmann: I'm overloading wait_until to accept PINGABLE and SSHABLE, both basically fallback to ACTIVE if validations aren't enabled | |
| 16:28:12 | lyarwood | gmann: but when they are we wait first until the server is ACTIVE and then pingable or sshable etc. | |
| 16:32:01 | gmann | lyarwood: yeah, currently this method make server pingable or sshable if validatable=True but not do actual ping/ssh and with the new way you are making wait_until to do actual ssh/ping and after Active state ? | |
| 16:42:34 | lyarwood | gmann: sorry on a call, yeah correct so wait_until=PINGABLE would wait until the server is ACTIVE and will then wait until it is pingable *if* validatable=True etc | |
| 16:42:56 | lyarwood | gmann: if not it just waits until it's ACTIVE and continues | |
| 16:43:12 | gmann | lyarwood: +1 | |
| 17:01:44 | opendevreview | mitya-eremeev-2 proposed openstack/nova master: Close Glance image if downloading failed. https://review.opendev.org/c/openstack/nova/+/815347 | |
| 17:13:54 | gibi | lyarwood, sean-k-mooney: the first run with the 30 sec sleep before detach finished and did not generated any kernel panic https://zuul.opendev.org/t/openstack/build/06cb831dd6c94a2fa8ca373662014dd2/logs | |
| 17:14:25 | gibi | but also the guest did not print anything to the console during that extra 30 seconds | |
| 17:14:36 | gibi | so no hard proof yet that we need to wait | |
| 17:14:54 | gibi | I will recheck it | |
| 17:15:26 | sean-k-mooney | ack | |
| 17:15:59 | sean-k-mooney | so you wait for it to be active then sleep 30 right | |
| 17:16:03 | sean-k-mooney | then attach and detach | |
| 17:17:10 | gibi | nope, the failures are happening during test class cleanup | |
| 17:17:13 | sean-k-mooney | oh you put it in the detach | |
| 17:17:18 | gibi | so I just wait before the detach | |