| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-18 | |||
| 12:08:02 | ygk_12345 | sean-k-mooney so what logs to check in particulaer ? | |
| 12:08:49 | sean-k-mooney | well first you need to look at the instance action logs for the affected instance and determin that no api action was performend on the guest | |
| 12:08:52 | ygk_12345 | sean-k-mooney its happening only with a particular image | |
| 12:09:18 | sean-k-mooney | the image cant affect this unless you are using the watchdog and the vm hung and was rebooted | |
| 12:09:26 | ygk_12345 | where can I find those logs ? | |
| 12:10:09 | sean-k-mooney | openstack server event list <uuid> | |
| 12:11:15 | ygk_12345 | no actions except two reboots which I did | |
| 12:11:28 | ygk_12345 | create,reboot,reboot | |
| 12:11:58 | sean-k-mooney | ok if there are no action at the time of the events you next need to look at the neutron agent logs | |
| 12:12:17 | sean-k-mooney | basically you need to check why the agent updated the port status | |
| 12:12:20 | ygk_12345 | which agent in particular ? | |
| 12:12:32 | ygk_12345 | dhcp ? | |
| 12:12:43 | sean-k-mooney | if it was devstack it would be devstack@q-agt but its the neutron ovs l2 agent | |
| 12:12:55 | ygk_12345 | ok | |
| 12:13:31 | sean-k-mooney | if nova is not perfroming any instance actions then the port vif:host-id should not be updating | |
| 12:14:05 | ygk_12345 | how can I track the port status of an instance in the ovs logs ? | |
| 12:14:14 | sean-k-mooney | so the only thing that would cause those event to be emitted is a change in state form the ovs agent | |
| 12:14:52 | sean-k-mooney | you need to have the neutron port uuid if the agent is in debug mode it will print the uuid of the port it is processign at differnt times | |
| 12:15:02 | ygk_12345 | ok let me check | |
| 12:15:14 | sean-k-mooney | what you are looking for is if the port is listed in teh added or deleted list of ports around teh time of the events | |
| 12:15:33 | sean-k-mooney | that would indicate the port being removed or added to the ovs bridge | |
| 12:15:50 | sean-k-mooney | simialrly if the admin state on the newon port was changed it would also show up there | |
| 12:16:38 | sean-k-mooney | ygk_12345: the other thing that you cold check before going into the agent logs is the qemu instance log | |
| 12:16:48 | ygk_12345 | oh ok | |
| 12:16:50 | sean-k-mooney | and confirm that there are no restarts | |
| 12:17:31 | sean-k-mooney | ygk_12345: do you enabel the watchdog or qemu guest agent in teh image that is affected | |
| 12:18:00 | ygk_12345 | no idea | |
| 12:18:15 | sean-k-mooney | they would be listed in the image metadata in glance | |
| 12:20:00 | sean-k-mooney | ygk_12345: those event typically mean 1 of two things. eighet the tap device was removed and added to ovs, or the port was updated (host-id or admin status) | |
| 12:20:10 | ygk_12345 | i dont think they are set for that image | |
| 12:20:17 | sean-k-mooney | ok | |
| 12:20:31 | sean-k-mooney | this sound like its not a nova issue for what its worth | |
| 12:21:41 | sean-k-mooney | but check the qemu log to confim there is not reboot of the instace when it happened. if so then you need to look at the l2 agent logs if that does not show anyting then you need to check the dhcp agent and neutron server logs for the port uuid | |
| 12:21:50 | sean-k-mooney | to track down why the event was sent | |
| 12:26:33 | ygk_12345 | ok | |
| 12:28:19 | ygk_12345 | sean-k-mooney even when the vnics have disappeared, those ports are being shown as active in nova | |
| 12:28:28 | ygk_12345 | how can this be ? | |
| 12:31:10 | sean-k-mooney | without logs i cant really help you resolve this. can you file a bug and attach some logs specificly the nova compute and l2 agent logs for aroudn the time when this happened | |
| 12:31:49 | ygk_12345 | ok | |
| 12:32:02 | sean-k-mooney | what your describing does not really match to any known broken behviaor that comes to mind | |
| 12:32:43 | sean-k-mooney | the only way to revmoe an interface form a vm that is runnign other then a nova interface detach which is not present in the event logs | |
| 12:32:50 | sean-k-mooney | is to delete the neutron port | |
| 12:33:05 | sean-k-mooney | that will send a network-vif-deleted event to nova | |
| 12:33:19 | sean-k-mooney | but presuably you did not delete the nuetorn port | |
| 12:33:32 | sean-k-mooney | so im not aware of any code path that could result in this | |
| 12:33:52 | ygk_12345 | strange | |
| 12:34:03 | sean-k-mooney | the info cache currption issue requires a hard reboot for the instance to be lost or a move operations | |
| 12:44:59 | ygk_12345 | sean-k-mooney when the ping is unreachable for the vm , I dont see any messages for that port in the neutron-server except when I do a manual reboot | |
| 12:46:03 | lyarwood | ubuntu-- | |
| 12:46:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: test_evacuate.sh: Support libvirt-bin and libvirtd systemd services https://review.opendev.org/752650 | |
| 12:46:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | |
| 12:46:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982 | |
| 12:46:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_NATIVE_TLS_VERSION https://review.opendev.org/746983 | |
| 12:46:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_BETTER_SIGKILL_HANDLING https://review.opendev.org/746984 | |
| 12:46:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985 | |
| 12:46:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986 | |
| 12:47:09 | stephenfin | lyarwood: https://media.tenor.com/images/a53dc07bf0a6f8822f16c0760299f915/tenor.gif | |
| 12:49:13 | gibi | :) | |
| 12:51:35 | lyarwood | yup sorry all | |
| 12:53:56 | lyarwood | back this evening to clean up any fallout caused by all of this | |
| 12:56:33 | gibi | ack | |
| 12:58:54 | ygk_12345 | sean-k-mooney i have posted this bug https://bugs.launchpad.net/nova/+bug/1896226 | |
| 12:58:54 | openstack | Launchpad bug 1896226 in OpenStack Compute (nova) "The vnics are disappearing in the vm" [Undecided,New] | |
| 13:02:59 | kashyap | lyarwood: Hey, so do you know how many CPU cores did that reproducer host had? | |
| 13:03:26 | kashyap | lyarwood: I'm asking because, depending on that I may need to fork my CPU workload to spread across multiple (CPU) cores | |
| 13:05:10 | kashyap | Meanwhile ... lovely: I just created a fresh F32 VM on Ubuntu Focal, and the bootloader is filling up little yellow squares on my screen. | |
| 13:10:15 | gibi | kashyap: I think it is 8 https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/zuul-info/host-info.controller.yaml#440 | |
| 13:11:01 | gibi | kashyap: at least this is the CI job run linked to the bug report | |
| 13:17:50 | kashyap | gibi: So that's the "host VM" where Tempest is running, yeaH? | |
| 13:17:50 | bauzas | gibi: thanks | |
| 13:18:00 | kashyap | gibi: I'm trying to recreate the _exact_ damn setup :D | |
| 13:18:03 | bauzas | (for the TC ping) | |
| 13:18:46 | gibi | kashyap: yes I think that is the "host" where tempest runs | |
| 13:18:58 | sean-k-mooney | kashyap: it has 8 i think | |
| 13:19:09 | kashyap | gibi: Okay, 8 vCPUs, and 7599 MB of memory | |
| 13:19:22 | kashyap | gibi: And do you see what's the disk size? | |
| 13:19:30 | sean-k-mooney | yep ist my small flavor | |
| 13:20:12 | kashyap | sean-k-mooney: The disk size is ~16GB? | |
| 13:20:13 | gibi | size: 80.00 GB | |
| 13:20:21 | kashyap | gibi: Oh, right | |
| 13:20:28 | gibi | https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/zuul-info/host-info.controller.yaml#259-293 | |
| 13:20:39 | kashyap | Okay; I need to tweak the setup so that I'm also doing QEMU-on-KVM nested and not KVM-on-KVM nested | |
| 13:20:42 | kashyap | Thanks; /me back in a few | |
| 13:20:55 | gibi | thanks kashyap | |
| 13:22:20 | openstackgerrit | Merged openstack/nova master: releasenote: Add known issue for bug #1894804 https://review.opendev.org/752654 | |
| 13:22:20 | openstack | bug 1894804 in qemu (Ubuntu) "Second DEVICE_DELETED event missing during virtio-blk disk device detach" [Undecided,New] https://launchpad.net/bugs/1894804 | |
| 13:33:44 | sean-k-mooney | kashyap: you dont need 80G of disk by the way 25-40G is typeically more then enough | |
| 13:38:55 | kashyap | sean-k-mooney: Yeah, I have something like that anyway | |
| 13:40:21 | kashyap | sean-k-mooney: Do we have the full guest command-line of the Focal VM "host" upstream is using? | |
| 13:40:23 | sean-k-mooney | if you want an account on my home cloud by the way let me know and i can create one for you | |
| 13:40:46 | sean-k-mooney | no bug its just an openstack vm | |
| 13:41:27 | sean-k-mooney | with 8G of ram and 8 cpus and 80G fo disk | |
| 13:41:42 | sean-k-mooney | im not sure that really matters too much | |
| 13:42:02 | kashyap | sean-k-mooney: Okay; so what you gave me suffices | |
| 13:42:26 | kashyap | sean-k-mooney: For now, stable access (and it _is_ stable) to this VM is sufficient :) Thx! | |
| 13:42:29 | sean-k-mooney | ya my small flaovr is basicaly a proxy for the ci vms | |
| 13:43:19 | sean-k-mooney | except i add hw:cpu_sockets=2 hw:cpu_threads=2 hw:numa_nodes=2 and hw:mem_page_size=large | |
| 13:43:52 | sean-k-mooney | that should not change the behvior in this case but just an fyi | |
| 13:44:49 | kashyap | sean-k-mooney: One more: what's the CirrOS version being run here? | |
| 13:45:03 | kashyap | "here" as in, in the Tempest env | |