| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-18 | |||
| 11:34:25 | sean-k-mooney | vinay_m: you can propose a patch via gerrit and the docs job will render a preview sight | |
| 11:34:31 | sean-k-mooney | you can also build the docs locally | |
| 11:34:35 | sean-k-mooney | using tox | |
| 11:34:40 | sean-k-mooney | if you donwload the repos | |
| 11:35:04 | sean-k-mooney | i think that woudl be enouch to test the search capablity | |
| 11:40:29 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883 | |
| 11:40:30 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | |
| 11:42:26 | lyarwood | gmann: https://review.opendev.org/#/c/752641/ would you mind taking a look at this if you have time | |
| 11:54:19 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: WIP/DNM - Image defined ephemeral storage encryption https://review.opendev.org/752284 | |
| 11:54:52 | ygk_12345 | hi all | |
| 11:55:03 | ygk_12345 | i am facing a unique problem with rocky | |
| 11:56:14 | ygk_12345 | when I create an instance, it is having 4 vnics inside and they are getting ips from four dhcp tenant networks. but after sometime the vm is lossing its two network interfaces | |
| 11:57:00 | ygk_12345 | so when I rebooth it then, then the vnics are coming back but again after 100 secs they are disappearing again . is this a knownw issue ? that image worked earlier | |
| 12:01:25 | sean-k-mooney | it sounds like you have a neutron issue | |
| 12:02:06 | sean-k-mooney | ygk_12345: if neutron is retruning incosistent data then the force refesh of the netwrok info cache will result in the cache being populated with that inconsitent data | |
| 12:02:35 | sean-k-mooney | we have seen this downstream with contrail and aci before | |
| 12:02:49 | sean-k-mooney | i have not see it be an issue with intree network backends | |
| 12:03:04 | ygk_12345 | sean-k-mooney I see these messages in the compute node where that instance is | |
| 12:03:07 | ygk_12345 | Received unexpected event network-vif-plugged-86f1034d-837d-4e67-ad5e-63d9642a0b2a for instance with vm_state active and task_state None. | |
| 12:03:34 | ygk_12345 | Received unexpected event network-vif-unplugged-86f1034d-837d-4e67-ad5e-63d9642a0b2a for instance with vm_state active and task_state None. | |
| 12:04:14 | sean-k-mooney | what neutron backend are you using | |
| 12:04:20 | ygk_12345 | ovs | |
| 12:04:25 | ygk_12345 | ovs | |
| 12:04:27 | sean-k-mooney | with iptables or contrack | |
| 12:04:29 | ygk_12345 | ovs | |
| 12:04:46 | sean-k-mooney | for the firewall driver | |
| 12:04:56 | ygk_12345 | how to check it ? | |
| 12:05:13 | sean-k-mooney | /etc/neutron/plugins/ml2/ml2_conf.ini | |
| 12:05:22 | ygk_12345 | sean-k-mooney let me check | |
| 12:06:00 | ygk_12345 | sean-k-mooney firewall_driver = openvswitch | |
| 12:06:07 | sean-k-mooney | ok that is conntrack | |
| 12:06:44 | ygk_12345 | sean-k-mooney so whats the workaround for this issue ? | |
| 12:07:00 | sean-k-mooney | you have not provided enough info to know what the issue is yet | |
| 12:07:10 | ygk_12345 | why i s the vif intermittently loosing connection and dropping ? | |
| 12:07:43 | sean-k-mooney | those event wont casue the vif to be removed form the guest | |
| 12:08:02 | ygk_12345 | sean-k-mooney so what logs to check in particulaer ? | |
| 12:08:49 | sean-k-mooney | well first you need to look at the instance action logs for the affected instance and determin that no api action was performend on the guest | |
| 12:08:52 | ygk_12345 | sean-k-mooney its happening only with a particular image | |
| 12:09:18 | sean-k-mooney | the image cant affect this unless you are using the watchdog and the vm hung and was rebooted | |
| 12:09:26 | ygk_12345 | where can I find those logs ? | |
| 12:10:09 | sean-k-mooney | openstack server event list <uuid> | |
| 12:11:15 | ygk_12345 | no actions except two reboots which I did | |
| 12:11:28 | ygk_12345 | create,reboot,reboot | |
| 12:11:58 | sean-k-mooney | ok if there are no action at the time of the events you next need to look at the neutron agent logs | |
| 12:12:17 | sean-k-mooney | basically you need to check why the agent updated the port status | |
| 12:12:20 | ygk_12345 | which agent in particular ? | |
| 12:12:32 | ygk_12345 | dhcp ? | |
| 12:12:43 | sean-k-mooney | if it was devstack it would be devstack@q-agt but its the neutron ovs l2 agent | |
| 12:12:55 | ygk_12345 | ok | |
| 12:13:31 | sean-k-mooney | if nova is not perfroming any instance actions then the port vif:host-id should not be updating | |
| 12:14:05 | ygk_12345 | how can I track the port status of an instance in the ovs logs ? | |
| 12:14:14 | sean-k-mooney | so the only thing that would cause those event to be emitted is a change in state form the ovs agent | |
| 12:14:52 | sean-k-mooney | you need to have the neutron port uuid if the agent is in debug mode it will print the uuid of the port it is processign at differnt times | |
| 12:15:02 | ygk_12345 | ok let me check | |
| 12:15:14 | sean-k-mooney | what you are looking for is if the port is listed in teh added or deleted list of ports around teh time of the events | |
| 12:15:33 | sean-k-mooney | that would indicate the port being removed or added to the ovs bridge | |
| 12:15:50 | sean-k-mooney | simialrly if the admin state on the newon port was changed it would also show up there | |
| 12:16:38 | sean-k-mooney | ygk_12345: the other thing that you cold check before going into the agent logs is the qemu instance log | |
| 12:16:48 | ygk_12345 | oh ok | |
| 12:16:50 | sean-k-mooney | and confirm that there are no restarts | |
| 12:17:31 | sean-k-mooney | ygk_12345: do you enabel the watchdog or qemu guest agent in teh image that is affected | |
| 12:18:00 | ygk_12345 | no idea | |
| 12:18:15 | sean-k-mooney | they would be listed in the image metadata in glance | |
| 12:20:00 | sean-k-mooney | ygk_12345: those event typically mean 1 of two things. eighet the tap device was removed and added to ovs, or the port was updated (host-id or admin status) | |
| 12:20:10 | ygk_12345 | i dont think they are set for that image | |
| 12:20:17 | sean-k-mooney | ok | |
| 12:20:31 | sean-k-mooney | this sound like its not a nova issue for what its worth | |
| 12:21:41 | sean-k-mooney | but check the qemu log to confim there is not reboot of the instace when it happened. if so then you need to look at the l2 agent logs if that does not show anyting then you need to check the dhcp agent and neutron server logs for the port uuid | |
| 12:21:50 | sean-k-mooney | to track down why the event was sent | |
| 12:26:33 | ygk_12345 | ok | |
| 12:28:19 | ygk_12345 | sean-k-mooney even when the vnics have disappeared, those ports are being shown as active in nova | |
| 12:28:28 | ygk_12345 | how can this be ? | |
| 12:31:10 | sean-k-mooney | without logs i cant really help you resolve this. can you file a bug and attach some logs specificly the nova compute and l2 agent logs for aroudn the time when this happened | |
| 12:31:49 | ygk_12345 | ok | |
| 12:32:02 | sean-k-mooney | what your describing does not really match to any known broken behviaor that comes to mind | |
| 12:32:43 | sean-k-mooney | the only way to revmoe an interface form a vm that is runnign other then a nova interface detach which is not present in the event logs | |
| 12:32:50 | sean-k-mooney | is to delete the neutron port | |
| 12:33:05 | sean-k-mooney | that will send a network-vif-deleted event to nova | |
| 12:33:19 | sean-k-mooney | but presuably you did not delete the nuetorn port | |
| 12:33:32 | sean-k-mooney | so im not aware of any code path that could result in this | |
| 12:33:52 | ygk_12345 | strange | |
| 12:34:03 | sean-k-mooney | the info cache currption issue requires a hard reboot for the instance to be lost or a move operations | |
| 12:44:59 | ygk_12345 | sean-k-mooney when the ping is unreachable for the vm , I dont see any messages for that port in the neutron-server except when I do a manual reboot | |
| 12:46:03 | lyarwood | ubuntu-- | |
| 12:46:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: test_evacuate.sh: Support libvirt-bin and libvirtd systemd services https://review.opendev.org/752650 | |
| 12:46:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | |
| 12:46:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982 | |
| 12:46:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_NATIVE_TLS_VERSION https://review.opendev.org/746983 | |
| 12:46:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_BETTER_SIGKILL_HANDLING https://review.opendev.org/746984 | |
| 12:46:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985 | |
| 12:46:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986 | |
| 12:47:09 | stephenfin | lyarwood: https://media.tenor.com/images/a53dc07bf0a6f8822f16c0760299f915/tenor.gif | |
| 12:49:13 | gibi | :) | |
| 12:51:35 | lyarwood | yup sorry all | |
| 12:53:56 | lyarwood | back this evening to clean up any fallout caused by all of this | |
| 12:56:33 | gibi | ack | |
| 12:58:54 | ygk_12345 | sean-k-mooney i have posted this bug https://bugs.launchpad.net/nova/+bug/1896226 | |
| 12:58:54 | openstack | Launchpad bug 1896226 in OpenStack Compute (nova) "The vnics are disappearing in the vm" [Undecided,New] | |
| 13:02:59 | kashyap | lyarwood: Hey, so do you know how many CPU cores did that reproducer host had? | |
| 13:03:26 | kashyap | lyarwood: I'm asking because, depending on that I may need to fork my CPU workload to spread across multiple (CPU) cores | |