| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-18 | |||
| 12:28:19 | ygk_12345 | sean-k-mooney even when the vnics have disappeared, those ports are being shown as active in nova | |
| 12:28:28 | ygk_12345 | how can this be ? | |
| 12:31:10 | sean-k-mooney | without logs i cant really help you resolve this. can you file a bug and attach some logs specificly the nova compute and l2 agent logs for aroudn the time when this happened | |
| 12:31:49 | ygk_12345 | ok | |
| 12:32:02 | sean-k-mooney | what your describing does not really match to any known broken behviaor that comes to mind | |
| 12:32:43 | sean-k-mooney | the only way to revmoe an interface form a vm that is runnign other then a nova interface detach which is not present in the event logs | |
| 12:32:50 | sean-k-mooney | is to delete the neutron port | |
| 12:33:05 | sean-k-mooney | that will send a network-vif-deleted event to nova | |
| 12:33:19 | sean-k-mooney | but presuably you did not delete the nuetorn port | |
| 12:33:32 | sean-k-mooney | so im not aware of any code path that could result in this | |
| 12:33:52 | ygk_12345 | strange | |
| 12:34:03 | sean-k-mooney | the info cache currption issue requires a hard reboot for the instance to be lost or a move operations | |
| 12:44:59 | ygk_12345 | sean-k-mooney when the ping is unreachable for the vm , I dont see any messages for that port in the neutron-server except when I do a manual reboot | |
| 12:46:03 | lyarwood | ubuntu-- | |
| 12:46:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: test_evacuate.sh: Support libvirt-bin and libvirtd systemd services https://review.opendev.org/752650 | |
| 12:46:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | |
| 12:46:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982 | |
| 12:46:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_NATIVE_TLS_VERSION https://review.opendev.org/746983 | |
| 12:46:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_BETTER_SIGKILL_HANDLING https://review.opendev.org/746984 | |
| 12:46:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985 | |
| 12:46:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986 | |
| 12:47:09 | stephenfin | lyarwood: https://media.tenor.com/images/a53dc07bf0a6f8822f16c0760299f915/tenor.gif | |
| 12:49:13 | gibi | :) | |
| 12:51:35 | lyarwood | yup sorry all | |
| 12:53:56 | lyarwood | back this evening to clean up any fallout caused by all of this | |
| 12:56:33 | gibi | ack | |
| 12:58:54 | ygk_12345 | sean-k-mooney i have posted this bug https://bugs.launchpad.net/nova/+bug/1896226 | |
| 12:58:54 | openstack | Launchpad bug 1896226 in OpenStack Compute (nova) "The vnics are disappearing in the vm" [Undecided,New] | |
| 13:02:59 | kashyap | lyarwood: Hey, so do you know how many CPU cores did that reproducer host had? | |
| 13:03:26 | kashyap | lyarwood: I'm asking because, depending on that I may need to fork my CPU workload to spread across multiple (CPU) cores | |
| 13:05:10 | kashyap | Meanwhile ... lovely: I just created a fresh F32 VM on Ubuntu Focal, and the bootloader is filling up little yellow squares on my screen. | |
| 13:10:15 | gibi | kashyap: I think it is 8 https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/zuul-info/host-info.controller.yaml#440 | |
| 13:11:01 | gibi | kashyap: at least this is the CI job run linked to the bug report | |
| 13:17:50 | kashyap | gibi: So that's the "host VM" where Tempest is running, yeaH? | |
| 13:17:50 | bauzas | gibi: thanks | |
| 13:18:00 | kashyap | gibi: I'm trying to recreate the _exact_ damn setup :D | |
| 13:18:03 | bauzas | (for the TC ping) | |
| 13:18:46 | gibi | kashyap: yes I think that is the "host" where tempest runs | |
| 13:18:58 | sean-k-mooney | kashyap: it has 8 i think | |
| 13:19:09 | kashyap | gibi: Okay, 8 vCPUs, and 7599 MB of memory | |
| 13:19:22 | kashyap | gibi: And do you see what's the disk size? | |
| 13:19:30 | sean-k-mooney | yep ist my small flavor | |
| 13:20:12 | kashyap | sean-k-mooney: The disk size is ~16GB? | |
| 13:20:13 | gibi | size: 80.00 GB | |
| 13:20:21 | kashyap | gibi: Oh, right | |
| 13:20:28 | gibi | https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/zuul-info/host-info.controller.yaml#259-293 | |
| 13:20:39 | kashyap | Okay; I need to tweak the setup so that I'm also doing QEMU-on-KVM nested and not KVM-on-KVM nested | |
| 13:20:42 | kashyap | Thanks; /me back in a few | |
| 13:20:55 | gibi | thanks kashyap | |
| 13:22:20 | openstackgerrit | Merged openstack/nova master: releasenote: Add known issue for bug #1894804 https://review.opendev.org/752654 | |
| 13:22:20 | openstack | bug 1894804 in qemu (Ubuntu) "Second DEVICE_DELETED event missing during virtio-blk disk device detach" [Undecided,New] https://launchpad.net/bugs/1894804 | |
| 13:33:44 | sean-k-mooney | kashyap: you dont need 80G of disk by the way 25-40G is typeically more then enough | |
| 13:38:55 | kashyap | sean-k-mooney: Yeah, I have something like that anyway | |
| 13:40:21 | kashyap | sean-k-mooney: Do we have the full guest command-line of the Focal VM "host" upstream is using? | |
| 13:40:23 | sean-k-mooney | if you want an account on my home cloud by the way let me know and i can create one for you | |
| 13:40:46 | sean-k-mooney | no bug its just an openstack vm | |
| 13:41:27 | sean-k-mooney | with 8G of ram and 8 cpus and 80G fo disk | |
| 13:41:42 | sean-k-mooney | im not sure that really matters too much | |
| 13:42:02 | kashyap | sean-k-mooney: Okay; so what you gave me suffices | |
| 13:42:26 | kashyap | sean-k-mooney: For now, stable access (and it _is_ stable) to this VM is sufficient :) Thx! | |
| 13:42:29 | sean-k-mooney | ya my small flaovr is basicaly a proxy for the ci vms | |
| 13:43:19 | sean-k-mooney | except i add hw:cpu_sockets=2 hw:cpu_threads=2 hw:numa_nodes=2 and hw:mem_page_size=large | |
| 13:43:52 | sean-k-mooney | that should not change the behvior in this case but just an fyi | |
| 13:44:49 | kashyap | sean-k-mooney: One more: what's the CirrOS version being run here? | |
| 13:45:03 | kashyap | "here" as in, in the Tempest env | |
| 13:45:06 | sean-k-mooney | the default one form devstack so like 4.2 ish | |
| 13:45:23 | kashyap | I'd like to know the exact version, although it shouldn't matter _that_ much | |
| 13:45:25 | sean-k-mooney | 0.5.1 | |
| 13:45:32 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/stackrc#L670 | |
| 13:45:42 | kashyap | Excellent | |
| 13:46:13 | sean-k-mooney | it hasnt been updated in 7 months https://github.com/openstack/devstack/commit/7a0fa4fd9e5db7253fee0820fc002703d43bca3c so that should be what its using | |
| 13:46:32 | sean-k-mooney | the fine i think shoudl still be in /opt/stack/data i think | |
| 14:08:19 | kashyap | Also, annoyingly enough, CirrOS from 0.4.0 onwards, it's only initrd/kernel, so I can't use 'guestfish' to look around in it: https://paste.centos.org/view/10bee1e1 | |
| 14:21:36 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/ussuri: tests: Add regression test for bug 1894966 https://review.opendev.org/752371 | |
| 14:21:36 | openstack | bug 1894966 in OpenStack Compute (nova) ussuri "Create servergroup failed with unexpected error" [Undecided,Confirmed] https://launchpad.net/bugs/1894966 | |
| 14:21:37 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/ussuri: api: Set min, maxItems for server_group.policies field https://review.opendev.org/752702 | |
| 14:31:58 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: tests: Add regression test for bug 1894966 https://review.opendev.org/752706 | |
| 14:31:58 | openstack | bug 1894966 in OpenStack Compute (nova) ussuri "Create servergroup failed with unexpected error" [Undecided,In progress] https://launchpad.net/bugs/1894966 - Assigned to Stephen Finucane (stephenfinucane) | |
| 14:31:59 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: api: Set min, maxItems for server_group.policies field https://review.opendev.org/752707 | |
| 14:46:50 | mnaser | i'm curious as to why nova needs to ssh for cold migrations if using shared storage | |
| 14:47:03 | mnaser | maybe no one just worked on that test path or? | |
| 14:47:26 | mnaser | aka if shared storage: skip ssh | |
| 14:54:24 | gibi | mnaser: I might be wrong but I remember nova detects that the instance is on share path by checking that the same file exists on both the source and the dest | |
| 14:54:44 | mnaser | gibi: yeah i think that might be the only reason it still does it | |
| 14:55:04 | mnaser | i think breaking out early if the VM is using volumes only would make sense cause that would eliminate the whole ssh key thing for us | |
| 14:55:09 | mnaser | the only reason we still have it is just for that | |
| 14:55:14 | gmann | gibi: bauzas lyarwood what is consensus for Focal migration? same as we discussed in meeting? | |
| 14:55:24 | sean-k-mooney | mnaser: we could do that via an rpc i guess | |
| 14:55:56 | mnaser | sean-k-mooney: can one n-cpu rpc to another n-cpui | |
| 14:56:03 | sean-k-mooney | yes | |
| 14:56:12 | sean-k-mooney | it happens alot in move operations | |
| 14:56:26 | sean-k-mooney | well not a lot but we do rpcs between teh computes invovled | |
| 14:57:12 | gibi | mnaser: hm, only volume instances still can have config drives | |
| 14:57:29 | mnaser | gibi: right but i think we dont actually move that we just rebuild it on the target with a cold migrate | |
| 14:57:30 | mnaser | i _think_ | |
| 14:57:59 | gibi | most probably yes | |
| 14:58:33 | mnaser | https://github.com/openstack/nova/blob/cff7382fb3de4eef7aeddc34b6ae3409cbe50ea0/nova/virt/libvirt/driver.py#L8340-L8376 | |
| 14:58:33 | gibi | so, yeah we can stop doing that ssh shared storage check if the instance is only using volumes | |
| 14:58:39 | mnaser | hmm this doesnt seem to use ssh | |
| 15:00:39 | mnaser | i cant seem to honestly find the code that does the ssh to remote host.. | |