| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-25 | |||
| 11:54:06 | sean-k-mooney | stephenfin: i forget tox -e py3 worked | |
| 12:03:36 | openstack | bug 1880389 in OpenStack Compute (nova) "lost net connection when live migration" [Undecided,New] https://launchpad.net/bugs/1880389 | |
| 12:03:36 | gibi | sean-k-mooney: I know that you have a better view than me regarding network outage during live migration. there is a new bug 1880389 reported about it. Could you check it if you have time? | |
| 12:09:41 | sean-k-mooney | sure i can take a look shortly just grabing coffee but i have it open for when i get back | |
| 12:10:23 | sean-k-mooney | also without looking is sure bad things proably happen if there is a network outage while live migrating | |
| 12:16:18 | sean-k-mooney | ok so the issue is with the vm losing network connectivyt not a network partion at the host level causeing the migration to fail in weird ways which is less scary | |
| 12:18:19 | openstack | Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] | |
| 12:18:19 | sean-k-mooney | gibi: i need to dig deaper but its proably a duplicate of https://bugs.launchpad.net/neutron/+bug/1815989 which is cause by the root cause of https://bugs.launchpad.net/neutron/+bug/1734320 | |
| 12:18:20 | openstack | Launchpad bug 1734320 in OpenStack Compute (nova) "Eavesdropping private traffic" [Undecided,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 12:19:05 | bauzas | stephenfin: ack will look | |
| 12:19:19 | sean-k-mooney | gibi: https://review.opendev.org/#/c/724386/ is a potential workaround which some have said fixes the issue | |
| 12:19:48 | sean-k-mooney | gibi: this is how i actully wanted to fix it https://review.opendev.org/#/q/status:open++branch:master+topic:bug/1734320 | |
| 12:20:14 | sean-k-mooney | that said they are using linux bridge so it could be something else | |
| 12:22:00 | sean-k-mooney | "the ping was broken down. and the ping package from the physic switch was still set to the source." | |
| 12:22:28 | sean-k-mooney | so they were observing that pings are switched to the souce host which implies the mac learning frames were lost | |
| 12:22:55 | openstack | bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] https://launchpad.net/bugs/1815989 | |
| 12:22:55 | sean-k-mooney | which is bug: #1815989 | |
| 12:23:14 | sean-k-mooney | although in this case they are reportign the same behavior for linux bridge | |
| 12:23:53 | sean-k-mooney | ah last paragraph "And I found the time of 5 RARP from VM is before the time of the vlan sub interface insert to the linuxbridge bridge when live migration. maybe this will help us to make sure the problem I found." | |
| 12:24:19 | sean-k-mooney | so yes its the same underlying issue | |
| 12:25:19 | sean-k-mooney | the issue happens in the ovs case when libvirt recreated the port that os-vif has allready created resulting in a race with the neutron l2 agent | |
| 12:25:38 | sean-k-mooney | i did not think we had the same race for linux bridge but ill take a look | |
| 12:26:04 | gibi | sean-k-mooney: thanks | |
| 12:48:15 | sean-k-mooney | so we use interface type bridge for linux bridge which is not very surpising | |
| 12:48:18 | sean-k-mooney | <interface type='bridge'> | |
| 12:48:20 | sean-k-mooney | <source bridge='br1'/> | |
| 12:48:22 | sean-k-mooney | <target dev='vnet7'/> | |
| 12:48:24 | sean-k-mooney | <mac address="00:11:22:33:44:55"/> | |
| 12:48:26 | sean-k-mooney | </interface> | |
| 12:48:48 | sean-k-mooney | and se the target, mac and source to the values provided by neutron | |
| 12:49:41 | sean-k-mooney | before the vm is spawned on the destination host we invoke os-vif to ensure that the bridge exists | |
| 12:50:29 | sean-k-mooney | so when we define the libvirt domian there should be nothing for libvirt to do it should be alredy configured | |
| 12:51:06 | sean-k-mooney | this is all os-vif does https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L97-L112 | |
| 12:52:45 | openstackgerrit | Merged openstack/os-vif master: Remove .testr.conf https://review.opendev.org/729156 | |
| 12:55:53 | sean-k-mooney | i should proably go clean up that code since nova net is now dead and we should not really be doing any ip filtering in os-vif even in neutron is not anymore since that was really for nova net | |
| 13:00:01 | gibi | sean-k-mooney: thanks for triageing that bug | |
| 13:05:28 | sean-k-mooney | im debating if im goint to try and repoduce it or not | |
| 13:06:18 | sean-k-mooney | i would need to check the libvirt sorce code but i dont think it should have the same race | |
| 13:06:32 | sean-k-mooney | it might but i did not think it should | |
| 13:32:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: virt: Add 'context', drop 'network_info' parameters for 'unrescue' https://review.opendev.org/730382 | |
| 13:32:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 13:32:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 13:51:11 | openstackgerrit | sean mooney proposed openstack/os-vif master: update tox envs and support pdf docs https://review.opendev.org/728037 | |
| 14:02:11 | sean-k-mooney | stephenfin: hopefully you are more happy with ^ | |
| 14:02:49 | sean-k-mooney | but also that is more a merge when convenient patch. as long as we merge it before m3 (perferably m1) its fine | |
| 14:02:53 | sean-k-mooney | so no rush | |
| 14:03:22 | stephenfin | better but still not correct :) | |
| 14:04:26 | sean-k-mooney | hehe ok im goint to update my osc patch for evacuate ill take a look when that is done | |
| 14:09:02 | sean-k-mooney | just looked. ya ill do some of the chagne in a follow up and adress some in that patch. ill push both patches togeter this evening or tommorow | |
| 14:09:25 | sean-k-mooney | that patch is getting a bit long but the change you suggest make sense | |
| 14:18:52 | sean-k-mooney | 'Wait for instance evacuation to complete' sound less worng then 'Wait for evacuation to complete' right? | |
| 14:20:01 | stephenfin | The operation affects multiple instances, right? | |
| 14:20:10 | sean-k-mooney | no | |
| 14:20:13 | sean-k-mooney | only one instance | |
| 14:20:21 | stephenfin | then the first one is better, yes | |
| 14:20:28 | sean-k-mooney | im porting evacuate not host-evacuate | |
| 14:20:48 | stephenfin | although I don't know if osc uses the term "instance" | |
| 14:20:59 | sean-k-mooney | host-evacuate it the clinet side command that enumarate the instance on the host and evacuates all of them | |
| 14:21:20 | sean-k-mooney | i think it might use server but that is kind of wronge | |
| 14:21:23 | sean-k-mooney | i could use server | |
| 14:22:02 | sean-k-mooney | server is proably more inline with the other messages in teh file so ill use that | |
| 14:33:02 | openstackgerrit | Merged openstack/python-novaclient master: Remove Babel requirement https://review.opendev.org/723139 | |
| 15:37:42 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Make Feature Liaison optional https://review.opendev.org/730638 | |
| 16:57:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 16:57:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: virt: Add 'context', drop 'network_info' parameters for 'unrescue' https://review.opendev.org/730382 | |
| 16:57:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 23:28:38 | openstackgerrit | Merged openstack/nova stable/queens: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/726864 | |
| #openstack-nova - 2020-05-26 | |||
| 01:10:02 | openstackgerrit | Wenping Song proposed openstack/nova master: WIP: Cyborg suspend/resume support https://review.opendev.org/729945 | |
| 02:47:15 | openstackgerrit | Merged openstack/nova master: docs: Resolve issue with deprecated extra specs https://review.opendev.org/726315 | |
| 04:08:56 | openstackgerrit | Merged openstack/nova master: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/722187 | |
| 05:14:50 | tacco | ho everyone. Have a mostly clean setup. But now i have a problem with downsizing of a VM. | |
| 05:15:28 | tacco | i cannot apply a smaller flavor anymore, but there is still a bunch of space on HVs and it is also allowed to migrate to the same host. | |
| 05:15:34 | tacco | have no clue what is broken in here. | |
| 05:15:41 | tacco | Will the RAM allocated two times? | |
| 05:15:59 | tacco | because it looks as it would be the RAM that causes this issue. | |
| 05:16:04 | tacco | any ideas? | |
| 06:55:31 | gibi | good morning nova | |
| 07:09:38 | bauzas | good morning Nova | |
| 07:10:11 | openstackgerrit | Wenping Song proposed openstack/nova master: WIP: Cyborg suspend/resume support https://review.opendev.org/729945 | |
| 07:20:59 | slaweq | gibi: bauzas: good morning nova (from neutron) ;) | |
| 07:24:48 | gibi | slaweq: :) | |
| 07:42:00 | tacco | morning everyone. :) | |
| 07:42:52 | tacco | anyone has a idea why placement shows targets for possible placement of a large flavor. but... downsize a vm to a smaler flavor is not possible? any hints? tips and tricks? | |
| 07:59:30 | gibi | tacco: during resize there is doubled up allocation. The source host allocation is held by the migration_uuid and the dest host allocation held by the instance_uuid. If you resize to same host then both allocation is against the same host | |
| 08:00:51 | openstack | Launchpad bug 1609193 in OpenStack Compute (nova) "resize error on the same current host with enough vcpu resource" [Wishlist,Confirmed] | |
| 08:00:51 | gibi | tacco: https://bugs.launchpad.net/nova/+bug/1609193 | |
| 08:09:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Make Feature Liaison optional https://review.opendev.org/730638 | |
| 08:29:24 | tacco | gibi: thanks. was thinking about shelving the instance and try to set a new flavor. because even shutdown dosn't fixed the problem. | |
| 08:29:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Make Feature Liaison optional https://review.opendev.org/730638 | |
| 08:37:37 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Make Feature Liaison optional https://review.opendev.org/730638 | |
| 08:41:13 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: stable-only: skip volume backup tests in cellsv1 job https://review.opendev.org/727147 | |
| 08:47:06 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Make devstack-plugin-ceph-tempest-py3 a voting check job again https://review.opendev.org/730320 | |
| 08:50:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: get_ksa_adapter: nix by-service-type confgrp hack https://review.opendev.org/664689 | |
| 09:37:22 | masterpe | hi, I don't know if this question is in the right place but, when I create an instances with "openstack server create" I get an adminPass, how does cloud-init pick this up in a Ubuntu instance? | |
| 09:45:55 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/578846 | |
| 09:45:55 | openstack | bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Lee Yarwood (lyarwood) | |
| 09:45:55 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Add functional test for bug 1550919 https://review.opendev.org/631294 | |
| 10:27:22 | lyarwood | stephenfin: stupid question, do we have docs for the custom sphinx extenstions we have for referencing oslo config etc? | |
| 10:27:56 | stephenfin | lyarwood: https://docs.openstack.org/oslo.config/latest/reference/sphinxext.html | |
| 10:28:40 | lyarwood | ah thanks | |