Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-20
16:00:58 gibi gmann: if you respin it then I can re +2
16:01:13 gibi gmann: but I might be slower during my evening
16:01:26 gmann gibi: cool. dojng
16:01:28 gmann doing
16:01:35 gibi cool
16:10:05 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh https://review.opendev.org/720129
16:10:43 gmann gibi: ^^
16:10:52 gibi looking
16:12:05 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh https://review.opendev.org/720129
16:12:26 gibi just fixed a missing verb in the same sentence ^^
16:12:28 gibi but +2
16:19:00 gmann thanks
17:17:41 stephenfin gibi: If you can hit https://review.opendev.org/717884 https://review.opendev.org/719100 and https://review.opendev.org/720042 then we're done with policy, afaict
17:18:09 openstackgerrit Merged openstack/nova stable/train: libvirt: Calculate disk_over_committed for raw instances https://review.opendev.org/718964
17:36:34 openstackgerrit Merged openstack/nova master: Add retry to cinder API calls related to volume detach https://review.opendev.org/669674
18:06:48 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix the followup comment of policy doc https://review.opendev.org/721322
18:18:43 artom gmann, wait, are we trying to land that policy doc before RC?
18:19:03 artom Didn't mean to sabotage that - but then my -1 carries less weight than gibi's +2, so :)
18:23:07 gmann artom: yeah before RC. I am fixing your comment in follow up along with stephenfin comments -https://review.opendev.org/721322
18:24:00 openstackgerrit Merged openstack/nova master: Introduce scope_types in servers attributes Policies https://review.opendev.org/719729
18:33:57 openstackgerrit Merged openstack/nova master: Add new default roles in servers attributes policies https://review.opendev.org/719730
18:34:03 openstackgerrit Merged openstack/nova master: Add test coverage of existing remaining servers policies https://review.opendev.org/720104
18:34:11 openstackgerrit Merged openstack/nova master: Introduce scope_types in remaining servers Policies https://review.opendev.org/720106
18:34:19 openstackgerrit Merged openstack/nova master: Add new default roles in remaining servers policies https://review.opendev.org/720116
18:54:06 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix the followup comment of policy doc https://review.opendev.org/721322
18:54:43 gmann stephenfin: artom i fixed the policy doc comment in followup, please check - https://review.opendev.org/#/c/721322/
19:13:54 openstackgerrit Merged openstack/nova master: Fix follow up comments on policy work https://review.opendev.org/717835
19:17:01 openstackgerrit Merged openstack/nova master: Pass allocations to virt drivers when resizing https://review.opendev.org/589085
19:56:48 mnaser sean-k-mooney: sorry to ping you here but i don't know what other channelt o find you in -- happy to hear your thoughts on https://review.opendev.org/#/c/720107/3 :)
19:57:53 sean-k-mooney mnaser: im usally in nova,neuton,plamcent,kolla and somethim infra or oslo
19:58:08 sean-k-mooney but ya ill take a look now
19:58:13 mnaser sean-k-mooney: fair :) whois showed a lot less than those today :P
19:58:27 sean-k-mooney whois sean-k-mooney
19:59:12 sean-k-mooney i has a few but ya so container images
19:59:15 sean-k-mooney fun
20:00:23 sean-k-mooney im not sure that its fair to describe kolla image as like system image e.g. lxc style but they are not that light weight either
20:04:30 zigo What's blocking this backport patch ? https://review.opendev.org/#/c/711233/
20:04:55 zigo The bug https://bugs.launchpad.net/nova/+bug/1788014 is causing real life troubles and a fix would be really nice.
20:04:55 openstack Launchpad bug 1788014 in OpenStack Compute (nova) rocky "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Elod Illes (elod-illes)
20:05:38 zigo We had all sorts of down time due to it, lots of head scratching until we understood what was going on...
20:08:37 melwitt elod: question for your morrow ^
22:47:20 abaindur Hello, I have a question about post copy live migration. What happens if live_migration_permit_post_copy is only set on nova compute on some hypervisors? Does it need to be the same across every host?
22:47:31 sean-k-mooney mnaser: this is my counter proposal https://review.opendev.org/#/c/720107/3/goals/proposed/container-images.rst@14
23:11:53 sean-k-mooney abaindur: i think it is based on teh source node
23:12:17 sean-k-mooney but we dont test it so it shoudl be the same on all node but it might work if its different
23:12:19 abaindur would there be any issues if we migrated from a source host that had post copy enabled, but a destination host that didnt?
23:12:48 abaindur we want to give it a shot - but only wanted to run it on a subset of hypervisors
23:14:50 abaindur sean-k-mooney: one other question about live migration: reason we are going to post-copy is because we're seeing significant downtime (15 - 30+ seconds) during live migration. Seems to always start when VM is Paused on source/Resumed on dest, then start working shortly after port-binding activate call is made, and port is plugged on the host
23:15:31 abaindur We thought that maybe giving post-copy a shot would help, since it would give us the benefit of this fix: https://opendev.org/openstack/nova/commit/1f48d3d83b4d5f6f9cd96ee06d2fc005635c1ff9
23:16:54 abaindur But are there any known issues around pre-copy live migration? Bulk of the time seems to be taken up in _post_copy_live_migration() function on the source host. For example it took 18+ seconds from statr of that function until the port-binding activate call was sent to neutron
23:18:37 abaindur sorry, not _post_copy_live_migration(). I meant _post_live_migration() function
23:22:30 sean-k-mooney abaindur: libvirt will check if the qemu and libvirt on each host support it
23:22:39 sean-k-mooney and only enable it if both do i belive
23:23:38 sean-k-mooney so in principal i dont think it would have a negitive effect just be aware that you would see different behavior migration too a host with it enabled vs migrating form a host with it enabled
23:24:23 sean-k-mooney i dont recall off the top of my head which config we check to enable it but i belive it would have a asymetric behavior as i think we only check one of them
23:25:52 sean-k-mooney abaindur: for what it si worth the port binding events shoudl work with or without post copy
23:27:12 sean-k-mooney _post_live_migration is the function that cleans up the image on the source node and finishes and work reqiured on the dest
23:28:05 sean-k-mooney if you are using a nova and neutron that do not support multiple port bining _post_live_migration_at_dest is where we will do the port binding
23:29:06 sean-k-mooney but in a nova that support neutron multiple port bindign api we will prebind the port on the souce and activate it either in responce to the live migrion even tor the start of _post_live_migration
23:29:16 abaindur yea, but we noticed that in pre-copy LV mode, the port binding activate call is sent towards the end of that function. And connectivity is disrupted when VM is Paused.
23:29:18 abaindur https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6931
23:29:45 sean-k-mooney ya so are you using ovs with the ovs firewall driver by any chance
23:29:51 abaindur and here is where the port binding activation call is invoekd: https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005
23:29:52 sean-k-mooney or are you using iptables
23:30:42 abaindur between those 2 lines of code, we noticed nova taking 18+ seconds - 2 seocnds for bolume cleanup, few more seconds for disconnection, 5 sec for get_instance_nw_info, 5 more sec for compute_utils.notify_about_instance_action, etc...
23:31:01 abaindur One the port binding activate call came, another 3-5 seconds for port to be plugged by OVS-agent
23:31:04 abaindur we're using iptables
23:31:26 sean-k-mooney on what release
23:31:33 abaindur Rocky
23:31:49 sean-k-mooney the iptable firewall should have less downtime the openvswithc
23:32:34 sean-k-mooney as we can pre plug the port and have neutron wire it up while we are waiting for migration too happen at the libvirt level.
23:33:08 sean-k-mooney when using the ovs firewall driver because libvirt recreate the ovs port it takes longer as neutron has to do it twice
23:33:14 abaindur as mentioned, we're seeing non-trivial downtime in that _post_live_migration() function on the src host, between the log "'_post_live_migration() is started.." (or when VM is Paused), and when neutron receives the port-binding /activate call at https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005
23:33:40 sean-k-mooney abaindur: yes but before the multiple port binding change it used to be even longer
23:34:04 abaindur so is that down time then to be expected or unavoidable? :(
23:34:23 sean-k-mooney no its avoidable
23:34:31 sean-k-mooney so this is the feature you are trying to use https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html
23:34:52 sean-k-mooney i did not think this required post-copy but let me double check
23:35:17 abaindur well thats why we were considering trying out post-copy - to see if it speeds up when nova makes the port binding call
23:35:30 sean-k-mooney ok so yes if you want the quick setup we are waiting for VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY
23:35:43 abaindur post-copy seems to trigger that call based on VIR_DOMAIN_EVENT states
23:36:14 abaindur pre-copy waits for that _post_live_migration() on the host to invoke network_api.migrate_instance_start(), which seems to take a while for us
23:36:29 sean-k-mooney that will allow us to activate the port bidning from the souce earlier then _post_live_migration
23:37:13 abaindur Right, thats why we will try post-copy mode. But was wondering if the downtime/delay we are seeing with pre-copy is expected or unavoidable?
23:37:19 sean-k-mooney abaindur: ya so it is a littl strange that your _post_live_migration function completion is os long
23:37:39 sean-k-mooney i would not expect _post_live_migration to take multiple seconds
23:38:41 abaindur what are downsides to post copy besides that VM needs to be rebooted if theres a live migration error?
23:39:06 abaindur and page faults may slow down the VM as memory needs to be copied over the network?
23:39:17 sean-k-mooney that is the main one. if there is a network outage while its still in post copy phase then the vm will crash
23:39:35 sean-k-mooney yes page falts acroos the network might beut all write happen locally
23:39:47 sean-k-mooney so the vm will only pause if it need to read un copied data
23:40:05 sean-k-mooney and once its copied local update to that will happen to the dest memmory
23:40:19 sean-k-mooney as you pointed out we are activating the port here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005
23:40:54 sean-k-mooney which happen near the star of the function so the dely is likely related to cinder performance
23:41:10 sean-k-mooney abaindur: have you tried live migrating vms that dont have cinder volumes
23:41:28 abaindur yea, we tried VMs both volume and ephemeral based
23:41:42 sean-k-mooney did you see the same delay?
23:42:03 abaindur pretty much
23:42:24 abaindur timed the volume code with some logs of our own, heres what we observed:
23:42:36 abaindur 1.913 seconds for _get_instance_block_device_info and self.driver.post_live_migration

Earlier   Later