Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-20
13:27:45 bauzas ta
13:31:26 gibi francoisp, lyarwood: +A-d the cinder retry
13:32:25 francoisp ok great, thank you gibi
13:32:48 lyarwood yup thanks gibi
13:37:43 openstackgerrit jayaditya gupta proposed openstack/nova master: Support for --force flag for nova-manage placement heal_allocations command https://review.opendev.org/715395
13:47:29 nightmare_unreal mriedem: thanks for the review :) As you have suggested I have made changes accordingly but I am still facing 1 issue. It seems the allocated ram ( bogus ram) won't change if you call heal allocation with force flag or without force flag :/ I have added comments for it https://review.opendev.org/#/c/715395/10
13:47:30 nightmare_unreal Thanks
14:05:25 gmann melwitt: stephenfin gibi seems like johnthetubaguy is not online. how we should proceed on these last bits to merge as 23rd is hard string freeze - https://review.opendev.org/#/q/topic:bp/policy-defaults-refresh+status:open
14:06:29 gibi gmann: I'm on a call I will ping back in an hour
14:06:48 gibi but overall I can try to spend 1 hour on those today
14:08:41 gmann gibi: thanks
14:26:55 openstackgerrit sean mooney proposed openstack/nova-specs master: move implemented spec for train https://review.opendev.org/706276
14:27:50 openstackgerrit sean mooney proposed openstack/nova-specs master: move implemented spec for train https://review.opendev.org/706276
14:29:53 mriedem nightmare_unreal: that's the point of the feature, correct? if it's not working you're going to need to debug it.
14:30:05 mriedem but that's why i asked for that kind of test
14:31:42 nightmare_unreal yeaah
14:35:23 openstackgerrit sean mooney proposed openstack/nova-specs master: move implemented spec for ussuri https://review.opendev.org/721278
14:40:53 kashyap lyarwood: Hey, do you have the reproducer for that 'q35' thing on Ubuntu?
14:41:26 kashyap Ah, you updated this morning
14:42:43 kashyap Okay, it looks like a nudge. (Because, I don't see any 'diff' b/n 3..4: https://review.opendev.org/#/c/708701/3..4/.zuul.yaml)
15:02:15 lyarwood kashyap: sorry was hacking away on something downstream
15:02:25 lyarwood kashyap: I've just rebased that today to see if it still reproduces
15:02:36 lyarwood kashyap: I don't have anything written up, I only manually reproduced it before.
15:02:37 kashyap lyarwood: No problem; I asked on the change
15:02:44 kashyap (I don't count on instant responses :))
15:33:43 gibi gmann: I'm +2 on the remaining policy changes.
15:48:07 gmann gibi: thanks. should i revise this as per comment if you are online and can re+2 - https://review.opendev.org/#/c/720129/7
16:00:58 gibi gmann: if you respin it then I can re +2
16:01:13 gibi gmann: but I might be slower during my evening
16:01:26 gmann gibi: cool. dojng
16:01:28 gmann doing
16:01:35 gibi cool
16:10:05 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh https://review.opendev.org/720129
16:10:43 gmann gibi: ^^
16:10:52 gibi looking
16:12:05 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh https://review.opendev.org/720129
16:12:26 gibi just fixed a missing verb in the same sentence ^^
16:12:28 gibi but +2
16:19:00 gmann thanks
17:17:41 stephenfin gibi: If you can hit https://review.opendev.org/717884 https://review.opendev.org/719100 and https://review.opendev.org/720042 then we're done with policy, afaict
17:18:09 openstackgerrit Merged openstack/nova stable/train: libvirt: Calculate disk_over_committed for raw instances https://review.opendev.org/718964
17:36:34 openstackgerrit Merged openstack/nova master: Add retry to cinder API calls related to volume detach https://review.opendev.org/669674
18:06:48 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix the followup comment of policy doc https://review.opendev.org/721322
18:18:43 artom gmann, wait, are we trying to land that policy doc before RC?
18:19:03 artom Didn't mean to sabotage that - but then my -1 carries less weight than gibi's +2, so :)
18:23:07 gmann artom: yeah before RC. I am fixing your comment in follow up along with stephenfin comments -https://review.opendev.org/721322
18:24:00 openstackgerrit Merged openstack/nova master: Introduce scope_types in servers attributes Policies https://review.opendev.org/719729
18:33:57 openstackgerrit Merged openstack/nova master: Add new default roles in servers attributes policies https://review.opendev.org/719730
18:34:03 openstackgerrit Merged openstack/nova master: Add test coverage of existing remaining servers policies https://review.opendev.org/720104
18:34:11 openstackgerrit Merged openstack/nova master: Introduce scope_types in remaining servers Policies https://review.opendev.org/720106
18:34:19 openstackgerrit Merged openstack/nova master: Add new default roles in remaining servers policies https://review.opendev.org/720116
18:54:06 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix the followup comment of policy doc https://review.opendev.org/721322
18:54:43 gmann stephenfin: artom i fixed the policy doc comment in followup, please check - https://review.opendev.org/#/c/721322/
19:13:54 openstackgerrit Merged openstack/nova master: Fix follow up comments on policy work https://review.opendev.org/717835
19:17:01 openstackgerrit Merged openstack/nova master: Pass allocations to virt drivers when resizing https://review.opendev.org/589085
19:56:48 mnaser sean-k-mooney: sorry to ping you here but i don't know what other channelt o find you in -- happy to hear your thoughts on https://review.opendev.org/#/c/720107/3 :)
19:57:53 sean-k-mooney mnaser: im usally in nova,neuton,plamcent,kolla and somethim infra or oslo
19:58:08 sean-k-mooney but ya ill take a look now
19:58:13 mnaser sean-k-mooney: fair :) whois showed a lot less than those today :P
19:58:27 sean-k-mooney whois sean-k-mooney
19:59:12 sean-k-mooney i has a few but ya so container images
19:59:15 sean-k-mooney fun
20:00:23 sean-k-mooney im not sure that its fair to describe kolla image as like system image e.g. lxc style but they are not that light weight either
20:04:30 zigo What's blocking this backport patch ? https://review.opendev.org/#/c/711233/
20:04:55 openstack Launchpad bug 1788014 in OpenStack Compute (nova) rocky "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Elod Illes (elod-illes)
20:04:55 zigo The bug https://bugs.launchpad.net/nova/+bug/1788014 is causing real life troubles and a fix would be really nice.
20:05:38 zigo We had all sorts of down time due to it, lots of head scratching until we understood what was going on...
20:08:37 melwitt elod: question for your morrow ^
22:47:20 abaindur Hello, I have a question about post copy live migration. What happens if live_migration_permit_post_copy is only set on nova compute on some hypervisors? Does it need to be the same across every host?
22:47:31 sean-k-mooney mnaser: this is my counter proposal https://review.opendev.org/#/c/720107/3/goals/proposed/container-images.rst@14
23:11:53 sean-k-mooney abaindur: i think it is based on teh source node
23:12:17 sean-k-mooney but we dont test it so it shoudl be the same on all node but it might work if its different
23:12:19 abaindur would there be any issues if we migrated from a source host that had post copy enabled, but a destination host that didnt?
23:12:48 abaindur we want to give it a shot - but only wanted to run it on a subset of hypervisors
23:14:50 abaindur sean-k-mooney: one other question about live migration: reason we are going to post-copy is because we're seeing significant downtime (15 - 30+ seconds) during live migration. Seems to always start when VM is Paused on source/Resumed on dest, then start working shortly after port-binding activate call is made, and port is plugged on the host
23:15:31 abaindur We thought that maybe giving post-copy a shot would help, since it would give us the benefit of this fix: https://opendev.org/openstack/nova/commit/1f48d3d83b4d5f6f9cd96ee06d2fc005635c1ff9
23:16:54 abaindur But are there any known issues around pre-copy live migration? Bulk of the time seems to be taken up in _post_copy_live_migration() function on the source host. For example it took 18+ seconds from statr of that function until the port-binding activate call was sent to neutron
23:18:37 abaindur sorry, not _post_copy_live_migration(). I meant _post_live_migration() function
23:22:30 sean-k-mooney abaindur: libvirt will check if the qemu and libvirt on each host support it
23:22:39 sean-k-mooney and only enable it if both do i belive
23:23:38 sean-k-mooney so in principal i dont think it would have a negitive effect just be aware that you would see different behavior migration too a host with it enabled vs migrating form a host with it enabled
23:24:23 sean-k-mooney i dont recall off the top of my head which config we check to enable it but i belive it would have a asymetric behavior as i think we only check one of them
23:25:52 sean-k-mooney abaindur: for what it si worth the port binding events shoudl work with or without post copy
23:27:12 sean-k-mooney _post_live_migration is the function that cleans up the image on the source node and finishes and work reqiured on the dest
23:28:05 sean-k-mooney if you are using a nova and neutron that do not support multiple port bining _post_live_migration_at_dest is where we will do the port binding
23:29:06 sean-k-mooney but in a nova that support neutron multiple port bindign api we will prebind the port on the souce and activate it either in responce to the live migrion even tor the start of _post_live_migration
23:29:16 abaindur yea, but we noticed that in pre-copy LV mode, the port binding activate call is sent towards the end of that function. And connectivity is disrupted when VM is Paused.
23:29:18 abaindur https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6931
23:29:45 sean-k-mooney ya so are you using ovs with the ovs firewall driver by any chance
23:29:51 abaindur and here is where the port binding activation call is invoekd: https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005
23:29:52 sean-k-mooney or are you using iptables
23:30:42 abaindur between those 2 lines of code, we noticed nova taking 18+ seconds - 2 seocnds for bolume cleanup, few more seconds for disconnection, 5 sec for get_instance_nw_info, 5 more sec for compute_utils.notify_about_instance_action, etc...
23:31:01 abaindur One the port binding activate call came, another 3-5 seconds for port to be plugged by OVS-agent
23:31:04 abaindur we're using iptables
23:31:26 sean-k-mooney on what release
23:31:33 abaindur Rocky
23:31:49 sean-k-mooney the iptable firewall should have less downtime the openvswithc
23:32:34 sean-k-mooney as we can pre plug the port and have neutron wire it up while we are waiting for migration too happen at the libvirt level.
23:33:08 sean-k-mooney when using the ovs firewall driver because libvirt recreate the ovs port it takes longer as neutron has to do it twice
23:33:14 abaindur as mentioned, we're seeing non-trivial downtime in that _post_live_migration() function on the src host, between the log "'_post_live_migration() is started.." (or when VM is Paused), and when neutron receives the port-binding /activate call at https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005

Earlier   Later