| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-20 | |||
| 14:31:42 | nightmare_unreal | yeaah | |
| 14:35:23 | openstackgerrit | sean mooney proposed openstack/nova-specs master: move implemented spec for ussuri https://review.opendev.org/721278 | |
| 14:40:53 | kashyap | lyarwood: Hey, do you have the reproducer for that 'q35' thing on Ubuntu? | |
| 14:41:26 | kashyap | Ah, you updated this morning | |
| 14:42:43 | kashyap | Okay, it looks like a nudge. (Because, I don't see any 'diff' b/n 3..4: https://review.opendev.org/#/c/708701/3..4/.zuul.yaml) | |
| 15:02:15 | lyarwood | kashyap: sorry was hacking away on something downstream | |
| 15:02:25 | lyarwood | kashyap: I've just rebased that today to see if it still reproduces | |
| 15:02:36 | lyarwood | kashyap: I don't have anything written up, I only manually reproduced it before. | |
| 15:02:37 | kashyap | lyarwood: No problem; I asked on the change | |
| 15:02:44 | kashyap | (I don't count on instant responses :)) | |
| 15:33:43 | gibi | gmann: I'm +2 on the remaining policy changes. | |
| 15:48:07 | gmann | gibi: thanks. should i revise this as per comment if you are online and can re+2 - https://review.opendev.org/#/c/720129/7 | |
| 16:00:58 | gibi | gmann: if you respin it then I can re +2 | |
| 16:01:13 | gibi | gmann: but I might be slower during my evening | |
| 16:01:26 | gmann | gibi: cool. dojng | |
| 16:01:28 | gmann | doing | |
| 16:01:35 | gibi | cool | |
| 16:10:05 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh https://review.opendev.org/720129 | |
| 16:10:43 | gmann | gibi: ^^ | |
| 16:10:52 | gibi | looking | |
| 16:12:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh https://review.opendev.org/720129 | |
| 16:12:26 | gibi | just fixed a missing verb in the same sentence ^^ | |
| 16:12:28 | gibi | but +2 | |
| 16:19:00 | gmann | thanks | |
| 17:17:41 | stephenfin | gibi: If you can hit https://review.opendev.org/717884 https://review.opendev.org/719100 and https://review.opendev.org/720042 then we're done with policy, afaict | |
| 17:18:09 | openstackgerrit | Merged openstack/nova stable/train: libvirt: Calculate disk_over_committed for raw instances https://review.opendev.org/718964 | |
| 17:36:34 | openstackgerrit | Merged openstack/nova master: Add retry to cinder API calls related to volume detach https://review.opendev.org/669674 | |
| 18:06:48 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix the followup comment of policy doc https://review.opendev.org/721322 | |
| 18:18:43 | artom | gmann, wait, are we trying to land that policy doc before RC? | |
| 18:19:03 | artom | Didn't mean to sabotage that - but then my -1 carries less weight than gibi's +2, so :) | |
| 18:23:07 | gmann | artom: yeah before RC. I am fixing your comment in follow up along with stephenfin comments -https://review.opendev.org/721322 | |
| 18:24:00 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in servers attributes Policies https://review.opendev.org/719729 | |
| 18:33:57 | openstackgerrit | Merged openstack/nova master: Add new default roles in servers attributes policies https://review.opendev.org/719730 | |
| 18:34:03 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing remaining servers policies https://review.opendev.org/720104 | |
| 18:34:11 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in remaining servers Policies https://review.opendev.org/720106 | |
| 18:34:19 | openstackgerrit | Merged openstack/nova master: Add new default roles in remaining servers policies https://review.opendev.org/720116 | |
| 18:54:06 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix the followup comment of policy doc https://review.opendev.org/721322 | |
| 18:54:43 | gmann | stephenfin: artom i fixed the policy doc comment in followup, please check - https://review.opendev.org/#/c/721322/ | |
| 19:13:54 | openstackgerrit | Merged openstack/nova master: Fix follow up comments on policy work https://review.opendev.org/717835 | |
| 19:17:01 | openstackgerrit | Merged openstack/nova master: Pass allocations to virt drivers when resizing https://review.opendev.org/589085 | |
| 19:56:48 | mnaser | sean-k-mooney: sorry to ping you here but i don't know what other channelt o find you in -- happy to hear your thoughts on https://review.opendev.org/#/c/720107/3 :) | |
| 19:57:53 | sean-k-mooney | mnaser: im usally in nova,neuton,plamcent,kolla and somethim infra or oslo | |
| 19:58:08 | sean-k-mooney | but ya ill take a look now | |
| 19:58:13 | mnaser | sean-k-mooney: fair :) whois showed a lot less than those today :P | |
| 19:58:27 | sean-k-mooney | whois sean-k-mooney | |
| 19:59:12 | sean-k-mooney | i has a few but ya so container images | |
| 19:59:15 | sean-k-mooney | fun | |
| 20:00:23 | sean-k-mooney | im not sure that its fair to describe kolla image as like system image e.g. lxc style but they are not that light weight either | |
| 20:04:30 | zigo | What's blocking this backport patch ? https://review.opendev.org/#/c/711233/ | |
| 20:04:55 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) rocky "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Elod Illes (elod-illes) | |
| 20:04:55 | zigo | The bug https://bugs.launchpad.net/nova/+bug/1788014 is causing real life troubles and a fix would be really nice. | |
| 20:05:38 | zigo | We had all sorts of down time due to it, lots of head scratching until we understood what was going on... | |
| 20:08:37 | melwitt | elod: question for your morrow ^ | |
| 22:47:20 | abaindur | Hello, I have a question about post copy live migration. What happens if live_migration_permit_post_copy is only set on nova compute on some hypervisors? Does it need to be the same across every host? | |
| 22:47:31 | sean-k-mooney | mnaser: this is my counter proposal https://review.opendev.org/#/c/720107/3/goals/proposed/container-images.rst@14 | |
| 23:11:53 | sean-k-mooney | abaindur: i think it is based on teh source node | |
| 23:12:17 | sean-k-mooney | but we dont test it so it shoudl be the same on all node but it might work if its different | |
| 23:12:19 | abaindur | would there be any issues if we migrated from a source host that had post copy enabled, but a destination host that didnt? | |
| 23:12:48 | abaindur | we want to give it a shot - but only wanted to run it on a subset of hypervisors | |
| 23:14:50 | abaindur | sean-k-mooney: one other question about live migration: reason we are going to post-copy is because we're seeing significant downtime (15 - 30+ seconds) during live migration. Seems to always start when VM is Paused on source/Resumed on dest, then start working shortly after port-binding activate call is made, and port is plugged on the host | |
| 23:15:31 | abaindur | We thought that maybe giving post-copy a shot would help, since it would give us the benefit of this fix: https://opendev.org/openstack/nova/commit/1f48d3d83b4d5f6f9cd96ee06d2fc005635c1ff9 | |
| 23:16:54 | abaindur | But are there any known issues around pre-copy live migration? Bulk of the time seems to be taken up in _post_copy_live_migration() function on the source host. For example it took 18+ seconds from statr of that function until the port-binding activate call was sent to neutron | |
| 23:18:37 | abaindur | sorry, not _post_copy_live_migration(). I meant _post_live_migration() function | |
| 23:22:30 | sean-k-mooney | abaindur: libvirt will check if the qemu and libvirt on each host support it | |
| 23:22:39 | sean-k-mooney | and only enable it if both do i belive | |
| 23:23:38 | sean-k-mooney | so in principal i dont think it would have a negitive effect just be aware that you would see different behavior migration too a host with it enabled vs migrating form a host with it enabled | |
| 23:24:23 | sean-k-mooney | i dont recall off the top of my head which config we check to enable it but i belive it would have a asymetric behavior as i think we only check one of them | |
| 23:25:52 | sean-k-mooney | abaindur: for what it si worth the port binding events shoudl work with or without post copy | |
| 23:27:12 | sean-k-mooney | _post_live_migration is the function that cleans up the image on the source node and finishes and work reqiured on the dest | |
| 23:28:05 | sean-k-mooney | if you are using a nova and neutron that do not support multiple port bining _post_live_migration_at_dest is where we will do the port binding | |
| 23:29:06 | sean-k-mooney | but in a nova that support neutron multiple port bindign api we will prebind the port on the souce and activate it either in responce to the live migrion even tor the start of _post_live_migration | |
| 23:29:16 | abaindur | yea, but we noticed that in pre-copy LV mode, the port binding activate call is sent towards the end of that function. And connectivity is disrupted when VM is Paused. | |
| 23:29:18 | abaindur | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6931 | |
| 23:29:45 | sean-k-mooney | ya so are you using ovs with the ovs firewall driver by any chance | |
| 23:29:51 | abaindur | and here is where the port binding activation call is invoekd: https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005 | |
| 23:29:52 | sean-k-mooney | or are you using iptables | |
| 23:30:42 | abaindur | between those 2 lines of code, we noticed nova taking 18+ seconds - 2 seocnds for bolume cleanup, few more seconds for disconnection, 5 sec for get_instance_nw_info, 5 more sec for compute_utils.notify_about_instance_action, etc... | |
| 23:31:01 | abaindur | One the port binding activate call came, another 3-5 seconds for port to be plugged by OVS-agent | |
| 23:31:04 | abaindur | we're using iptables | |
| 23:31:26 | sean-k-mooney | on what release | |
| 23:31:33 | abaindur | Rocky | |
| 23:31:49 | sean-k-mooney | the iptable firewall should have less downtime the openvswithc | |
| 23:32:34 | sean-k-mooney | as we can pre plug the port and have neutron wire it up while we are waiting for migration too happen at the libvirt level. | |
| 23:33:08 | sean-k-mooney | when using the ovs firewall driver because libvirt recreate the ovs port it takes longer as neutron has to do it twice | |
| 23:33:14 | abaindur | as mentioned, we're seeing non-trivial downtime in that _post_live_migration() function on the src host, between the log "'_post_live_migration() is started.." (or when VM is Paused), and when neutron receives the port-binding /activate call at https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005 | |
| 23:33:40 | sean-k-mooney | abaindur: yes but before the multiple port binding change it used to be even longer | |
| 23:34:04 | abaindur | so is that down time then to be expected or unavoidable? :( | |
| 23:34:23 | sean-k-mooney | no its avoidable | |
| 23:34:31 | sean-k-mooney | so this is the feature you are trying to use https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html | |
| 23:34:52 | sean-k-mooney | i did not think this required post-copy but let me double check | |
| 23:35:17 | abaindur | well thats why we were considering trying out post-copy - to see if it speeds up when nova makes the port binding call | |
| 23:35:30 | sean-k-mooney | ok so yes if you want the quick setup we are waiting for VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY | |
| 23:35:43 | abaindur | post-copy seems to trigger that call based on VIR_DOMAIN_EVENT states | |
| 23:36:14 | abaindur | pre-copy waits for that _post_live_migration() on the host to invoke network_api.migrate_instance_start(), which seems to take a while for us | |
| 23:36:29 | sean-k-mooney | that will allow us to activate the port bidning from the souce earlier then _post_live_migration | |
| 23:37:13 | abaindur | Right, thats why we will try post-copy mode. But was wondering if the downtime/delay we are seeing with pre-copy is expected or unavoidable? | |
| 23:37:19 | sean-k-mooney | abaindur: ya so it is a littl strange that your _post_live_migration function completion is os long | |
| 23:37:39 | sean-k-mooney | i would not expect _post_live_migration to take multiple seconds | |
| 23:38:41 | abaindur | what are downsides to post copy besides that VM needs to be rebooted if theres a live migration error? | |
| 23:39:06 | abaindur | and page faults may slow down the VM as memory needs to be copied over the network? | |