| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-09 | |||
| 16:49:07 | mriedem | we cleanup after ourselves for ports but not volumes | |
| 16:51:00 | lyarwood | ah right understood, should be easy enough to show in another functional test | |
| 16:51:46 | mriedem | maybe.....i'm not sure the fixture is setup for that really | |
| 16:51:57 | mriedem | devstack is probably much easier/faster to start | |
| 16:52:02 | mriedem | if you have 2 nodes... | |
| 16:57:03 | lyarwood | mriedem: I don't to hand but I'll make a note to give this a go | |
| 17:21:48 | openstackgerrit | Lee Yarwood proposed openstack/nova master: block_device: Rollback volumes to in-use on DeviceDetachFailed https://review.openstack.org/590439 | |
| 17:34:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | |
| 17:38:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | |
| 17:44:48 | openstackgerrit | Merged openstack/nova master: Update the parameter explain when updating a volume attachment https://review.openstack.org/565181 | |
| 19:11:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | |
| 19:15:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix image-defined numa claims during evacuate https://review.openstack.org/588657 | |
| 19:18:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add encrypted volume support to feature matrix docs https://review.openstack.org/570255 | |
| 19:28:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: fix GET /flavors?is_public description https://review.openstack.org/588092 | |
| 19:37:11 | prometheanfire | is https://github.com/openstack/nova/commit/ff747792b8f5aefe1bebb01bdf49dacc01353348#diff-f4019782d93a196a0d026479e6aa61b1R6938 run multiple times or only checked once? | |
| 19:37:16 | openstackgerrit | Merged openstack/nova stable/queens: Fix host validity check for live-migration https://review.openstack.org/590262 | |
| 19:37:23 | openstackgerrit | Merged openstack/nova master: [placement] api-ref: add description for 1.29 https://review.openstack.org/589407 | |
| 19:37:36 | prometheanfire | live migrations seem to be limited to 1M a sec and never increase | |
| 19:44:23 | melwitt | hm | |
| 19:45:18 | melwitt | do you know anything about that mriedem ^ | |
| 19:45:56 | mriedem | prometheanfire: linuxbridge? | |
| 19:46:23 | mriedem | https://github.com/openstack/nova/commit/ff747792b8f5aefe1bebb01bdf49dacc01353348#diff-f4019782d93a196a0d026479e6aa61b1R5380 | |
| 19:46:27 | mriedem | are you using linuxbridge i mean | |
| 19:47:46 | prometheanfire | ya, lb | |
| 19:47:56 | prometheanfire | vlan interface on the VM | |
| 19:48:35 | mriedem | well, we should be waiting on network-vif-plugged events from neutron and if we get them, we set the bw back up and resume the live migration, else we should fail the live migration | |
| 19:48:56 | mriedem | i'm assuming you have vif_plugging_timeout left at the default config of 300? | |
| 19:49:03 | prometheanfire | the migration finishes, it's just slow | |
| 19:49:10 | prometheanfire | I don't think we changed it | |
| 19:49:30 | mriedem | does it finish in under 5 minutes? | |
| 19:49:38 | prometheanfire | takes ~10 min | |
| 19:49:47 | prometheanfire | debug log | |
| 19:49:50 | prometheanfire | https://gist.githubusercontent.com/mheler/475d21b741aa58f320a456c3ac0d0f45/raw/ff76c45c3b6968b0d0514a9a7dcf478f054f6b70/gistfile1.txt | |
| 19:49:54 | mriedem | we should have either timed out and failed by then or reconfigured the guest to go back to the normal bw | |
| 19:50:42 | prometheanfire | which includes x-auth info, great | |
| 19:50:46 | mriedem | i don't see either the timeout or "VIF events received, continuing migration" messages in those logs | |
| 19:50:57 | prometheanfire | ya, either do I, which is why I'm confused | |
| 19:51:01 | melwitt | I'm not seeing that message "LOG.debug('VIF events received, continuing migration with max bandwidth configured" in your logs | |
| 19:51:16 | prometheanfire | that was the first thing I looked for | |
| 19:51:22 | mriedem | _http_log_request /openstack/venvs/nova-r16.2.2/lib/python2.7/site-packages/keystoneauth1/session.py:375 for that x-auth thing | |
| 19:51:23 | melwitt | bah, lag | |
| 19:51:44 | mriedem | besides me, sahid and dansmith are the other two that know about that change, | |
| 19:51:52 | mriedem | but are you sure you actually have that code? | |
| 19:52:02 | mriedem | would be nice if we logged something like "waiting for events" in that block | |
| 19:52:45 | prometheanfire | ya, set using b58c7f033771e3ea228e4b40c796d1bc95a087f5 from nova | |
| 19:54:04 | mriedem | prometheanfire: well your token thing isn't a problem :) https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L371 | |
| 19:54:06 | mriedem | it's redacted | |
| 19:54:28 | mriedem | prometheanfire: do you know the instance id in question here? | |
| 19:54:36 | mriedem | checking logs w/o an instance id is kind of hard | |
| 19:54:36 | prometheanfire | yes | |
| 19:54:55 | mriedem | also, | |
| 19:55:05 | mriedem | are these logs from the source or dest host? | |
| 19:55:11 | mriedem | b/c what we're looking for would be source host | |
| 19:55:20 | prometheanfire | the logs are from grepping it for c37d7489-a67b-47ea-a4f7-9323804cc552 | |
| 19:55:32 | prometheanfire | ya, source | |
| 19:56:02 | mriedem | 2018-08-09 21:20:20.557 12111 DEBUG nova.compute.manager [req-84ca4a17-0d3d-4597-91d6-f5721989dd41 143ee57edd4d4e3b9a165d375d0e7e1a a727713d2c0a4ed69b730d9cb2116af6 - default default] [instance: c37d7489-a67b-47ea-a4f7-9323804cc552] Received event network-vif-plugged-490f6f25-8b88-487c-a76b-62d16e3c0da1 external_instance_event /openstack/venvs/nova-r16.2.2/lib/python2.7/site-packages/nova/compute/manager.py:7071 | |
| 19:56:10 | mriedem | 2018-08-09 21:20:20.558 12111 DEBUG nova.compute.manager [req-84ca4a17-0d3d-4597-91d6-f5721989dd41 143ee57edd4d4e3b9a165d375d0e7e1a a727713d2c0a4ed69b730d9cb2116af6 - default default] [instance: c37d7489-a67b-47ea-a4f7-9323804cc552] No waiting events found dispatching network-vif-plugged-490f6f25-8b88-487c-a76b-62d16e3c0da1 pop_instance_event /openstack/venvs/nova-r16.2.2/lib/python2.7/site-packages/nova/compute/manager.py:36 | |
| 19:56:18 | mriedem | 2018-08-09 21:20:20.559 12111 WARNING nova.compute.manager [req-84ca4a17-0d3d-4597-91d6-f5721989dd41 143ee57edd4d4e3b9a165d375d0e7e1a a727713d2c0a4ed69b730d9cb2116af6 - default default] [instance: c37d7489-a67b-47ea-a4f7-9323804cc552] Received unexpected event network-vif-plugged-490f6f25-8b88-487c-a76b-62d16e3c0da1 for instance | |
| 19:56:33 | prometheanfire | this is a pike install btw | |
| 19:56:46 | mriedem | maybe we're getting the event before we're waiting for it? | |
| 19:56:47 | prometheanfire | but it was backported to pike, so meh | |
| 19:56:51 | prometheanfire | perhaps | |
| 19:57:11 | mriedem | we could also be getting ^ from the vif plug that happens on the dest host during pre-live migration | |
| 19:57:19 | mriedem | the events are going to go to the source host | |
| 19:57:22 | mriedem | which isn't waiting for those | |
| 19:57:29 | prometheanfire | not yet at least, ya | |
| 19:57:59 | mriedem | https://github.com/openstack/nova/commit/ff747792b8f5aefe1bebb01bdf49dacc01353348#diff-f4019782d93a196a0d026479e6aa61b1R6899 | |
| 19:58:32 | mriedem | "They are going to be # created by libvirt at the very beginning of the # live-migration process." | |
| 19:58:39 | prometheanfire | yep | |
| 19:58:42 | prometheanfire | I read that :P | |
| 19:58:42 | mriedem | that must mean plug_vifs during pre-live migration on the dest host | |
| 19:58:51 | mriedem | which triggers the event from neutron to the source host | |
| 19:58:58 | mriedem | and we're getting it before we start waiting it looks like | |
| 19:59:00 | prometheanfire | which isn't waiting yet? | |
| 19:59:02 | prometheanfire | ya | |
| 19:59:12 | mriedem | but, you should then hit this https://github.com/openstack/nova/commit/ff747792b8f5aefe1bebb01bdf49dacc01353348#diff-f4019782d93a196a0d026479e6aa61b1R6933 | |
| 19:59:19 | mriedem | and the migration should fail | |
| 19:59:26 | prometheanfire | also yes | |
| 20:00:03 | mriedem | added https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L675 in rocky but that doesn't help you here on pike | |
| 20:01:05 | prometheanfire | yarp | |
| 20:01:16 | mriedem | we can't backport that either b/c it's got rpc changes in it | |
| 20:01:41 | mriedem | so maybe the raised MigrationError isn't really doing anything? | |
| 20:02:05 | mriedem | i expect you to know exactly how all of this nova code works | |
| 20:02:14 | prometheanfire | lolol | |
| 20:02:56 | prometheanfire | ya, I'm honestly surprised that it finished, I expected it to fail | |
| 20:02:57 | mriedem | no it shoud raise up, | |
| 20:03:03 | mriedem | i was thinking we might be threaded but that's here https://github.com/openstack/nova/blob/ff747792b8f5aefe1bebb01bdf49dacc01353348/nova/virt/libvirt/driver.py#L6928 | |
| 20:03:11 | prometheanfire | maybe eventlet.timeout.Timeout isn't the actual error getting raised? | |
| 20:03:29 | openstackgerrit | Merged openstack/nova stable/queens: Update nova network info when doing rebuild for evacuate operation https://review.openstack.org/590062 | |
| 20:04:18 | prometheanfire | maybe we finish the migration before the timeout occurs? | |
| 20:04:34 | mriedem | Timeout is the right error | |
| 20:04:44 | mriedem | prometheanfire: well that's why i asked how long it took, | |
| 20:04:49 | mriedem | but the default timeout is 5 min | |
| 20:04:55 | mriedem | you said it completed in 10 min | |
| 20:05:01 | prometheanfire | went from 21:20 to 21:25 ish | |
| 20:05:16 | mriedem | oh, well that's not 10 min L( | |
| 20:05:17 | mriedem | :) | |
| 20:05:20 | prometheanfire | ya | |
| 20:05:26 | mriedem | so yeah i bet you completed before the timeout | |
| 20:06:25 | prometheanfire | 21:20:21.043 to 21:27:12.788 at least | |