| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-09 | |||
| 16:17:35 | efried | melwitt, mriedem: done. If we're not worried about getting the reno landed today, I'll happily wait for jichenjc to do it. | |
| 16:18:26 | mriedem | i'm not losing sleep over a detailed reno for the zvm driver which does a very small number of things - the docs are more important to me | |
| 16:18:30 | mriedem | the prelude mentions it | |
| 16:18:37 | mriedem | if someone wants to learn more, they can find the driver docs | |
| 16:19:29 | mriedem | the way i look at renos, if it's really detailed, it likely needs to be a doc, because release notes are a one time only thing | |
| 16:22:08 | mriedem | melwitt: should probably hold up stable releases on https://review.openstack.org/#/q/topic:bug/1784705+status:open | |
| 16:22:30 | melwitt | ok, that's helpful. fwiw, I was thinking basic detail in the reno like, what operations are supported (spawn, destroy, snapshot, get console output, power actions) | |
| 16:22:35 | melwitt | mriedem: ok, will do | |
| 16:22:39 | mriedem | anyone using ironic + ComputeCapabilitiesFilter will be hit by those | |
| 16:22:53 | mriedem | melwitt: supported ops for the zvm driver are in the feature support matrix | |
| 16:22:58 | mriedem | so i wouldn't put that in the reno | |
| 16:23:16 | melwitt | ok. efried, we don't need an additional reno ^ | |
| 16:24:14 | efried | Okay, wfm. jichenjc, in case you're snooping later ^ | |
| 16:24:31 | efried | The prelude could have linked to the admin config doc, if it was landed, but it ain't, so... | |
| 16:26:41 | melwitt | based on what mriedem said, we can have a rc2 because of the docs, and land them there | |
| 16:27:10 | melwitt | so I'll add those to https://etherpad.openstack.org/p/nova-rocky-release-candidate-todo | |
| 16:27:29 | melwitt | ok he already added them, thanks | |
| 16:28:01 | mriedem | lyarwood: should i start reviewing https://review.openstack.org/#/c/587013/ again or do you expect more from mdbooth/ | |
| 16:28:02 | mriedem | ? | |
| 16:28:37 | lyarwood | mriedem: that should be good now | |
| 16:31:16 | melwitt | dansmith, lyarwood: could you pls review these changes for the ironic bug, we're holding the stable releases on those fixes https://review.openstack.org/#/q/topic:bug/1784705+status:open | |
| 16:32:31 | lyarwood | melwitt: ack'd the Pike changes | |
| 16:33:17 | melwitt | thanks | |
| 16:38:50 | mriedem | lyarwood: just a couple of small things in https://review.openstack.org/#/c/587013/ | |
| 16:44:02 | mriedem | random musings in your functional test too; i.e. i wonder how many volumes we orphan when nova creates the root volume and we reschedule | |
| 16:44:19 | mriedem | nice fun way to go over volume quota | |
| 16:46:06 | lyarwood | mriedem: hmmm I forgot the compute did that, does it not see the existing bdm? | |
| 16:48:03 | mriedem | the existing bdm will have source_type=image on it or whatever | |
| 16:48:05 | mriedem | right? | |
| 16:48:07 | mriedem | we don't update that | |
| 16:48:19 | mriedem | so it will be transformed to a DriverImageBlockDevice or whatever | |
| 16:48:28 | openstackgerrit | Merged openstack/nova stable/pike: Fix bad links for admin-guide https://review.openstack.org/590072 | |
| 16:48:40 | mriedem | DriverVolImageBlockDevice | |
| 16:48:59 | mriedem | anyway, haven't tested it, but i'm pretty sure that's been busted since forever | |
| 16:49:07 | mriedem | we cleanup after ourselves for ports but not volumes | |
| 16:51:00 | lyarwood | ah right understood, should be easy enough to show in another functional test | |
| 16:51:46 | mriedem | maybe.....i'm not sure the fixture is setup for that really | |
| 16:51:57 | mriedem | devstack is probably much easier/faster to start | |
| 16:52:02 | mriedem | if you have 2 nodes... | |
| 16:57:03 | lyarwood | mriedem: I don't to hand but I'll make a note to give this a go | |
| 17:21:48 | openstackgerrit | Lee Yarwood proposed openstack/nova master: block_device: Rollback volumes to in-use on DeviceDetachFailed https://review.openstack.org/590439 | |
| 17:34:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | |
| 17:38:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | |
| 17:44:48 | openstackgerrit | Merged openstack/nova master: Update the parameter explain when updating a volume attachment https://review.openstack.org/565181 | |
| 19:11:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | |
| 19:15:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix image-defined numa claims during evacuate https://review.openstack.org/588657 | |
| 19:18:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add encrypted volume support to feature matrix docs https://review.openstack.org/570255 | |
| 19:28:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: fix GET /flavors?is_public description https://review.openstack.org/588092 | |
| 19:37:11 | prometheanfire | is https://github.com/openstack/nova/commit/ff747792b8f5aefe1bebb01bdf49dacc01353348#diff-f4019782d93a196a0d026479e6aa61b1R6938 run multiple times or only checked once? | |
| 19:37:16 | openstackgerrit | Merged openstack/nova stable/queens: Fix host validity check for live-migration https://review.openstack.org/590262 | |
| 19:37:23 | openstackgerrit | Merged openstack/nova master: [placement] api-ref: add description for 1.29 https://review.openstack.org/589407 | |
| 19:37:36 | prometheanfire | live migrations seem to be limited to 1M a sec and never increase | |
| 19:44:23 | melwitt | hm | |
| 19:45:18 | melwitt | do you know anything about that mriedem ^ | |
| 19:45:56 | mriedem | prometheanfire: linuxbridge? | |
| 19:46:23 | mriedem | https://github.com/openstack/nova/commit/ff747792b8f5aefe1bebb01bdf49dacc01353348#diff-f4019782d93a196a0d026479e6aa61b1R5380 | |
| 19:46:27 | mriedem | are you using linuxbridge i mean | |
| 19:47:46 | prometheanfire | ya, lb | |
| 19:47:56 | prometheanfire | vlan interface on the VM | |
| 19:48:35 | mriedem | well, we should be waiting on network-vif-plugged events from neutron and if we get them, we set the bw back up and resume the live migration, else we should fail the live migration | |
| 19:48:56 | mriedem | i'm assuming you have vif_plugging_timeout left at the default config of 300? | |
| 19:49:03 | prometheanfire | the migration finishes, it's just slow | |
| 19:49:10 | prometheanfire | I don't think we changed it | |
| 19:49:30 | mriedem | does it finish in under 5 minutes? | |
| 19:49:38 | prometheanfire | takes ~10 min | |
| 19:49:47 | prometheanfire | debug log | |
| 19:49:50 | prometheanfire | https://gist.githubusercontent.com/mheler/475d21b741aa58f320a456c3ac0d0f45/raw/ff76c45c3b6968b0d0514a9a7dcf478f054f6b70/gistfile1.txt | |
| 19:49:54 | mriedem | we should have either timed out and failed by then or reconfigured the guest to go back to the normal bw | |
| 19:50:42 | prometheanfire | which includes x-auth info, great | |
| 19:50:46 | mriedem | i don't see either the timeout or "VIF events received, continuing migration" messages in those logs | |
| 19:50:57 | prometheanfire | ya, either do I, which is why I'm confused | |
| 19:51:01 | melwitt | I'm not seeing that message "LOG.debug('VIF events received, continuing migration with max bandwidth configured" in your logs | |
| 19:51:16 | prometheanfire | that was the first thing I looked for | |
| 19:51:22 | mriedem | _http_log_request /openstack/venvs/nova-r16.2.2/lib/python2.7/site-packages/keystoneauth1/session.py:375 for that x-auth thing | |
| 19:51:23 | melwitt | bah, lag | |
| 19:51:44 | mriedem | besides me, sahid and dansmith are the other two that know about that change, | |
| 19:51:52 | mriedem | but are you sure you actually have that code? | |
| 19:52:02 | mriedem | would be nice if we logged something like "waiting for events" in that block | |
| 19:52:45 | prometheanfire | ya, set using b58c7f033771e3ea228e4b40c796d1bc95a087f5 from nova | |
| 19:54:04 | mriedem | prometheanfire: well your token thing isn't a problem :) https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L371 | |
| 19:54:06 | mriedem | it's redacted | |
| 19:54:28 | mriedem | prometheanfire: do you know the instance id in question here? | |
| 19:54:36 | mriedem | checking logs w/o an instance id is kind of hard | |
| 19:54:36 | prometheanfire | yes | |
| 19:54:55 | mriedem | also, | |
| 19:55:05 | mriedem | are these logs from the source or dest host? | |
| 19:55:11 | mriedem | b/c what we're looking for would be source host | |
| 19:55:20 | prometheanfire | the logs are from grepping it for c37d7489-a67b-47ea-a4f7-9323804cc552 | |
| 19:55:32 | prometheanfire | ya, source | |
| 19:56:02 | mriedem | 2018-08-09 21:20:20.557 12111 DEBUG nova.compute.manager [req-84ca4a17-0d3d-4597-91d6-f5721989dd41 143ee57edd4d4e3b9a165d375d0e7e1a a727713d2c0a4ed69b730d9cb2116af6 - default default] [instance: c37d7489-a67b-47ea-a4f7-9323804cc552] Received event network-vif-plugged-490f6f25-8b88-487c-a76b-62d16e3c0da1 external_instance_event /openstack/venvs/nova-r16.2.2/lib/python2.7/site-packages/nova/compute/manager.py:7071 | |
| 19:56:10 | mriedem | 2018-08-09 21:20:20.558 12111 DEBUG nova.compute.manager [req-84ca4a17-0d3d-4597-91d6-f5721989dd41 143ee57edd4d4e3b9a165d375d0e7e1a a727713d2c0a4ed69b730d9cb2116af6 - default default] [instance: c37d7489-a67b-47ea-a4f7-9323804cc552] No waiting events found dispatching network-vif-plugged-490f6f25-8b88-487c-a76b-62d16e3c0da1 pop_instance_event /openstack/venvs/nova-r16.2.2/lib/python2.7/site-packages/nova/compute/manager.py:36 | |
| 19:56:18 | mriedem | 2018-08-09 21:20:20.559 12111 WARNING nova.compute.manager [req-84ca4a17-0d3d-4597-91d6-f5721989dd41 143ee57edd4d4e3b9a165d375d0e7e1a a727713d2c0a4ed69b730d9cb2116af6 - default default] [instance: c37d7489-a67b-47ea-a4f7-9323804cc552] Received unexpected event network-vif-plugged-490f6f25-8b88-487c-a76b-62d16e3c0da1 for instance | |
| 19:56:33 | prometheanfire | this is a pike install btw | |
| 19:56:46 | mriedem | maybe we're getting the event before we're waiting for it? | |
| 19:56:47 | prometheanfire | but it was backported to pike, so meh | |
| 19:56:51 | prometheanfire | perhaps | |
| 19:57:11 | mriedem | we could also be getting ^ from the vif plug that happens on the dest host during pre-live migration | |
| 19:57:19 | mriedem | the events are going to go to the source host | |
| 19:57:22 | mriedem | which isn't waiting for those | |
| 19:57:29 | prometheanfire | not yet at least, ya | |