| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-04 | |||
| 22:37:32 | lyarwood | mriedem: <source protocol='rbd' name='volumes/volume-6d04520d-0029-499c-af81-516a7ba37a54'> <-- this one is changed, new ips | |
| 22:37:44 | mriedem | right, because the live migration flow calls _get_volume_config | |
| 22:37:50 | mriedem | using the bdm.connection_info | |
| 22:37:53 | mriedem | which is stale | |
| 22:38:00 | hemna_ | yup | |
| 22:38:01 | lyarwood | yup | |
| 22:38:06 | mriedem | what we said in denver, | |
| 22:38:18 | mriedem | was when we start live migration, and get the bdms, we call cinder to refresh the connection info | |
| 22:38:37 | mriedem | by pass refresh_conn_info=True to https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1683 | |
| 22:38:52 | mriedem | that will make a new os-initialize_connection call to cinder and return the latest connection_info | |
| 22:39:07 | mriedem | then we save that on the bdm.connection_info | |
| 22:39:18 | mriedem | for reasons, 'ive just never written that patch, | |
| 22:39:26 | mriedem | but i also have concerns about the new attach flow making that no longer work | |
| 22:39:33 | hemna_ | :( | |
| 22:39:46 | mriedem | with the new flow, nova calls cinder to get the attachment record which has the connection_info stored in the cinder db, | |
| 22:39:58 | mriedem | so i don't think it actually creates a new connection (export?) so we wouldn't refresh | |
| 22:40:05 | hemna_ | which could be stale... | |
| 22:40:12 | mriedem | this https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L557 | |
| 22:40:20 | hemna_ | in this case we just moved the stale info from nova to cinder, and cinder is stale? | |
| 22:40:39 | mriedem | hemna_: right, if cinder doesn't update that, and just returns whta's in the db, it would be stale - and we'd have the same problem as using the stale nova db info | |
| 22:40:43 | mriedem | yes | |
| 22:40:45 | mriedem | exactly | |
| 22:40:51 | hemna_ | bleh | |
| 22:42:22 | mriedem | yeah idk, nova could still call os-initialize_connection if we wanted to, | |
| 22:42:31 | mriedem | i don't know if that would update the attachment record in cinder or not | |
| 22:43:09 | hemna_ | so I thought the purpose of storing the conn info in cinder's db is for force delete time when nova doesn't know anything | |
| 22:43:12 | mriedem | otherwise we'd likely need like a refresh=True query parameter to GET /attachments/{id} | |
| 22:43:37 | mriedem | hemna_: that's part of it yeah | |
| 22:43:38 | hemna_ | in which case, can't cinder always refresh that if nova asks for it? | |
| 22:43:45 | hemna_ | I dunno | |
| 22:43:55 | mriedem | but how does cinder know that we're asking for a refresh? | |
| 22:43:58 | mriedem | or just a simple read-only get | |
| 22:44:13 | mriedem | this is where we got talking about API changes to force a refresh | |
| 22:44:13 | hemna_ | true | |
| 22:44:52 | hemna_ | I guess this could affect other cinder backends too | |
| 22:44:58 | hemna_ | not just ceph | |
| 22:46:05 | hemna_ | multipath IPs can change in between attaching the same volume, if the cinder backend gets changed | |
| 22:46:50 | hemna_ | ok so for now, I'll just tell my customer to bounce their VMs. :( | |
| 22:46:55 | mriedem | like a retype? | |
| 22:47:14 | hemna_ | well, more like a storage array loses one of it's interfaces | |
| 22:47:16 | hemna_ | and the IP changes | |
| 22:47:16 | mriedem | or you mean the backend backend | |
| 22:47:19 | mriedem | ok | |
| 22:47:20 | hemna_ | or new interfaces are added | |
| 22:47:39 | hemna_ | it's kinda the same thing as a new ceph monitor IP | |
| 22:48:11 | lyarwood | hemna_: FWIW restarting the instance is the only way to update the mon IPs for the ephemerial rbd images. | |
| 22:48:21 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in put (1) https://review.openstack.org/531258 | |
| 22:48:29 | lyarwood | ephemeral* | |
| 22:48:41 | hemna_ | lyarwood, yah. they were trying to avoid that, as there could be tons of VMs to bounce | |
| 22:49:03 | mriedem | hmm, actually it looks like for live migration we do refresh the connection_info https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5882 | |
| 22:49:10 | mriedem | in pre-live migration on the dest host | |
| 22:49:58 | lyarwood | right, the issue hemna_ reported in the bug isn't with rbd volumes :) | |
| 22:50:08 | mriedem | oh | |
| 22:50:30 | mriedem | now i get it | |
| 22:51:22 | mriedem | well, i'm reminded once again that i'd like to just give up and buy a farm in austria and just live out my days there | |
| 22:51:37 | mriedem | because this whole software business isn't worth it | |
| 22:51:41 | hemna_ | lolz | |
| 22:52:01 | hemna_ | you aren't the only one.....I've been in a youtube black whole, watching dudes build log cabins... | |
| 22:52:21 | mriedem | that's the manliest thing i've heard all day | |
| 22:58:49 | hemna_ | lyarwood, should I file a separate bug then to cover the rbd image update during LM ? | |
| 23:00:01 | lyarwood | hemna_: yeah I think so, it's a different codepath etc | |
| 23:00:09 | hemna_ | ok will do | |
| 23:00:25 | lyarwood | hemna_: cool thanks, I'll take a look tomorrow | |
| 23:06:47 | openstackgerrit | Eric Fried proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540 | |
| 23:06:47 | openstackgerrit | Eric Fried proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539 | |
| 23:06:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541 | |
| 23:06:48 | openstackgerrit | Eric Fried proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756 | |
| 23:06:49 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187 | |
| 23:06:49 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098 | |
| 23:06:50 | openstackgerrit | Eric Fried proposed openstack/nova master: Fix nits in update_provider_tree series https://review.openstack.org/531260 | |
| 23:06:50 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246 | |
| 23:07:15 | efried | jaypipes ^ -- and response to review comments on the way... | |
| 23:07:25 | jaypipes | coolio. | |
| 23:08:01 | hemna_ | lyarwood, https://bugs.launchpad.net/nova/+bug/1741364 | |
| 23:08:02 | openstack | Launchpad bug 1741364 in OpenStack Compute (nova) "ceph ephemeral info not updated during live migrate" [Undecided,New] | |
| 23:11:54 | efried | jaypipes See response on https://review.openstack.org/#/c/526539/ -- not sure if I've answered the right question there. | |
| 23:26:09 | mnaser | does nova not set the instance status to ERROR if cinder volume detach fails? | |
| 23:26:39 | mnaser | I have some cinder volumes which show attached servers, but the server is deleted | |
| 23:31:10 | mriedem | mnaser: correct | |
| 23:31:40 | mriedem | normal volume detach? or volume detach during instance delete? | |
| 23:31:53 | mnaser | mriedem: volume detach during instance delete | |
| 23:32:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass multiattach flag to reserve_block_device_name https://review.openstack.org/531244 | |
| 23:32:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments https://review.openstack.org/267587 | |
| 23:32:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: [api] Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 23:32:34 | mriedem | mnaser: yeah that happens here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2391 | |
| 23:32:35 | mnaser | (boot from volume instance) | |
| 23:32:49 | mriedem | if it fails we log something but keep going | |
| 23:32:54 | mriedem | because we've already destroyed the guest on the hypervisor | |
| 23:33:23 | mriedem | this is where we try to delete the volume if delete_on_termination=True https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2440 | |
| 23:33:59 | mnaser | hmmm | |
| 23:34:39 | mnaser | because now i end up with volumes with attachments (for whatever reason) and there doesn't seem to be a straight forward way of clearing that out | |
| 23:34:50 | mriedem | force detach in cinder? | |
| 23:35:15 | mnaser | i see force delete but not force detach | |
| 23:35:19 | mnaser | (force delete is ok for me though) | |
| 23:36:17 | mriedem | that might be what i was thinking of | |
| 23:36:29 | openstackgerrit | Andreas Karis proposed openstack/nova master: Add debug output for selected page size https://review.openstack.org/530662 | |
| 23:36:29 | mriedem | smcginnis: knows | |
| 23:36:57 | mnaser | Delete for volume 973d5a3d-07f3-4d05-b62c-8dfe746d298c failed: Invalid volume: Volume must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-ecfad96c-8125-47e4-8e25-9a9e892139aa) | |
| 23:37:01 | mnaser | i guess this is in cinder-land | |
| 23:37:21 | mnaser | (cant force delete an attach volume) | |