| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-13 | |||
| 18:32:36 | sean-k-mooney | am it did | |
| 18:32:59 | sean-k-mooney | oh you posted it upstream then downstream | |
| 18:36:49 | artom | I learn from my mistakes :D | |
| 18:37:27 | artom | For posterioriy, @a<line number> links to the "before"/left pane. | |
| 19:37:16 | openstackgerrit | Daniel Speichert proposed openstack/nova-specs master: Directly download and upload images to RBD https://review.opendev.org/658903 | |
| 19:37:17 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: [WIP] Add 'power-update' external event to listen to ironic https://review.opendev.org/645611 | |
| 19:39:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.opendev.org/641806 | |
| 19:39:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.opendev.org/614012 | |
| 19:39:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.opendev.org/614036 | |
| 19:39:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.opendev.org/631123 | |
| 19:39:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.opendev.org/627892 | |
| 19:39:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.opendev.org/631581 | |
| 19:39:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.opendev.org/633853 | |
| 19:39:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.opendev.org/621313 | |
| 19:39:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add prep_snapshot_based_resize_at_dest compute method https://review.opendev.org/633293 | |
| 19:39:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add PrepResizeAtDestTask https://review.opendev.org/627890 | |
| 19:39:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832 | |
| 19:39:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.opendev.org/637605 | |
| 19:39:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891 | |
| 19:39:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.opendev.org/642183 | |
| 19:39:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add power_on kwarg to ComputeDriver.spawn() method https://review.opendev.org/642590 | |
| 19:39:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080 | |
| 19:39:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646 | |
| 19:39:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Destination.allow_cross_cell_move field https://review.opendev.org/614035 | |
| 19:40:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.opendev.org/614353 | |
| 19:40:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478 | |
| 19:40:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269 | |
| 19:40:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656 | |
| 19:40:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Robustify attachment tracking in CinderFixtureNewAttachFlow https://review.opendev.org/658904 | |
| 19:41:11 | mriedem | dansmith: i've hit a not-so-fun snag in pre-validating the target host wrt attached volumes in the cross-cell resize stuff and i'd like to brain dump | |
| 19:41:25 | dansmith | okay | |
| 19:42:06 | mriedem | so we want to try and connect the volume on the dest host to see if it will work before starting the whole process, | |
| 19:42:22 | mriedem | to do that, i need to update the volume attachment with the dest host connector to get the connection_info dict that the virt driver uses, | |
| 19:42:58 | mriedem | the problem is, at the point i'm doing that, the source host volume attachment still exists and cinder pukes saying you essentially can't have more than one connected attachment for the same volume at the same time | |
| 19:43:26 | mriedem | which makes sense i guess, and is not something we hit in a normal move because we delete the source host attachment before finishing on the dest | |
| 19:44:02 | mriedem | granted normal resize could fail the same thing today - if you can't connect the volume on the dest after we've transferred the disks and stuff | |
| 19:44:34 | dansmith | cross-cell is actually easier in this case because you have two separate instances and attachments on the nova side | |
| 19:44:56 | dansmith | normal resize would need to stash that in case it needs to revert right? or maybe just ask cinder again.. tbh, I dunno how that attachment stuff works really :) | |
| 19:45:28 | mriedem | attachments in cinder are essentially ephemeral, we can create and delete them as needed, or that's the idea anyway | |
| 19:45:39 | mriedem | so if i want to make this work, | |
| 19:46:09 | openstackgerrit | Merged openstack/python-novaclient master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648659 | |
| 19:46:09 | mriedem | i'd need to do something like delete the source host attachment, update the attachment with the dest host connector so we can test the connection on the dest host, | |
| 19:46:39 | mriedem | and then if something fails and we're rolling back, i'd have to create a new attachment and complete it for the source host (using the source host connector, which we don't stash away) | |
| 19:46:48 | dansmith | which entity is going to give you the go/no-go on the new attachment? cinder? brick? libvirt? | |
| 19:46:52 | mriedem | so not impossible, but kinda gross | |
| 19:47:38 | mriedem | updating the attachment with the host connector is what creates the exports in the storage backend | |
| 19:47:45 | mriedem | and gives us back the connection_info dict | |
| 19:48:09 | dansmith | okay, so, cinder yeah? | |
| 19:48:23 | mriedem | i'm honestly not sure if that's enough or not, i.e. if you still need the low-level connection on the host for libvirt/brick | |
| 19:48:48 | mriedem | we maybe don't need that - we don't try to plug vifs on the dest host during pre-validation | |
| 19:48:52 | dansmith | cinder might tell you no, or it might not know and say "sure" but then the actual attach fails I guess | |
| 19:49:13 | mriedem | yeah i'm not sure, that's why i was trying to do it all | |
| 19:49:53 | mriedem | either way i'm not disconnecting the guest / volume from the source host until we've pre-validated the dest | |
| 19:50:00 | mriedem | and start doing the power off / snapshot stuff | |
| 19:50:21 | dansmith | hmm | |
| 19:50:42 | dansmith | in what case is cinder going to tell you no to the attempted attachment change? | |
| 19:51:01 | dansmith | I mean... why do you think you need to do this special for cross-cell? | |
| 19:51:39 | mriedem | i was thinking in case we hit a scenario where the target cell is using a different storage backend from the source cell | |
| 19:52:05 | mriedem | not sure how realistic that is though | |
| 19:52:10 | dansmith | but I mean, | |
| 19:52:23 | dansmith | how is that different from single-cell resize? | |
| 19:52:51 | sean-k-mooney | mriedem: in an edge deployment its likely not that uncommon that they would have different backends | |
| 19:52:55 | mriedem | it's not really, like i said before we could hit this same thing during normal resize and we really don't try to pre-validate that (in prep_resize) or rollback from that kind of failure | |
| 19:53:25 | dansmith | yeah, I'm not sure what the backend difference is that you're talking about | |
| 19:53:35 | dansmith | in this case, the volume is in cinder, on a backend, but migrating won't change what *cinder* backend it's on, regardless of where you move it | |
| 19:54:52 | sean-k-mooney | dansmith: that is ture but that may mean the backen is not reachale from the new host. although if we are using sharing resouce providres to model this in placement it shoudl be reachable | |
| 19:55:09 | sean-k-mooney | assuming you have set up the aggreate correctly | |
| 19:55:21 | mriedem | we are not using sharing providers for any of this, or anything | |
| 19:55:21 | dansmith | sean-k-mooney: right, but that's what I'm saying has nothing to do with cross-cell migration | |
| 19:55:47 | sean-k-mooney | dansmith: sure | |
| 19:56:06 | sean-k-mooney | its not a special case that only comes up with cross cell | |
| 19:57:04 | mriedem | so if this is really not any different from normal resize then i'm probably bending over backwards for no good reason | |
| 19:57:26 | mriedem | but yeah i was thinking kind of like what sean said, the target host can't connect to the storage for the volume | |
| 19:58:05 | dansmith | mriedem: yep, but that's just a thing that can happen any time, | |
| 19:58:28 | dansmith | so if we're going to do something different, implement this test for regular and cross-cell, and assume they're the same until then, IMHO | |
| 19:58:39 | tssurya | efried: heh was just about to reply to that thread :) | |
| 19:59:00 | mriedem | tssurya: since you're around, does cern use the same cinder for all of its cells? or most anyway? | |
| 19:59:02 | efried | please do tssurya. I'm sure you can provide better insight. | |
| 19:59:19 | mriedem | s/cinder/volume backend/ | |
| 19:59:21 | dansmith | mriedem: do you mean the same cinder, or the same backend? | |
| 19:59:23 | sean-k-mooney | dansmith: mriedem for normal resize do we pass a member of parmater to ensure we select a host that is attache to the same sharign resouce provider | |
| 19:59:24 | dansmith | yeah :) | |
| 19:59:34 | dansmith | the latter, I'm sure not, but that's what cross-az-attach is for | |
| 19:59:38 | tssurya | mriedem: cinder for specific cells as far as I know | |
| 19:59:50 | tssurya | but yes same cinder | |
| 20:00:27 | mriedem | i only know of 2 users of cross_az_attach=False (NeCTAR and Target) | |
| 20:00:54 | tssurya | efried: yea what they are saying is true though, just making resource_provider_association_refresh to large value doesn't still solve it for ironic | |
| 20:01:24 | tssurya | we have been having the same problem and mostly might end up adding more computes | |
| 20:01:59 | tssurya | the periodic task locks really long enough (sometimes ~20mins) and doesn't allow boot request to walk past it and we get stuff stuck in scheduling | |
| 20:02:26 | dansmith | mriedem: you mean only two that speak up often or what? we definitely have users of it and are prescribing it for all edge deploys with cinder of course | |
| 20:02:58 | tssurya | so its interesting to see if for them setting update_resources_interval stays fine or not (in terms of things going out of sync) | |
| 20:03:00 | mriedem | right, i only saw target people mention it in a talk once | |
| 20:05:54 | mriedem | well i've commented out the trouble code for now and will have a gate run with a volume-backed cross-cell resize so i'll see if i run into any other bumps along the way; if the only issues with volume-backed cross-cell resize is this pre-validation thing, i'll probably just drop it. | |
| 20:09:34 | efried | tssurya: oh, okay, I thought I might be on the wrong track with refresh interval | |
| 20:09:50 | efried | please do correct me on the thred. | |
| 20:09:56 | efried | thread | |
| 20:10:01 | tssurya | yep no problem | |
| 20:27:53 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove fake_libvirt_utils from connection tests. https://review.opendev.org/642557 | |
| 20:27:54 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove fake_libvirt_utils from snapshot tests. https://review.opendev.org/642558 | |
| 20:27:54 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove fake_libvirt_utils from virt driver tests. https://review.opendev.org/643894 | |