Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-24
16:22:02 melwitt didn't have anything to do with refresh_conn_info tho (as you saw)
16:22:38 mdbooth I'm trying to come up with a recovery recommendation for this pre_live_migration timeout.
16:23:43 mdbooth The bug is: we called cleanup on the source when the dest timed out, even though it continued to run. The result is that the instance still has all volumes connected on the source where it continues to run, but some bdms have conn_info for source and others for dest.
16:24:05 mdbooth So really we want some operation which would cause a refresh_conn_info.
16:24:29 mdbooth Options seem to be cold and live migrations.
16:24:38 mdbooth Wonder if reboot ought to be another.
16:25:09 mriedem mdbooth: refresh_conn_info with the new attachment record flow isn't the same as how things worked with the old flow
16:25:50 smcginnis melwitt: Looks like that last stable/rocky patch merged, and you were right that the hash needs to be updated.
16:26:02 mdbooth mriedem: I'll take a look.
16:26:03 melwitt smcginnis: doing it right now, thanks
16:26:35 mriedem mdbooth: https://review.openstack.org/#/c/584648/20/nova/scheduler/client/report.py@2055 the difference is in the old flow, we'd always create a new connection to the storage backend which gives us a fresh connection_info from cinder,
16:26:48 mriedem with the new flow, we are getting the attachment record which has a stashed connection_info, which might be potentially stale
16:26:58 mriedem which is what i think you're worried about
16:27:28 mriedem btw this is related https://review.openstack.org/#/c/579004/
16:27:34 mdbooth mriedem: Ah, good point. So in that case attachment_id also be out of date.
16:28:07 mdbooth From that commit message: Note: Only hard reboots will currently update connection_info.
16:28:16 mdbooth I wonder if I missed something
16:28:47 mriedem that's just a result of the patch
16:29:02 mriedem but i think the comment is wrong
16:30:10 mdbooth mriedem: That's super-useful, thanks.
16:30:22 mriedem i am both very useful and very handsome
16:30:27 mriedem i think we can all agree
16:31:47 mdbooth mriedem: Your modesty is also unparalleled in all of history ;)
16:32:22 mriedem that's very kind of you
16:32:24 mriedem so,
16:32:28 mriedem without digging into the logs,
16:32:40 mriedem i expect this is blowing up b/c attachment_update isn't meant to be used this way once it's already got a connector
16:32:55 mriedem likely should be creating a new attachment record with new connector (and id) and deleting the old one
16:38:29 mriedem commented and pinged jgriffith
16:38:39 mdbooth Oh, the irony!
16:39:09 mdbooth So, if connection_info is wrong that's obviously going to break any attempt to attach
16:39:32 mdbooth But in this particular case because the volumes are already attached they're already working
16:40:04 mdbooth But since the robustification patch added detach/attach in hard reboot, we actually broke it
16:40:21 mdbooth It needs to be combined with refresh_conn_info
16:40:59 openstackgerrit Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698
16:41:00 openstackgerrit Dan Smith proposed openstack/nova master: List instances from all cells explicitly https://review.openstack.org/593717
16:41:01 openstackgerrit Dan Smith proposed openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131
16:41:02 openstackgerrit Dan Smith proposed openstack/nova master: Record cell success/failure/timeout in CrossCellLister https://review.openstack.org/594265
16:41:03 openstackgerrit Dan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list https://review.openstack.org/594577
16:41:28 mriedem umm,
16:41:31 mriedem well,
16:42:00 mriedem we've talked about this hard refresh thing for a long time b/c of the ceph ip or auth changing and not wanting to live migrate your entire cloud to update the attachments
16:44:35 mriedem i think coreycb is trying to fix this one case for reboot, and then see if we can extend to other operations
16:44:54 mriedem "This only includes a fix for the 'openstack server reboot --hard' path as I didn't want to get too deep before knowing if this is a proper fix. A complete fix would also provide soft reboot, stop/start support, perhaps more?"
16:46:02 coreycb mriedem: thanks for your comments. i dropped that patch for a bit but will get back to it once we get rocky done.
16:47:05 mdbooth mriedem: I'm coming from a different angle. A weird unfixable bug means I've now got inconsistent state, and I want a thing to make everything right again.
16:48:11 mriedem mdbooth: is that related to your other live migration patch?
16:48:27 mriedem https://review.openstack.org/#/c/551302/
16:49:01 mriedem bw, you probably need some more spinning plates
16:50:43 mdbooth mriedem: No.
16:51:02 openstackgerrit Dan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list https://review.openstack.org/594577
16:51:07 mdbooth mriedem: I thought about that one, but I don't think it would fix this.
16:51:59 mdbooth I should resurrect that patch as being just a generally good idea, even if the original motivation for it has gone. It would have fixed a couple of other things since iirc.
16:53:54 stephenfin mriedem: I'm not sure what I can do about this https://review.openstack.org/#/c/588570/
16:54:54 stephenfin mriedem: Have tried drafting a functional test but it looks like it's going to require a good deal of changes to the fake libvirt driver, assuming it's even possible to validate
16:55:01 mriedem mdbooth: fwiw i think i was ok with that change but wanted to see the same rollback stuff done in a follow up first
16:56:50 mriedem stephenfin: ok...i'm about out of here for lunch and errands, so probably need to hit me next week when i'm in more of a "f it" kind of mood
16:57:19 stephenfin mriedem: Heh, fair
16:57:22 mriedem because this definitely seems like new feature capability
16:57:30 mriedem if it takes this much to fix a thing, it probably never worked in the first place right?
16:58:37 mriedem the bug says a certain configuration is not "optimal"
16:58:40 stephenfin I think it was more a case of encoding faulty assumptions
16:58:42 mriedem this is optimizing things yes?
16:58:59 stephenfin namely, that asymmetric host topologies weren't a thing
16:59:24 stephenfin Yeah, optimization is probably fair
16:59:50 stephenfin without the patch, you see issues when your hosts get a bit congested
16:59:59 stephenfin ...and you're using emulator threads
17:00:07 mriedem well, given (1) how large this is and multiple changes and (2) it sounds low priority bug-wise if it's not outright failing, and (3) i'm in no way an expert on this code, i'm hesitant to even get close to a backport of this
17:00:29 stephenfin Also fair
17:00:32 mriedem if dan puts on his nfv shoes maybe he'd care
17:00:33 mriedem idk
17:00:59 stephenfin I'll see if I can rope leakypipes into it next week, given that he reviewed it first time round
17:01:07 stephenfin If not...I'll think of something
17:01:52 stephenfin Aight, home time o/
17:02:47 melwitt how do nova.conf get used with services running under uwsgi? how do you specify which conf
17:05:48 dansmith melwitt: depends on what wsgi container you're using I think
17:07:34 melwitt dansmith: what does that mean? which nova service? or something external to that?
17:08:05 dansmith no, like apache, ngnix, and then which wsgi module on each you use (uwsgi, etc)
17:08:12 melwitt oh, ok
17:09:33 dansmith I don't know much about uwsgi, so not sure where to point you to for it
17:10:16 melwitt context is, I was thinking this proposed devstack change is wrong https://review.openstack.org/#/c/564978/2/lib/nova but then, I don't get how the standalone metadata API service is currently working with querying instance metadata, for example, in the neutron-tempest-multinode-full job. it must be using the indirection api because it's standalone, but then how is it accessing the cell database?
17:10:31 dansmith with mod_wsgi you have a little app definition python file which is where you'd do it (or, I'd do it)
17:10:43 melwitt and I can't see how to see what nova.conf the uwsgi metadata service is running with
17:10:55 melwitt I see
17:11:16 dansmith do we run the standalone metadata api under wsgi? I don't think we do
17:11:55 dansmith even still,
17:12:07 dansmith the indirection_api thing isn't part of config so I don't think there's anything inconfig that is relevant
17:12:08 dansmith https://github.com/openstack/nova/blob/master/nova/cmd/api_metadata.py#L45-L45
17:12:14 dansmith that's where it gets forced on for the standalone one
17:12:21 melwitt okay, maybe I got confused by something. because there's a separate n-api-metadata file, but that doesn't mean it's running standalone necessarily right?
17:12:40 melwitt I was thinking it meant it's standalone
17:12:40 dansmith file meaning.. log file?
17:12:44 melwitt yeah the log file
17:13:07 dansmith yeah, but that doesn't mean it's running under wsgi...
17:13:32 melwitt I can see it's running under wsgi from the ps.txt.gz here http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/ps.txt.gz
17:13:52 melwitt but I'm not sure how to tell whether it's the standalone service or the all-in-one
17:14:10 dansmith apparently we are
17:14:11 dansmith http://logs.openstack.org/77/594577/6/check/neutron-grenade/f10f831/logs/etc/nova/nova-metadata-uwsgi.ini.txt.gz
17:14:20 dansmith that's the uwsgi definition for it
17:14:24 melwitt oh yeah, that too

Earlier   Later