Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-24
16:25:09 mriedem mdbooth: refresh_conn_info with the new attachment record flow isn't the same as how things worked with the old flow
16:25:50 smcginnis melwitt: Looks like that last stable/rocky patch merged, and you were right that the hash needs to be updated.
16:26:02 mdbooth mriedem: I'll take a look.
16:26:03 melwitt smcginnis: doing it right now, thanks
16:26:35 mriedem mdbooth: https://review.openstack.org/#/c/584648/20/nova/scheduler/client/report.py@2055 the difference is in the old flow, we'd always create a new connection to the storage backend which gives us a fresh connection_info from cinder,
16:26:48 mriedem with the new flow, we are getting the attachment record which has a stashed connection_info, which might be potentially stale
16:26:58 mriedem which is what i think you're worried about
16:27:28 mriedem btw this is related https://review.openstack.org/#/c/579004/
16:27:34 mdbooth mriedem: Ah, good point. So in that case attachment_id also be out of date.
16:28:07 mdbooth From that commit message: Note: Only hard reboots will currently update connection_info.
16:28:16 mdbooth I wonder if I missed something
16:28:47 mriedem that's just a result of the patch
16:29:02 mriedem but i think the comment is wrong
16:30:10 mdbooth mriedem: That's super-useful, thanks.
16:30:22 mriedem i am both very useful and very handsome
16:30:27 mriedem i think we can all agree
16:31:47 mdbooth mriedem: Your modesty is also unparalleled in all of history ;)
16:32:22 mriedem that's very kind of you
16:32:24 mriedem so,
16:32:28 mriedem without digging into the logs,
16:32:40 mriedem i expect this is blowing up b/c attachment_update isn't meant to be used this way once it's already got a connector
16:32:55 mriedem likely should be creating a new attachment record with new connector (and id) and deleting the old one
16:38:29 mriedem commented and pinged jgriffith
16:38:39 mdbooth Oh, the irony!
16:39:09 mdbooth So, if connection_info is wrong that's obviously going to break any attempt to attach
16:39:32 mdbooth But in this particular case because the volumes are already attached they're already working
16:40:04 mdbooth But since the robustification patch added detach/attach in hard reboot, we actually broke it
16:40:21 mdbooth It needs to be combined with refresh_conn_info
16:40:59 openstackgerrit Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698
16:41:00 openstackgerrit Dan Smith proposed openstack/nova master: List instances from all cells explicitly https://review.openstack.org/593717
16:41:01 openstackgerrit Dan Smith proposed openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131
16:41:02 openstackgerrit Dan Smith proposed openstack/nova master: Record cell success/failure/timeout in CrossCellLister https://review.openstack.org/594265
16:41:03 openstackgerrit Dan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list https://review.openstack.org/594577
16:41:28 mriedem umm,
16:41:31 mriedem well,
16:42:00 mriedem we've talked about this hard refresh thing for a long time b/c of the ceph ip or auth changing and not wanting to live migrate your entire cloud to update the attachments
16:44:35 mriedem i think coreycb is trying to fix this one case for reboot, and then see if we can extend to other operations
16:44:54 mriedem "This only includes a fix for the 'openstack server reboot --hard' path as I didn't want to get too deep before knowing if this is a proper fix. A complete fix would also provide soft reboot, stop/start support, perhaps more?"
16:46:02 coreycb mriedem: thanks for your comments. i dropped that patch for a bit but will get back to it once we get rocky done.
16:47:05 mdbooth mriedem: I'm coming from a different angle. A weird unfixable bug means I've now got inconsistent state, and I want a thing to make everything right again.
16:48:11 mriedem mdbooth: is that related to your other live migration patch?
16:48:27 mriedem https://review.openstack.org/#/c/551302/
16:49:01 mriedem bw, you probably need some more spinning plates
16:50:43 mdbooth mriedem: No.
16:51:02 openstackgerrit Dan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list https://review.openstack.org/594577
16:51:07 mdbooth mriedem: I thought about that one, but I don't think it would fix this.
16:51:59 mdbooth I should resurrect that patch as being just a generally good idea, even if the original motivation for it has gone. It would have fixed a couple of other things since iirc.
16:53:54 stephenfin mriedem: I'm not sure what I can do about this https://review.openstack.org/#/c/588570/
16:54:54 stephenfin mriedem: Have tried drafting a functional test but it looks like it's going to require a good deal of changes to the fake libvirt driver, assuming it's even possible to validate
16:55:01 mriedem mdbooth: fwiw i think i was ok with that change but wanted to see the same rollback stuff done in a follow up first
16:56:50 mriedem stephenfin: ok...i'm about out of here for lunch and errands, so probably need to hit me next week when i'm in more of a "f it" kind of mood
16:57:19 stephenfin mriedem: Heh, fair
16:57:22 mriedem because this definitely seems like new feature capability
16:57:30 mriedem if it takes this much to fix a thing, it probably never worked in the first place right?
16:58:37 mriedem the bug says a certain configuration is not "optimal"
16:58:40 stephenfin I think it was more a case of encoding faulty assumptions
16:58:42 mriedem this is optimizing things yes?
16:58:59 stephenfin namely, that asymmetric host topologies weren't a thing
16:59:24 stephenfin Yeah, optimization is probably fair
16:59:50 stephenfin without the patch, you see issues when your hosts get a bit congested
16:59:59 stephenfin ...and you're using emulator threads
17:00:07 mriedem well, given (1) how large this is and multiple changes and (2) it sounds low priority bug-wise if it's not outright failing, and (3) i'm in no way an expert on this code, i'm hesitant to even get close to a backport of this
17:00:29 stephenfin Also fair
17:00:32 mriedem if dan puts on his nfv shoes maybe he'd care
17:00:33 mriedem idk
17:00:59 stephenfin I'll see if I can rope leakypipes into it next week, given that he reviewed it first time round
17:01:07 stephenfin If not...I'll think of something
17:01:52 stephenfin Aight, home time o/
17:02:47 melwitt how do nova.conf get used with services running under uwsgi? how do you specify which conf
17:05:48 dansmith melwitt: depends on what wsgi container you're using I think
17:07:34 melwitt dansmith: what does that mean? which nova service? or something external to that?
17:08:05 dansmith no, like apache, ngnix, and then which wsgi module on each you use (uwsgi, etc)
17:08:12 melwitt oh, ok
17:09:33 dansmith I don't know much about uwsgi, so not sure where to point you to for it
17:10:16 melwitt context is, I was thinking this proposed devstack change is wrong https://review.openstack.org/#/c/564978/2/lib/nova but then, I don't get how the standalone metadata API service is currently working with querying instance metadata, for example, in the neutron-tempest-multinode-full job. it must be using the indirection api because it's standalone, but then how is it accessing the cell database?
17:10:31 dansmith with mod_wsgi you have a little app definition python file which is where you'd do it (or, I'd do it)
17:10:43 melwitt and I can't see how to see what nova.conf the uwsgi metadata service is running with
17:10:55 melwitt I see
17:11:16 dansmith do we run the standalone metadata api under wsgi? I don't think we do
17:11:55 dansmith even still,
17:12:07 dansmith the indirection_api thing isn't part of config so I don't think there's anything inconfig that is relevant
17:12:08 dansmith https://github.com/openstack/nova/blob/master/nova/cmd/api_metadata.py#L45-L45
17:12:14 dansmith that's where it gets forced on for the standalone one
17:12:21 melwitt okay, maybe I got confused by something. because there's a separate n-api-metadata file, but that doesn't mean it's running standalone necessarily right?
17:12:40 dansmith file meaning.. log file?
17:12:40 melwitt I was thinking it meant it's standalone
17:12:44 melwitt yeah the log file
17:13:07 dansmith yeah, but that doesn't mean it's running under wsgi...
17:13:32 melwitt I can see it's running under wsgi from the ps.txt.gz here http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/ps.txt.gz
17:13:52 melwitt but I'm not sure how to tell whether it's the standalone service or the all-in-one
17:14:10 dansmith apparently we are
17:14:11 dansmith http://logs.openstack.org/77/594577/6/check/neutron-grenade/f10f831/logs/etc/nova/nova-metadata-uwsgi.ini.txt.gz
17:14:20 dansmith that's the uwsgi definition for it
17:14:24 melwitt oh yeah, that too
17:15:13 dansmith not sure if we capture that /usr/local/bin/nova-api-metadata-wsgi file though
17:15:15 dansmith maybe that's auto-generated
17:15:32 melwitt ah, so maybe it's set in devstack somewhere
17:16:33 dansmith but again, what in the config are you looking for related to the indirection api?
17:17:03 dansmith I'm not sure what you meant above about "it's standalone so much be using the indirection api"
17:18:17 openstackgerrit Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698

Earlier   Later