Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-24
16:58:40 stephenfin I think it was more a case of encoding faulty assumptions
16:58:42 mriedem this is optimizing things yes?
16:58:59 stephenfin namely, that asymmetric host topologies weren't a thing
16:59:24 stephenfin Yeah, optimization is probably fair
16:59:50 stephenfin without the patch, you see issues when your hosts get a bit congested
16:59:59 stephenfin ...and you're using emulator threads
17:00:07 mriedem well, given (1) how large this is and multiple changes and (2) it sounds low priority bug-wise if it's not outright failing, and (3) i'm in no way an expert on this code, i'm hesitant to even get close to a backport of this
17:00:29 stephenfin Also fair
17:00:32 mriedem if dan puts on his nfv shoes maybe he'd care
17:00:33 mriedem idk
17:00:59 stephenfin I'll see if I can rope leakypipes into it next week, given that he reviewed it first time round
17:01:07 stephenfin If not...I'll think of something
17:01:52 stephenfin Aight, home time o/
17:02:47 melwitt how do nova.conf get used with services running under uwsgi? how do you specify which conf
17:05:48 dansmith melwitt: depends on what wsgi container you're using I think
17:07:34 melwitt dansmith: what does that mean? which nova service? or something external to that?
17:08:05 dansmith no, like apache, ngnix, and then which wsgi module on each you use (uwsgi, etc)
17:08:12 melwitt oh, ok
17:09:33 dansmith I don't know much about uwsgi, so not sure where to point you to for it
17:10:16 melwitt context is, I was thinking this proposed devstack change is wrong https://review.openstack.org/#/c/564978/2/lib/nova but then, I don't get how the standalone metadata API service is currently working with querying instance metadata, for example, in the neutron-tempest-multinode-full job. it must be using the indirection api because it's standalone, but then how is it accessing the cell database?
17:10:31 dansmith with mod_wsgi you have a little app definition python file which is where you'd do it (or, I'd do it)
17:10:43 melwitt and I can't see how to see what nova.conf the uwsgi metadata service is running with
17:10:55 melwitt I see
17:11:16 dansmith do we run the standalone metadata api under wsgi? I don't think we do
17:11:55 dansmith even still,
17:12:07 dansmith the indirection_api thing isn't part of config so I don't think there's anything inconfig that is relevant
17:12:08 dansmith https://github.com/openstack/nova/blob/master/nova/cmd/api_metadata.py#L45-L45
17:12:14 dansmith that's where it gets forced on for the standalone one
17:12:21 melwitt okay, maybe I got confused by something. because there's a separate n-api-metadata file, but that doesn't mean it's running standalone necessarily right?
17:12:40 melwitt I was thinking it meant it's standalone
17:12:40 dansmith file meaning.. log file?
17:12:44 melwitt yeah the log file
17:13:07 dansmith yeah, but that doesn't mean it's running under wsgi...
17:13:32 melwitt I can see it's running under wsgi from the ps.txt.gz here http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/ps.txt.gz
17:13:52 melwitt but I'm not sure how to tell whether it's the standalone service or the all-in-one
17:14:10 dansmith apparently we are
17:14:11 dansmith http://logs.openstack.org/77/594577/6/check/neutron-grenade/f10f831/logs/etc/nova/nova-metadata-uwsgi.ini.txt.gz
17:14:20 dansmith that's the uwsgi definition for it
17:14:24 melwitt oh yeah, that too
17:15:13 dansmith not sure if we capture that /usr/local/bin/nova-api-metadata-wsgi file though
17:15:15 dansmith maybe that's auto-generated
17:15:32 melwitt ah, so maybe it's set in devstack somewhere
17:16:33 dansmith but again, what in the config are you looking for related to the indirection api?
17:17:03 dansmith I'm not sure what you meant above about "it's standalone so much be using the indirection api"
17:18:17 openstackgerrit Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698
17:18:18 openstackgerrit Dan Smith proposed openstack/nova master: List instances from all cells explicitly https://review.openstack.org/593717
17:18:18 melwitt no, sorry, I'm trying to determine whether that proposed change to devstack is right or not. it looks like it's trying to make the standalone service default to the cell database so it can find instance metadata. in devstack, it looks like if services_enabled includes n-api-meta, it is doing the standalone service which we said is not supposed to be used with multiple cells
17:18:19 openstackgerrit Dan Smith proposed openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131
17:18:20 openstackgerrit Dan Smith proposed openstack/nova master: Record cell success/failure/timeout in CrossCellLister https://review.openstack.org/594265
17:18:21 openstackgerrit Dan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list https://review.openstack.org/594577
17:19:10 melwitt then, I looked at an existing multinode job to see, how is the metadata API service currently working then without that change?
17:19:52 dansmith well, I think that change isn't right,
17:20:06 dansmith because it means we won't do multi-cell mode _unless_ the standalone metadata service is enabled right?
17:20:10 dansmith which is clearly not what we want
17:20:18 melwitt here, it looks like the neutron multinode job is running with the standalone meta service http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/devstacklog.txt.gz#_2018-08-23_21_19_27_916 so how is it finding metadata? just confused
17:21:36 melwitt oh, I was thinking that was an OR. my mistake I think
17:21:44 dansmith Aug 23 21:29:00.426509 ubuntu-xenial-inap-mtl01-0001520794 devstack@n-api-meta.service[4018]: DEBUG nova.api.openstack.wsgi_app [None req-848ebec7-678d-4d46-923a-2b41bf306753 None None] config files: ['/etc/nova/api-paste.ini', '/etc/nova/nova.conf'] {{(pid=4020) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:3009}}
17:21:59 melwitt oh, thank you
17:22:01 dansmith that standalone service is running from nova.conf
17:22:40 melwitt ok, and nova.conf is pointing at the cell0 database
17:22:51 dansmith well, but,
17:23:00 dansmith it's using indirection, which means it goes to superconductor
17:23:15 dansmith which should be pointing at cell0 too
17:23:53 melwitt yeah... so shouldn't getting metadata not work? that's what's confusing me. but I see requests going successfully in the meta API log
17:24:16 dansmith sorry, I'm not being clear
17:24:19 mnaser throwing this out here: how does barbican stable/rocky look right now? is it pretty much likely going to be the released version? (i.e. will/how much i get in trouble for deploying a new region on stable/rocky)
17:24:20 mnaser :p
17:24:39 mnaser er
17:24:43 mnaser s/barbican/nova/
17:24:46 dansmith melwitt: the standalone runner would be using indirection_api, going to superconductor and hitting cell0 and failing
17:25:25 dansmith melwitt: but we're not running _our_ standalone metadata service, we're running metadata api as non-standalone, but in its own wsgi container
17:25:40 melwitt dansmith: oh, geez, okay
17:25:42 dansmith so it's behaving like the combined global api service would have previously when we smashed them all together,
17:26:02 dansmith but with wsgi you yank them into life a little differently, without the nova-api service bootstrapping code
17:26:19 dansmith which means it should be looking up the mapping for instances and talking to the right cell, like in global mode
17:26:57 melwitt cray cray
17:27:24 dansmith calling it "non-standalone" is a bit of a misnomer, but compared to our old combined service bootstrapper thing, it's in "non-standalone" mode of operation
17:27:33 dansmith but.. it's fairly standalone in that it's running all by itself and has its own log file
17:27:39 melwitt dansmith: okay, so what is this then here? this looks like it means if service enabled n-api-meta, then remove the metadata from nova_enabled_apis and run it is _our_ standalone service. right? https://review.openstack.org/#/c/564978/2/lib/nova@470
17:27:54 melwitt *as
17:28:07 dansmith that's probably now confused,
17:28:51 dansmith as that used to be how devstack referred to *our* standalone service, and may still in some cases, idk, but now post-systemd and post-wsgi probably refers to the, uh, other one :)
17:29:04 melwitt mnaser: we think so, but we also thought we'd only have one RC and now we have 3
17:29:22 mnaser maybe ican help you make it 4, hah :P
17:29:24 dansmith melwitt: why are you even reviewing this with no details in the bug or commit message?
17:29:46 mnaser thats reasonable. ill proceed being careful and give input :)
17:30:53 dansmith melwitt: mriedem_afk: is this known? http://logs.openstack.org/71/594571/2/gate/nova-tox-functional-py35/fd2d9ac/testr_results.html.gz
17:31:01 melwitt mnaser: haha :) always happy to get your testing of the code, and sorry we've been behind on the cells orphaned records patch reviews. things have been just really crazy
17:31:16 dansmith failed on both jobs of one of my patches
17:31:32 mnaser melwitt: nah its cool, my workaround for that was make sure our underlying infra doesnt break and thats ok enough :p
17:31:51 mnaser i dont think things that break after unrelated nova issues would be priority number 1, imho
17:32:28 melwitt thanks for the understanding
17:32:49 melwitt dansmith: that's the first time I've seen that
17:33:39 melwitt hm, weird FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'
17:34:22 melwitt it tries to write a file to test for directio support
17:35:38 melwitt and that failed. does that mean the nova/instances/ dir missing from the test tree probably? hm
17:36:15 dansmith just not sure what would have changed
17:36:32 melwitt I can't remember if that was one of those wonky directories that get created during test runs, to put temp files in for tests
17:37:08 melwitt like, what is that instances/ directory, I can't remember
17:37:59 melwitt oh wait, it's that the actual code is trying to write that file in a real life scenario and in the test env there is no nova/instances dir. but I don't know how that used to be working if it was working before

Earlier   Later