Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-24
17:12:44 melwitt yeah the log file
17:13:07 dansmith yeah, but that doesn't mean it's running under wsgi...
17:13:32 melwitt I can see it's running under wsgi from the ps.txt.gz here http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/ps.txt.gz
17:13:52 melwitt but I'm not sure how to tell whether it's the standalone service or the all-in-one
17:14:10 dansmith apparently we are
17:14:11 dansmith http://logs.openstack.org/77/594577/6/check/neutron-grenade/f10f831/logs/etc/nova/nova-metadata-uwsgi.ini.txt.gz
17:14:20 dansmith that's the uwsgi definition for it
17:14:24 melwitt oh yeah, that too
17:15:13 dansmith not sure if we capture that /usr/local/bin/nova-api-metadata-wsgi file though
17:15:15 dansmith maybe that's auto-generated
17:15:32 melwitt ah, so maybe it's set in devstack somewhere
17:16:33 dansmith but again, what in the config are you looking for related to the indirection api?
17:17:03 dansmith I'm not sure what you meant above about "it's standalone so much be using the indirection api"
17:18:17 openstackgerrit Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698
17:18:18 openstackgerrit Dan Smith proposed openstack/nova master: List instances from all cells explicitly https://review.openstack.org/593717
17:18:18 melwitt no, sorry, I'm trying to determine whether that proposed change to devstack is right or not. it looks like it's trying to make the standalone service default to the cell database so it can find instance metadata. in devstack, it looks like if services_enabled includes n-api-meta, it is doing the standalone service which we said is not supposed to be used with multiple cells
17:18:19 openstackgerrit Dan Smith proposed openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131
17:18:20 openstackgerrit Dan Smith proposed openstack/nova master: Record cell success/failure/timeout in CrossCellLister https://review.openstack.org/594265
17:18:21 openstackgerrit Dan Smith proposed openstack/nova master: Optimize global marker re-lookup in multi_cell_list https://review.openstack.org/594577
17:19:10 melwitt then, I looked at an existing multinode job to see, how is the metadata API service currently working then without that change?
17:19:52 dansmith well, I think that change isn't right,
17:20:06 dansmith because it means we won't do multi-cell mode _unless_ the standalone metadata service is enabled right?
17:20:10 dansmith which is clearly not what we want
17:20:18 melwitt here, it looks like the neutron multinode job is running with the standalone meta service http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/devstacklog.txt.gz#_2018-08-23_21_19_27_916 so how is it finding metadata? just confused
17:21:36 melwitt oh, I was thinking that was an OR. my mistake I think
17:21:44 dansmith Aug 23 21:29:00.426509 ubuntu-xenial-inap-mtl01-0001520794 devstack@n-api-meta.service[4018]: DEBUG nova.api.openstack.wsgi_app [None req-848ebec7-678d-4d46-923a-2b41bf306753 None None] config files: ['/etc/nova/api-paste.ini', '/etc/nova/nova.conf'] {{(pid=4020) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:3009}}
17:21:59 melwitt oh, thank you
17:22:01 dansmith that standalone service is running from nova.conf
17:22:40 melwitt ok, and nova.conf is pointing at the cell0 database
17:22:51 dansmith well, but,
17:23:00 dansmith it's using indirection, which means it goes to superconductor
17:23:15 dansmith which should be pointing at cell0 too
17:23:53 melwitt yeah... so shouldn't getting metadata not work? that's what's confusing me. but I see requests going successfully in the meta API log
17:24:16 dansmith sorry, I'm not being clear
17:24:19 mnaser throwing this out here: how does barbican stable/rocky look right now? is it pretty much likely going to be the released version? (i.e. will/how much i get in trouble for deploying a new region on stable/rocky)
17:24:20 mnaser :p
17:24:39 mnaser er
17:24:43 mnaser s/barbican/nova/
17:24:46 dansmith melwitt: the standalone runner would be using indirection_api, going to superconductor and hitting cell0 and failing
17:25:25 dansmith melwitt: but we're not running _our_ standalone metadata service, we're running metadata api as non-standalone, but in its own wsgi container
17:25:40 melwitt dansmith: oh, geez, okay
17:25:42 dansmith so it's behaving like the combined global api service would have previously when we smashed them all together,
17:26:02 dansmith but with wsgi you yank them into life a little differently, without the nova-api service bootstrapping code
17:26:19 dansmith which means it should be looking up the mapping for instances and talking to the right cell, like in global mode
17:26:57 melwitt cray cray
17:27:24 dansmith calling it "non-standalone" is a bit of a misnomer, but compared to our old combined service bootstrapper thing, it's in "non-standalone" mode of operation
17:27:33 dansmith but.. it's fairly standalone in that it's running all by itself and has its own log file
17:27:39 melwitt dansmith: okay, so what is this then here? this looks like it means if service enabled n-api-meta, then remove the metadata from nova_enabled_apis and run it is _our_ standalone service. right? https://review.openstack.org/#/c/564978/2/lib/nova@470
17:27:54 melwitt *as
17:28:07 dansmith that's probably now confused,
17:28:51 dansmith as that used to be how devstack referred to *our* standalone service, and may still in some cases, idk, but now post-systemd and post-wsgi probably refers to the, uh, other one :)
17:29:04 melwitt mnaser: we think so, but we also thought we'd only have one RC and now we have 3
17:29:22 mnaser maybe ican help you make it 4, hah :P
17:29:24 dansmith melwitt: why are you even reviewing this with no details in the bug or commit message?
17:29:46 mnaser thats reasonable. ill proceed being careful and give input :)
17:30:53 dansmith melwitt: mriedem_afk: is this known? http://logs.openstack.org/71/594571/2/gate/nova-tox-functional-py35/fd2d9ac/testr_results.html.gz
17:31:01 melwitt mnaser: haha :) always happy to get your testing of the code, and sorry we've been behind on the cells orphaned records patch reviews. things have been just really crazy
17:31:16 dansmith failed on both jobs of one of my patches
17:31:32 mnaser melwitt: nah its cool, my workaround for that was make sure our underlying infra doesnt break and thats ok enough :p
17:31:51 mnaser i dont think things that break after unrelated nova issues would be priority number 1, imho
17:32:28 melwitt thanks for the understanding
17:32:49 melwitt dansmith: that's the first time I've seen that
17:33:39 melwitt hm, weird FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'
17:34:22 melwitt it tries to write a file to test for directio support
17:35:38 melwitt and that failed. does that mean the nova/instances/ dir missing from the test tree probably? hm
17:36:15 dansmith just not sure what would have changed
17:36:32 melwitt I can't remember if that was one of those wonky directories that get created during test runs, to put temp files in for tests
17:37:08 melwitt like, what is that instances/ directory, I can't remember
17:37:59 melwitt oh wait, it's that the actual code is trying to write that file in a real life scenario and in the test env there is no nova/instances dir. but I don't know how that used to be working if it was working before
17:38:29 melwitt in func test land
17:46:38 melwitt CONF.instances_path
18:05:36 jrock_cfdg could someone please help out? Trying to add a serial port with source mode='connect' to do low-level kernel debugging in windows instance; everytime i add device to instance it shuts down - is this even possible with nova? i can do this in libvirt w/out issue
18:08:20 Sundar_ fried_rice, efried: Do you have a moment?
18:08:41 fried_rice Sundar_: sure.
18:08:51 openstackgerrit Merged openstack/nova master: Make CELL_TIMEOUT a constant https://review.openstack.org/594570
18:08:55 fried_rice Sundar_: I *am* trying to get to reviewing your spec really soon
18:09:09 fried_rice Sundar_: I've just been inundated with other things.
18:09:21 Sundar_ In the scheduling spec, we defined a way to extend the granular resource request syntax to support Cyborg: https://git.openstack.org/cgit/openstack/cyborg/tree/doc/specs/rocky/cyborg-nova-sched.rst?h=refs/changes/17/554717/18#n315
18:09:43 Sundar_ Just want to make sure you have reviewed it.
18:10:30 Sundar_ fied_rice: NP :) I understand. We all have to roll on different tracks at the same time.
18:11:18 Sundar_ And I may have added yet another track for you to roll on.
18:12:36 fried_rice Sundar_: I remember seeing this and being okay with it, but I would have to do a major context switch to be able to discuss it intelligently at this point.
18:14:16 fried_rice I noticed you said the extended keys aren't interpreted by nova. Presumably that means they're interpreted by cyborg. How does that information get to cyborg? And does it translate to some new/different placement syntax? I assume not, that it just tells cyborg how to do certain things.
18:17:32 Sundar_ fried_rice: Good to know that it is accepted. Yes, they will be interpreted by Cyborg (not sure if os-acc will also interpret it). The extra specs in the request are passed from Nova virt to os-acc to Cyborg.
18:17:44 Sundar_ No new placement APIs needed.
18:19:18 melwitt I have to run to an appointment, will be back later
18:34:16 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge security groups extension response into server view builder https://review.openstack.org/585475
18:35:49 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge extended_status extension response into server view builder https://review.openstack.org/592092
18:41:41 openstackgerrit Merged openstack/nova master: tests: Create functional libvirt test base class https://review.openstack.org/407055
18:50:31 mriedem dansmith: i haven't seen that functional test failure
19:00:55 mriedem dansmith: umm, could https://review.openstack.org/#/c/407055/ that just merged be a regression?
19:02:44 mriedem or the change below it
19:02:55 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22FileNotFoundError%3A%20%5BErrno%202%5D%20No%20such%20file%20or%20directory%3A%20'%2Fhome%2Fzuul%2Fsrc%2Fgit.openstack.org%2Fopenstack%2Fnova%2Finstances%2F.directio.test'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
19:03:28 mriedem just started
19:03:32 mriedem so those are probably it
19:05:16 mriedem https://bugs.launchpad.net/nova/+bug/1788922
19:05:17 openstack Launchpad bug 1788922 in OpenStack Compute (nova) "SRIOVServersTest.test_create_server_with_VF intermittently fails due to "FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'"" [Undecided,New]
19:07:15 mriedem -1ed https://review.openstack.org/#/c/564978/ which it looks like you and melwitt were talking about - the change has no context
19:07:20 mriedem "this fixes something"
19:14:40 mriedem oh you know, this func test might only be failing on one node provider

Earlier   Later