| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-24 | |||
| 17:19:10 | melwitt | then, I looked at an existing multinode job to see, how is the metadata API service currently working then without that change? | |
| 17:19:52 | dansmith | well, I think that change isn't right, | |
| 17:20:06 | dansmith | because it means we won't do multi-cell mode _unless_ the standalone metadata service is enabled right? | |
| 17:20:10 | dansmith | which is clearly not what we want | |
| 17:20:18 | melwitt | here, it looks like the neutron multinode job is running with the standalone meta service http://logs.openstack.org/90/595890/2/check/neutron-tempest-multinode-full/a4a0803/logs/devstacklog.txt.gz#_2018-08-23_21_19_27_916 so how is it finding metadata? just confused | |
| 17:21:36 | melwitt | oh, I was thinking that was an OR. my mistake I think | |
| 17:21:44 | dansmith | Aug 23 21:29:00.426509 ubuntu-xenial-inap-mtl01-0001520794 devstack@n-api-meta.service[4018]: DEBUG nova.api.openstack.wsgi_app [None req-848ebec7-678d-4d46-923a-2b41bf306753 None None] config files: ['/etc/nova/api-paste.ini', '/etc/nova/nova.conf'] {{(pid=4020) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:3009}} | |
| 17:21:59 | melwitt | oh, thank you | |
| 17:22:01 | dansmith | that standalone service is running from nova.conf | |
| 17:22:40 | melwitt | ok, and nova.conf is pointing at the cell0 database | |
| 17:22:51 | dansmith | well, but, | |
| 17:23:00 | dansmith | it's using indirection, which means it goes to superconductor | |
| 17:23:15 | dansmith | which should be pointing at cell0 too | |
| 17:23:53 | melwitt | yeah... so shouldn't getting metadata not work? that's what's confusing me. but I see requests going successfully in the meta API log | |
| 17:24:16 | dansmith | sorry, I'm not being clear | |
| 17:24:19 | mnaser | throwing this out here: how does barbican stable/rocky look right now? is it pretty much likely going to be the released version? (i.e. will/how much i get in trouble for deploying a new region on stable/rocky) | |
| 17:24:20 | mnaser | :p | |
| 17:24:39 | mnaser | er | |
| 17:24:43 | mnaser | s/barbican/nova/ | |
| 17:24:46 | dansmith | melwitt: the standalone runner would be using indirection_api, going to superconductor and hitting cell0 and failing | |
| 17:25:25 | dansmith | melwitt: but we're not running _our_ standalone metadata service, we're running metadata api as non-standalone, but in its own wsgi container | |
| 17:25:40 | melwitt | dansmith: oh, geez, okay | |
| 17:25:42 | dansmith | so it's behaving like the combined global api service would have previously when we smashed them all together, | |
| 17:26:02 | dansmith | but with wsgi you yank them into life a little differently, without the nova-api service bootstrapping code | |
| 17:26:19 | dansmith | which means it should be looking up the mapping for instances and talking to the right cell, like in global mode | |
| 17:26:57 | melwitt | cray cray | |
| 17:27:24 | dansmith | calling it "non-standalone" is a bit of a misnomer, but compared to our old combined service bootstrapper thing, it's in "non-standalone" mode of operation | |
| 17:27:33 | dansmith | but.. it's fairly standalone in that it's running all by itself and has its own log file | |
| 17:27:39 | melwitt | dansmith: okay, so what is this then here? this looks like it means if service enabled n-api-meta, then remove the metadata from nova_enabled_apis and run it is _our_ standalone service. right? https://review.openstack.org/#/c/564978/2/lib/nova@470 | |
| 17:27:54 | melwitt | *as | |
| 17:28:07 | dansmith | that's probably now confused, | |
| 17:28:51 | dansmith | as that used to be how devstack referred to *our* standalone service, and may still in some cases, idk, but now post-systemd and post-wsgi probably refers to the, uh, other one :) | |
| 17:29:04 | melwitt | mnaser: we think so, but we also thought we'd only have one RC and now we have 3 | |
| 17:29:22 | mnaser | maybe ican help you make it 4, hah :P | |
| 17:29:24 | dansmith | melwitt: why are you even reviewing this with no details in the bug or commit message? | |
| 17:29:46 | mnaser | thats reasonable. ill proceed being careful and give input :) | |
| 17:30:53 | dansmith | melwitt: mriedem_afk: is this known? http://logs.openstack.org/71/594571/2/gate/nova-tox-functional-py35/fd2d9ac/testr_results.html.gz | |
| 17:31:01 | melwitt | mnaser: haha :) always happy to get your testing of the code, and sorry we've been behind on the cells orphaned records patch reviews. things have been just really crazy | |
| 17:31:16 | dansmith | failed on both jobs of one of my patches | |
| 17:31:32 | mnaser | melwitt: nah its cool, my workaround for that was make sure our underlying infra doesnt break and thats ok enough :p | |
| 17:31:51 | mnaser | i dont think things that break after unrelated nova issues would be priority number 1, imho | |
| 17:32:28 | melwitt | thanks for the understanding | |
| 17:32:49 | melwitt | dansmith: that's the first time I've seen that | |
| 17:33:39 | melwitt | hm, weird FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test' | |
| 17:34:22 | melwitt | it tries to write a file to test for directio support | |
| 17:35:38 | melwitt | and that failed. does that mean the nova/instances/ dir missing from the test tree probably? hm | |
| 17:36:15 | dansmith | just not sure what would have changed | |
| 17:36:32 | melwitt | I can't remember if that was one of those wonky directories that get created during test runs, to put temp files in for tests | |
| 17:37:08 | melwitt | like, what is that instances/ directory, I can't remember | |
| 17:37:59 | melwitt | oh wait, it's that the actual code is trying to write that file in a real life scenario and in the test env there is no nova/instances dir. but I don't know how that used to be working if it was working before | |
| 17:38:29 | melwitt | in func test land | |
| 17:46:38 | melwitt | CONF.instances_path | |
| 18:05:36 | jrock_cfdg | could someone please help out? Trying to add a serial port with source mode='connect' to do low-level kernel debugging in windows instance; everytime i add device to instance it shuts down - is this even possible with nova? i can do this in libvirt w/out issue | |
| 18:08:20 | Sundar_ | fried_rice, efried: Do you have a moment? | |
| 18:08:41 | fried_rice | Sundar_: sure. | |
| 18:08:51 | openstackgerrit | Merged openstack/nova master: Make CELL_TIMEOUT a constant https://review.openstack.org/594570 | |
| 18:08:55 | fried_rice | Sundar_: I *am* trying to get to reviewing your spec really soon | |
| 18:09:09 | fried_rice | Sundar_: I've just been inundated with other things. | |
| 18:09:21 | Sundar_ | In the scheduling spec, we defined a way to extend the granular resource request syntax to support Cyborg: https://git.openstack.org/cgit/openstack/cyborg/tree/doc/specs/rocky/cyborg-nova-sched.rst?h=refs/changes/17/554717/18#n315 | |
| 18:09:43 | Sundar_ | Just want to make sure you have reviewed it. | |
| 18:10:30 | Sundar_ | fied_rice: NP :) I understand. We all have to roll on different tracks at the same time. | |
| 18:11:18 | Sundar_ | And I may have added yet another track for you to roll on. | |
| 18:12:36 | fried_rice | Sundar_: I remember seeing this and being okay with it, but I would have to do a major context switch to be able to discuss it intelligently at this point. | |
| 18:14:16 | fried_rice | I noticed you said the extended keys aren't interpreted by nova. Presumably that means they're interpreted by cyborg. How does that information get to cyborg? And does it translate to some new/different placement syntax? I assume not, that it just tells cyborg how to do certain things. | |
| 18:17:32 | Sundar_ | fried_rice: Good to know that it is accepted. Yes, they will be interpreted by Cyborg (not sure if os-acc will also interpret it). The extra specs in the request are passed from Nova virt to os-acc to Cyborg. | |
| 18:17:44 | Sundar_ | No new placement APIs needed. | |
| 18:19:18 | melwitt | I have to run to an appointment, will be back later | |
| 18:34:16 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge security groups extension response into server view builder https://review.openstack.org/585475 | |
| 18:35:49 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge extended_status extension response into server view builder https://review.openstack.org/592092 | |
| 18:41:41 | openstackgerrit | Merged openstack/nova master: tests: Create functional libvirt test base class https://review.openstack.org/407055 | |
| 18:50:31 | mriedem | dansmith: i haven't seen that functional test failure | |
| 19:00:55 | mriedem | dansmith: umm, could https://review.openstack.org/#/c/407055/ that just merged be a regression? | |
| 19:02:44 | mriedem | or the change below it | |
| 19:02:55 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22FileNotFoundError%3A%20%5BErrno%202%5D%20No%20such%20file%20or%20directory%3A%20'%2Fhome%2Fzuul%2Fsrc%2Fgit.openstack.org%2Fopenstack%2Fnova%2Finstances%2F.directio.test'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 19:03:28 | mriedem | just started | |
| 19:03:32 | mriedem | so those are probably it | |
| 19:05:16 | mriedem | https://bugs.launchpad.net/nova/+bug/1788922 | |
| 19:05:17 | openstack | Launchpad bug 1788922 in OpenStack Compute (nova) "SRIOVServersTest.test_create_server_with_VF intermittently fails due to "FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'"" [Undecided,New] | |
| 19:07:15 | mriedem | -1ed https://review.openstack.org/#/c/564978/ which it looks like you and melwitt were talking about - the change has no context | |
| 19:07:20 | mriedem | "this fixes something" | |
| 19:14:40 | mriedem | oh you know, this func test might only be failing on one node provider | |
| 19:14:52 | mriedem | but that means we aren't mocking something out properly and accidentally passing on other hosts | |
| 19:14:55 | mriedem | maybe | |
| 19:23:52 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] split gigantor SQL query, add logging https://review.openstack.org/590041 | |
| 19:49:35 | mriedem | melwitt: question in https://review.openstack.org/#/c/540258/ | |
| 20:09:30 | openstackgerrit | Merged openstack/nova master: Stash the cell uuid on the context when targeting https://review.openstack.org/594571 | |
| 20:11:44 | mriedem | melwitt: have you thought about an agenda/schedule for the ptg? just thought of it since gorka was asking in the thread on cross-cell migration | |
| 20:23:53 | mriedem | dansmith: i must not be understanding this comment https://review.openstack.org/#/c/592698/12/nova/tests/unit/compute/test_multi_cell_list.py@118 | |
| 20:24:34 | dansmith | mriedem: copied from L89 above, and yes | |
| 20:25:35 | dansmith | it means that since key0 is the same on both it should actually sort the two instances by key1 | |
| 20:25:49 | dansmith | but since I copied it for this test and only have one instance it obviously doesn't need to be htere | |
| 20:26:01 | mriedem | yeah... | |
| 20:26:02 | mriedem | i was like, | |
| 20:26:07 | mriedem | but it's only 1 record... | |
| 20:26:37 | mriedem | cleanup patch on top of the series? | |
| 20:27:15 | dansmith | whatever you want | |
| 20:28:17 | dansmith | if we're blocked on that functional test, I might as well just fix it | |
| 20:29:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698 | |
| 20:29:57 | openstackgerrit | Dan Smith proposed openstack/nova master: List instances from all cells explicitly https://review.openstack.org/593717 | |
| 20:29:58 | openstackgerrit | Dan Smith proposed openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131 | |