| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-30 | |||
| 13:55:06 | edleafe | bauzas: in meetings for the next few hours. Will review after | |
| 13:55:11 | andymccr | but booting an instance will work fine if you try manually after the failure, although it fails consistently since those patches merged | |
| 13:55:31 | mriedem | andymccr: i'm not sure i understand - which option? | |
| 13:56:00 | mriedem | andymccr: do you mean discover_hosts_in_cells_interval ? | |
| 13:56:30 | mriedem | the default on that, -1, means it won't run as a periodic task | |
| 13:56:32 | andymccr | mriedem: yeah - i figured it could be a race condition on that - but since we're setting it to -1 for the gate and then manually running host_discover i figured it should work | |
| 13:56:40 | mriedem | so you'd have to run nova-manage cell_v2 discover_hosts manually | |
| 13:56:41 | kaisers1 | Hi! I've a fix for an old and rather nasty driver bug, the fix (https://review.openstack.org/#/c/432344/) has been through a few cycles with mbooth and now requires some Core love. Would be great to see this finally land. :) | |
| 13:56:54 | mriedem | andymccr: is discover_hosts running before the server boot failure? | |
| 13:57:20 | andymccr | mriedem: yeah. the boot failure happens as part of the tempest run we do after nova is setup | |
| 13:57:41 | mriedem | andymccr: do you have logs for one of these failures? | |
| 13:57:45 | andymccr | mriedem: sure | |
| 13:57:53 | andymccr | http://logs.openstack.org/21/468321/2/check/gate-openstack-ansible-os_nova-ansible-func-ubuntu-xenial/2b36790/logs/openstack/openstack1/nova/nova-scheduler.log.txt.gz#_2017-05-30_12_00_41_488 | |
| 13:58:15 | andymccr | thats the scheduler log although i should have any other nova logs that would be helpful | |
| 13:58:26 | andymccr | appreciate the help! | |
| 13:58:39 | mriedem | andymccr: ^ is a different thing | |
| 13:58:43 | mriedem | which i have a patch for | |
| 13:58:58 | mriedem | https://review.openstack.org/#/c/454426/ | |
| 13:59:33 | andymccr | mriedem: ahh awesome - i'll test that then. thanks! | |
| 13:59:52 | mriedem | that won't fix a boot failure, but it fixes the ugly stacktrace on startup | |
| 14:00:07 | artom | Hrrm, what if Tempest got a "synchronized" test decorator - ie, run this test by itself | |
| 14:00:29 | andymccr | ahh so the stack trace is a red herring | |
| 14:00:40 | artom | For evacuation testing, for example (to sort of continue the previous discussion) | |
| 14:02:10 | mriedem | artom: they already have a lock fixture in tempest | |
| 14:02:21 | mriedem | we use it for the default quota class tests | |
| 14:02:44 | mriedem | https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_quotas.py#L32 | |
| 14:03:23 | artom | Ah, but you have to specify a lock name | |
| 14:03:26 | artom | (Right?) | |
| 14:03:40 | artom | So if we want to use it for evacuation tests, we'd have to modify every other test to lock them out | |
| 14:03:41 | mriedem | yes, and define that lock in other tests too, that you need to lock with | |
| 14:03:51 | mriedem | correct, or do something sneaky in the base compute class | |
| 14:04:02 | openstackgerrit | Mikhail Feoktistov proposed openstack/nova master: Add is_vz_container function https://review.openstack.org/445947 | |
| 14:04:15 | artom | At least it shows they're amenable to that sort of stuff | |
| 14:04:36 | artom | Maybe a "synchronized" thing would fly | |
| 14:04:46 | jaypipes | edleafe, bauzas, cdent, mriedem: fancy doing a quick pow-wow about scheduler topics given we missed yesterday's meeting? | |
| 14:04:56 | bauzas | why not | |
| 14:05:02 | bauzas | I just provide some new revs | |
| 14:05:20 | mriedem | andymccr: is there a log with the tempest console logs in it from the failed tempest run? http://logs.openstack.org/21/468321/2/check/gate-openstack-ansible-os_nova-ansible-func-ubuntu-xenial/2b36790/console.html#_2017-05-30_12_03_59_084558 | |
| 14:05:26 | bauzas | about a thought I had about not providing instance UUIDs but rather a list of Spec objects | |
| 14:05:48 | mriedem | bauzas: so just totally explode rpc then? | |
| 14:05:52 | cdent | jaypipes: ten minutes ago edleafe said he was in meetings for next few hours, and I'm out in the big city on yet another dead car curn | |
| 14:05:57 | cdent | mriedem: that's what I thought too | |
| 14:06:00 | bauzas | mriedem: we already do that for the conductor | |
| 14:06:20 | bauzas | mriedem: as dan wrote it :) | |
| 14:06:23 | mriedem | we should do it less if we can help it | |
| 14:06:42 | bauzas | as I said in the commit msg, it's not really a performance issue | |
| 14:06:55 | mriedem | how do you know? | |
| 14:06:55 | bauzas | only if you're asking for more than one instance | |
| 14:07:00 | mriedem | right, which people do | |
| 14:07:02 | mriedem | people/systems | |
| 14:07:27 | bauzas | mriedem: see code, you'll see api calling conductor with passing a list of buildrequests and request_specs | |
| 14:07:34 | andymccr | double checking that. it looks like we might not be copying it for the gate purposes. will run a test and check! | |
| 14:08:03 | bauzas | not sure why it would be a problem now with the scheduler API, while it wasn't a problem with the conductor one | |
| 14:08:34 | cdent | bauzas: do you want to do it for sake of consistency, or is there some other reason? | |
| 14:08:59 | mriedem | bauzas: it might have been a problem that no one directly complained about, but people have complained about the memory usage bloat in nova over the last few releases | |
| 14:09:07 | bauzas | cdent: because of the comments I made in PS7 | |
| 14:09:16 | mriedem | and i have to assume some of that is tied to us passing big lists of big objects around and assuming it's fine | |
| 14:09:38 | bauzas | mriedem: you mean RPC payload size, or memory consumption for services ? | |
| 14:09:41 | mriedem | reqspecs were new in mitaka right? and people are just still getting to mitaka and beyond | |
| 14:09:48 | mriedem | both | |
| 14:10:42 | bauzas | mriedem: again, I'm still not convinced it's a performance issue | |
| 14:10:58 | bauzas | we're passing way more huge objects thru RPC | |
| 14:11:16 | bauzas | while the Spec object is fairly small, except the flavor bit | |
| 14:11:21 | mriedem | which patch are you even referring to? | |
| 14:11:40 | mriedem | https://review.openstack.org/#/c/465171/ ? | |
| 14:11:46 | bauzas | mriedem: I made a comment on PS7 https://review.openstack.org/#/c/465171/7 | |
| 14:11:54 | bauzas | mriedem: and I just provided a PS8 | |
| 14:12:34 | mriedem | so, | |
| 14:12:51 | mriedem | how did we go from agreeing to pass just a list of instance uuids over rpc last week, to now passing a list of request spec objects? | |
| 14:13:59 | mriedem | i think that was a hangout between myself, superdan, jaypipes and maybe edleafe, b/c cdent had to drop and we didn't recap our discussion anywhere | |
| 14:14:03 | mriedem | and bauzas was on PTO | |
| 14:14:38 | bauzas | that's a typical distributed problem :) | |
| 14:14:54 | bauzas | anyway, if folks thought about that... | |
| 14:14:58 | jaypipes | mriedem: we didn't agree to pass a list of reqspec objects. | |
| 14:15:09 | mriedem | jaypipes: i know, we never even talked about that | |
| 14:15:12 | bauzas | tbc, I wasn't aware of that agreement | |
| 14:15:35 | mriedem | jaypipes: we agreed to pass a list of instance uuids over the rpc method, not embedded in the request spec that's already getting passed, yes? | |
| 14:15:38 | bauzas | I just provided a patch for that purpose because I thought it was nicier from an interface PoV | |
| 14:15:45 | jaypipes | bauzas: Sylvain, you were on the hangout where we discussed doing the instance_uuids parameter. | |
| 14:15:53 | jaypipes | bauzas: it was right before you went on PTO | |
| 14:16:02 | bauzas | jaypipes: we agreed on making the claims | |
| 14:16:02 | jaypipes | mriedem: yes | |
| 14:16:18 | bauzas | I don't remember if we agreed on the interface | |
| 14:16:51 | bauzas | anyway, looks like the general direction was on another trend | |
| 14:16:57 | bauzas | edleafe: I'll provide a PS9 | |
| 14:17:08 | bauzas | because I still have concerns with PS7 | |
| 14:17:17 | jaypipes | bauzas: yes, I'm pretty sure we agreed to pass the instance_uuids as a new param on RPC. I remember that because I had said changing the RequestSpec object definition was cleaner but was persuaded that wasn't a good idea to add non-DB fields to an object definition. | |
| 14:17:41 | mriedem | maybe we should make a rule that anything discussed while one of the 5 people coding this is out, that we actually communicate decisions back into the mailing list, or something :) | |
| 14:17:58 | jaypipes | mriedem: hindsight 20/20 | |
| 14:18:17 | bauzas | so | |
| 14:19:10 | bauzas | I'm not particularly tied to my PS8 | |
| 14:19:29 | bauzas | I just think https://review.openstack.org/#/c/469037/2 is worth it anyway | |
| 14:19:40 | mriedem | ok, good, so 1 reqspec and list of instance uuids over the rpc method, not within the reqspec | |
| 14:20:26 | mriedem | bauzas: isn't that https://blueprints.launchpad.net/nova/+spec/request-spec-use-by-compute which hasn't been started yet? | |
| 14:20:38 | bauzas | I'm just surprised we had a consensus on an interface that made it very tied with how we use placement | |
| 14:20:45 | bauzas | but okay | |
| 14:21:18 | bauzas | mriedem: yup, it can be supported by that bp | |
| 14:22:09 | mriedem | andymccr: circling back, ok so i see at least one tempest test failed - but it's unclear why really, so if there are tempest console logs in a separate file it'd be cool to see those to debug | |
| 14:22:15 | mriedem | andymccr: if you're working on that, ack | |
| 14:23:09 | andymccr | mriedem: got it. thanks! I'm running a test manually and i'll get some logs and circle back if i cant figure it out, but tl;dr the hostmappingnotfound error is not the cause of the issue | |