| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-14 | |||
| 18:41:48 | superdan | mriedem: what does the failure look like? | |
| 18:41:50 | mriedem | oslo.service updates went into upper-constraints then too | |
| 18:42:06 | leakypipes | cdent, figleaf, mriedem, superdan: looks like I fixed the issues with the num_instances > 1 tempest test failures in the placement-claims series. | |
| 18:42:09 | mriedem | https://bugs.launchpad.net/cinder/+bug/1694509 | |
| 18:42:10 | openstack | Launchpad bug 1694509 in Cinder "eventlet errors and cannot switch to a different thread" [High,Confirmed] | |
| 18:42:23 | mriedem | http://logs.openstack.org/49/483149/2/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/d37b26f/logs/screen-c-vol.txt.gz#_Jul_13_20_07_49_514341 | |
| 18:42:50 | mriedem | there is also a GMR in this other one http://logs.openstack.org/00/469800/31/check/gate-tempest-dsvm-cells-ubuntu-xenial/7156eed/logs/screen-c-vol.txt.gz | |
| 18:43:02 | smcginnis | Just looked at this one: http://logs.openstack.org/62/468262/4/gate/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/e3ad57c/logs/screen-c-vol.txt.gz#_Jul_14_17_05_58_531817 | |
| 18:43:07 | mriedem | http://logs.openstack.org/00/469800/31/check/gate-tempest-dsvm-cells-ubuntu-xenial/7156eed/logs/screen-c-vol.txt.gz#_Jul_14_13_59_21_489470 | |
| 18:43:11 | smcginnis | Slightly different signature. | |
| 18:44:14 | mriedem | so, | |
| 18:44:16 | mriedem | cinder uses taskflow, | |
| 18:44:22 | mriedem | and it looks like taskflow uses futursit | |
| 18:44:25 | mriedem | *futurist | |
| 18:44:32 | mriedem | and futurist had an update in upper-constraints on 7/11 | |
| 18:44:57 | smcginnis | Hmm, suspicious. | |
| 18:45:20 | mriedem | https://review.openstack.org/#/c/482273/ | |
| 18:45:59 | mriedem | clarkb: ^ | |
| 18:46:36 | mriedem | maybe we should try blacklisting that and see if things trend back down | |
| 18:46:57 | superdan | sounds plausible to me | |
| 18:47:23 | mriedem | https://github.com/openstack/futurist/compare/1.1.0...1.2.0 | |
| 18:47:35 | smcginnis | mriedem: It might be worth a shot to at least get more data around this and prove or eliminate possibilities. | |
| 18:47:55 | mriedem | the only thing in the 1.2.0 release for futurist is reqs updates https://github.com/openstack/futurist/compare/1.1.0...1.2.0 | |
| 18:48:02 | mriedem | but eventlet is in there | |
| 18:48:20 | mriedem | although i'm not sure that would make a difference | |
| 18:48:22 | mriedem | since we just use u-c | |
| 18:49:38 | mriedem | this module could also be fishy https://github.com/openstack/cinder/blob/master/cinder/coordination.py | |
| 18:49:47 | mriedem | uses eventlet and tooz | |
| 18:49:53 | mriedem | tooz was also updated in u-c on 7/11 | |
| 18:54:27 | mriedem | https://github.com/openstack/tooz/compare/1.57.3...1.57.4 | |
| 18:54:40 | mriedem | https://github.com/openstack/tooz/commit/c94b2a39b3bc484eff6a9c3c9dc094bbfae070c4 | |
| 18:54:41 | mriedem | yay | |
| 18:56:32 | superdan | mriedem: "replace lock code" should be an auto--1 regex | |
| 18:56:44 | superdan | maybe also .*with etcd.* | |
| 18:57:40 | mriedem | and https://github.com/openstack/tooz/commit/7987f4455a7ff4e75d64c46a42d50017f53bfc9e | |
| 18:57:59 | mriedem | superdan: i'm not following you | |
| 18:58:18 | mriedem | however, i did find that this cinder.coordination code that uses tooz and is in the create volume GMR stacktrace, | |
| 18:58:19 | mriedem | http://logs.openstack.org/49/483149/2/check/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/d37b26f/logs/screen-c-vol.txt.gz#_Jul_13_19_55_02_134315 | |
| 18:58:22 | superdan | mriedem: I'm making a joke that "replace lock code" is something that probably deserves a lot of scrutiny | |
| 18:58:23 | mriedem | is uses the tooz etc3 backend | |
| 18:58:34 | mriedem | no way due throw it in \...q | |
| 18:58:37 | mriedem | it's gnarly | |
| 18:58:40 | mriedem | *dude | |
| 19:04:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'keymap' options https://review.openstack.org/483994 | |
| 19:09:42 | openstackgerrit | Octave Orgeron proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643 | |
| 19:14:12 | mriedem | https://review.openstack.org/484019 | |
| 19:14:13 | mriedem | clarkb: ^ | |
| 19:14:17 | mriedem | dims: ^ | |
| 19:15:26 | dims | mriedem : clarkb : +2 | |
| 19:15:45 | mriedem | ok i have a good e-r query now | |
| 19:16:06 | PTCbro | Hello! I have a question about instance network cache. In neutronv2/api.py I notice _get_instance_nw_info calls _build_network_info_model which queries neutron for the port info. However it also calls _gather_port_ids_and_networks which asks the cache for the current port_ids. If the cache is empty, then _build_network_info_model will ignore the port info it got from neutron. Thus the empty cache forever remains empty | |
| 19:17:31 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%2Fusr%2Flocal%2Flib%2Fpython2.7%2Fdist-packages%2Ftooz%2Fdrivers%2Fetcd3gw.py%3A113%20in%20acquire%5C%22%20AND%20tags%3A%5C%22screen-c-vol.txt%5C%22&from=7d | |
| 19:17:35 | PTCbro | Is there any way to populate an empty cache? | |
| 20:08:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add tags to instance.create Notification https://review.openstack.org/459493 | |
| 20:08:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Support tag instances when boot(4/4) https://review.openstack.org/469800 | |
| 20:22:24 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova master: [WIP] Getting rid of crud around castellan options https://review.openstack.org/484036 | |
| 20:28:32 | mriedem | \o/ finally got this resize-confirm + quota check novaclient functional test passing https://review.openstack.org/#/c/483559/ | |
| 20:28:41 | mriedem | going to roll a change on top of that to point at the counting quotas series | |
| 20:29:53 | mriedem | melwitt: did you see this in your counting instances change? http://logs.openstack.org/21/416521/60/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/8d48280/logs/screen-n-api.txt.gz?level=TRACE#_Jul_14_16_30_54_208963 | |
| 20:30:58 | melwitt | mriedem: yeah, looking now. don't know what's going on there yet | |
| 20:35:36 | melwitt | I wonder if it's related to the tooz locking thing. it doesn't show anything happening before that other than acquiring/releasing a lock and the test passed on the previous PS | |
| 20:36:13 | melwitt | *before the trace | |
| 20:37:02 | mriedem | the tooz locking thing is specifically hitting cinder because the cinder DLM code uses tooz with the etcd3 backend | |
| 20:37:09 | mriedem | and we don't use tooz | |
| 20:37:12 | mriedem | b/c we're not crazy | |
| 20:37:25 | mriedem | oh shit we do use tooz | |
| 20:37:40 | mriedem | oh only for the ironic hashring stuff | |
| 20:38:10 | smcginnis | :P | |
| 20:38:42 | mriedem | did i say all of these things out loud? | |
| 20:39:06 | melwitt | oh. well, crap | |
| 20:40:24 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ServerPersonalityTestJSON%5C%22%20AND%20message%3A%5C%22Caught%20error%3A%20Circular%20reference%20detected%3A%20ValueError%3A%20Circular%20reference%20detected%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d | |
| 20:40:28 | mriedem | are those all changes in your series alone? | |
| 20:40:59 | mriedem | yeah looks like | |
| 20:41:01 | mriedem | something in your change | |
| 20:41:15 | melwitt | gdi | |
| 20:41:21 | melwitt | okay. there's almost no info in the log | |
| 20:44:07 | mriedem | well, it's got to be something in here https://review.openstack.org/#/c/416521/58..60/nova/objects/instance.py | |
| 20:44:12 | mriedem | it's serializing the server response | |
| 20:44:13 | mriedem | and blowing up | |
| 20:44:55 | mriedem | but no idea what | |
| 20:46:00 | melwitt | I'm googling | |
| 20:50:48 | melwitt | I notice almost all the other methods use the @db.select_db_reader_mode decorator instead of what I used @db_api.pick_context_manager_reader | |
| 20:50:58 | melwitt | dunno if that could be a thing | |
| 20:51:06 | mriedem | well, | |
| 20:51:11 | mriedem | why would it just be that one test too? | |
| 20:51:17 | mriedem | for the personality (injected) files | |
| 20:53:01 | mriedem | https://review.openstack.org/#/c/416521/58..60/nova/conductor/manager.py ? | |
| 20:53:19 | mriedem | no nvm | |
| 20:55:36 | melwitt | it's weird how there's almost no info in the logging of that | |
| 20:57:38 | mriedem | oh it's in File "tempest/lib/services/compute/limits_client.py", line 26, in show_limits | |
| 20:58:43 | mriedem | so something to GET /limits is blowing up | |
| 21:01:52 | mriedem | so that would call QUOTAS.get_project_quotas, | |
| 21:02:10 | mriedem | which would eventually call through to _instances_cores_ram_count | |
| 21:02:24 | mriedem | which calls objects.InstanceList.get_counts | |
| 21:02:44 | melwitt | I think it makes that call with usages=False though which wouldn't count | |
| 21:03:05 | melwitt | unless this is a call to used_limits rather than limits | |
| 21:03:15 | melwitt | they'e both /limits | |
| 21:03:24 | mriedem | yeah it's an extension | |
| 21:03:28 | melwitt | so I guess must be | |
| 21:03:49 | mriedem | are the results from the db queries in the instance object actual primitives? | |