| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-08 | |||
| 12:44:47 | openstackgerrit | Takahito Hirose proposed openstack/python-novaclient master: api_version decorator becomes an error in Python 3.5.0. https://review.openstack.org/564702 | |
| 13:16:59 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Deprecate support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 13:17:56 | kashyap | mriedem: When you get a minute, I read the scrollback from yesterday here, and went with the: "deprecate in Rocky and hard-fail in Stein" | |
| 13:19:22 | kashyap | I don't think I got the "assert_called_once_with" quite right here: https://review.openstack.org/#/c/565242/5/nova/tests/unit/virt/libvirt/test_driver.py@6623 | |
| 13:20:16 | wznoinsk | mriedem, hi | |
| 13:20:42 | zzzeek | jaypipes: what would cause lock wait timeout exceeded for an INSERT? | |
| 13:21:29 | jaypipes | zzzeek: another thread executing LOCK TABLES <table>? | |
| 13:21:50 | zzzeek | jaypipes: just that? nothing more subtle? ceilometer is doing it | |
| 13:22:32 | jaypipes | zzzeek: got a log output or something more for me? :) | |
| 13:22:58 | zzzeek | jaypipes: i have the error message and the query id have to spend time looking for the logs. | |
| 13:23:10 | zzzeek | jaypipes: but there's nothing like, the "auto increment" feature or somethign locks | |
| 13:23:14 | jaypipes | zzzeek: the only other thing I can think of would be threads attempting to execute huge transactions. | |
| 13:23:26 | jaypipes | zzzeek: all concurrently | |
| 13:23:37 | zzzeek | jaypipes: right and then innodb locks ...a set of potential rows? | |
| 13:24:31 | jaypipes | zzzeek: no, autoinc won't produce that lock wait timeout generally, unless like I said, you have multiple threads simultaneously attempting to commit huge transactions (with thousands or tens of thousands of data modifications in each trx) | |
| 13:25:05 | jaypipes | zzzeek: yes, innodb will do its gap locks if the PK isn't autoinc. | |
| 13:25:11 | zzzeek | jaypipes: ok but in that csae, what is the lock that the INSERT is waiting for? OK gap locks. got it | |
| 13:25:25 | jaypipes | zzzeek: but again... you need some serious concurrency and huge trx to see this impact IME | |
| 13:25:38 | zzzeek | jaypipes: this is a load test | |
| 13:25:39 | BlackDex | Hello there. Does queens support active/active rw in multiple instance using ceph storage and the correct kvm version | |
| 13:25:41 | BlackDex | ? | |
| 13:26:18 | jaypipes | zzzeek: my guess would be ceilometer is attempting to commit batches of record changes. maybe try reducing the length of time between those commits? | |
| 13:26:39 | zzzeek | jaypipes: I dont even know wehre ceilometer's database code is | |
| 13:26:50 | jaypipes | zzzeek: what version? | |
| 13:26:55 | zzzeek | master | |
| 13:27:09 | jaypipes | zzzeek: lemme grep and see. | |
| 13:27:21 | jaypipes | zzzeek: been a very long time since I looked at ceilometer. | |
| 13:27:34 | zzzeek | [classic@photon2 ceilometer]$ | |
| 13:27:34 | zzzeek | jaypipes: $ find ceilometer/ -name "*.py" -exec grep -l sql {} \; | |
| 13:27:36 | zzzeek | zero | |
| 13:27:45 | zzzeek | they've hidden it | |
| 13:28:16 | jaypipes | zzzeek: gnocchi is now the backend data storage for meters, though, right? | |
| 13:28:19 | zzzeek | that's pretty impressive the string "sql" does not appear in their source base at all | |
| 13:28:24 | jaypipes | zzzeek: ceilometer is just the polling thing right? | |
| 13:28:39 | zzzeek | jaypipes: right. but the log is the "ceilometer agent-notification" | |
| 13:28:41 | jaypipes | zzzeek: https://github.com/openstack/ceilometer/blob/master/ceilometer/gnocchi_client.py | |
| 13:30:11 | zzzeek | jaypipes: table name is "event" | |
| 13:30:19 | zzzeek | jaypipes: isn't that the old mysql driver? | |
| 13:30:29 | jaypipes | zzzeek: no idea :( | |
| 13:30:32 | zzzeek | jaypipes: ok | |
| 13:33:07 | jaypipes | zzzeek: is this happening in like a tempest run or something? or is this in a prod env? | |
| 13:33:31 | jaypipes | zzzeek: https://github.com/openstack/ceilometer/blob/master/ceilometer/polling/manager.py#L46 <-- maybe try setting that to False and seeing if lock wait timeouts go down (due to smaller trx sizes) | |
| 13:33:43 | zzzeek | jaypipes: top seekrit :) | |
| 13:34:00 | zzzeek | jaypipes: dont worry, you've been a great help :) | |
| 13:34:20 | zzzeek | jaypipes: the issue here is writing to an "event" table and I think that is the ancient mysql backend | |
| 13:34:48 | jaypipes | zzzeek: yeah, sounds like it. | |
| 13:36:30 | jaypipes | zzzeek: you sure this is master? https://github.com/openstack/ceilometer/commit/9323f07f977f320882f8b536c3b54835274826fc | |
| 13:37:05 | zzzeek | jaypipes: in the error I'm seeing? it is purportedly at least queens | |
| 13:39:10 | jaypipes | zzzeek: I think you may need to reach out to jd__. | |
| 13:39:18 | zzzeek | jaypipes: yep | |
| 13:40:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NUMATopologyLimits.obj_from_db_obj' https://review.openstack.org/537412 | |
| 13:40:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_to_dict' functions https://review.openstack.org/537413 | |
| 13:40:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.openstack.org/537414 | |
| 13:58:02 | artom | dansmith, is there a trick in func tests to start a compute service with a specific version? I *could* mock object.Service.get_by_compute_host, which is what I want to return an "older" Service, but then it messes up other stuff that calls it | |
| 14:11:40 | kashyap | Cany unit test experts comment on what I can do differenlty here: https://review.openstack.org/#/c/565242/5/nova/tests/unit/virt/libvirt/test_driver.py@6623 | |
| 14:12:03 | kashyap | When I assert that, I get a: "AssertionError: Expected 'warning' to be called once. Called 4 times." | |
| 14:13:31 | dansmith | artom: yeah, that'd be failure-prone.. I'd just start it and then update its record manually | |
| 14:13:40 | dansmith | also, mocks in functional tests aren't good | |
| 14:14:08 | artom | dansmith, yeah... | |
| 14:14:22 | artom | dansmith, if you're up for it I can WIP-up what I got and you can give early feedback? | |
| 14:14:39 | dansmith | artom: okay | |
| 14:14:44 | artom | I'm basing it on existing tests, so... | |
| 14:15:06 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Service version check for NUMA live migration https://review.openstack.org/566723 | |
| 14:15:11 | artom | dansmith, ^^ | |
| 14:17:09 | dansmith | artom: you want me to comment about the mock then? | |
| 14:17:29 | artom | I want you to be happy :) | |
| 14:17:51 | artom | dansmith, in seriousness though, just... if I'm way off base, let me know so I can adjust my approach right away, instead of going down this rabbit hole | |
| 14:19:27 | dansmith | artom: just commented what I said above but with pseudocode | |
| 14:19:38 | dansmith | does that make sense? | |
| 14:20:00 | kashyap | Can anyone remind me again, mentioning text like these in Config file help is OK, right? | |
| 14:20:03 | kashyap | "Note that support for Intel CMT events (`cmt`, `mbmbt`, `mbml`) is deprecated in Nova, and will be removed in "Stein" release." | |
| 14:20:17 | kashyap | Because the config file help text is per release, it is okay... | |
| 14:20:20 | openstackgerrit | Julia Kreger proposed openstack/nova master: ironic: add instance_uuid before any other spawn activity https://review.openstack.org/563722 | |
| 14:20:27 | artom | dansmith, ah, yeah, that's probably smarter. Cheers! | |
| 14:21:43 | artom | kashyap, you can iterate through the calls to see what they were, maybe something else is logging at warn level that you haven't considered? | |
| 14:22:34 | kashyap | artom: This is purely help text. All I am asking is, is it okay to call out future release names like what I noted above is okay in the help text | |
| 14:22:54 | artom | kashyap, I was answered your earlier question about the calls assetion :) | |
| 14:23:03 | artom | *answering | |
| 14:23:05 | artom | *asserting | |
| 14:23:14 | kashyap | artom: Aaah, darn. My memory is like a gold fish | |
| 14:23:35 | artom | kashyap, https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.mock_calls | |
| 14:23:46 | kashyap | artom: It is the specific warning: https://review.openstack.org/#/c/565242/5/nova/tests/unit/virt/libvirt/test_driver.py@6623 | |
| 14:23:49 | kashyap | On that line | |
| 14:24:17 | artom | kashyap, yeah, and as a debugging aid I'm suggesting you examine what the calls were | |
| 14:24:25 | kashyap | Which should log this: https://review.openstack.org/#/c/565242/5/nova/virt/libvirt/driver.py@4799 | |
| 14:24:45 | kashyap | artom: Yep, digging...Thx for the (non-null) pointer | |
| 14:24:48 | jroll | kashyap: if there's other warn calls happening, you could also do mock_warn.assert_has_calls([mock.call('Monitoring...')]) | |
| 14:25:22 | artom | kashyap, I'm thinking something else called LOG.warning somewhere along that test's execution | |
| 14:25:38 | kashyap | jroll: I don't think it's other warn calls, because I was calling it with a specific warning message | |
| 14:25:46 | kashyap | jroll: But let me try your trick. | |
| 14:25:46 | artom | If those were legit calls, you can adjust your tests to only assert on the call you care about | |
| 14:25:52 | jroll | kashyap: what artom said :) | |
| 14:25:55 | artom | If they weren't legit, you fix your code :) | |
| 14:26:14 | jroll | ++ | |
| 14:26:25 | kashyap | Thx for the comments, folks | |
| 14:27:06 | artom | Btw, asserting on *log messages* is horrible testing practive | |
| 14:27:08 | artom | *practice | |
| 14:27:20 | artom | I know Nova is side-effect land, so we don't have much choice | |
| 14:27:38 | artom | But in an ideal world, we'd be asserting stuff on output, given a certain input | |
| 14:27:44 | kashyap | artom: I was actually asked to do it. I firt did the self.assertTrue(mock_warn.called) | |