| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-08 | |||
| 13:25:41 | BlackDex | ? | |
| 13:26:18 | jaypipes | zzzeek: my guess would be ceilometer is attempting to commit batches of record changes. maybe try reducing the length of time between those commits? | |
| 13:26:39 | zzzeek | jaypipes: I dont even know wehre ceilometer's database code is | |
| 13:26:50 | jaypipes | zzzeek: what version? | |
| 13:26:55 | zzzeek | master | |
| 13:27:09 | jaypipes | zzzeek: lemme grep and see. | |
| 13:27:21 | jaypipes | zzzeek: been a very long time since I looked at ceilometer. | |
| 13:27:34 | zzzeek | [classic@photon2 ceilometer]$ | |
| 13:27:34 | zzzeek | jaypipes: $ find ceilometer/ -name "*.py" -exec grep -l sql {} \; | |
| 13:27:36 | zzzeek | zero | |
| 13:27:45 | zzzeek | they've hidden it | |
| 13:28:16 | jaypipes | zzzeek: gnocchi is now the backend data storage for meters, though, right? | |
| 13:28:19 | zzzeek | that's pretty impressive the string "sql" does not appear in their source base at all | |
| 13:28:24 | jaypipes | zzzeek: ceilometer is just the polling thing right? | |
| 13:28:39 | zzzeek | jaypipes: right. but the log is the "ceilometer agent-notification" | |
| 13:28:41 | jaypipes | zzzeek: https://github.com/openstack/ceilometer/blob/master/ceilometer/gnocchi_client.py | |
| 13:30:11 | zzzeek | jaypipes: table name is "event" | |
| 13:30:19 | zzzeek | jaypipes: isn't that the old mysql driver? | |
| 13:30:29 | jaypipes | zzzeek: no idea :( | |
| 13:30:32 | zzzeek | jaypipes: ok | |
| 13:33:07 | jaypipes | zzzeek: is this happening in like a tempest run or something? or is this in a prod env? | |
| 13:33:31 | jaypipes | zzzeek: https://github.com/openstack/ceilometer/blob/master/ceilometer/polling/manager.py#L46 <-- maybe try setting that to False and seeing if lock wait timeouts go down (due to smaller trx sizes) | |
| 13:33:43 | zzzeek | jaypipes: top seekrit :) | |
| 13:34:00 | zzzeek | jaypipes: dont worry, you've been a great help :) | |
| 13:34:20 | zzzeek | jaypipes: the issue here is writing to an "event" table and I think that is the ancient mysql backend | |
| 13:34:48 | jaypipes | zzzeek: yeah, sounds like it. | |
| 13:36:30 | jaypipes | zzzeek: you sure this is master? https://github.com/openstack/ceilometer/commit/9323f07f977f320882f8b536c3b54835274826fc | |
| 13:37:05 | zzzeek | jaypipes: in the error I'm seeing? it is purportedly at least queens | |
| 13:39:10 | jaypipes | zzzeek: I think you may need to reach out to jd__. | |
| 13:39:18 | zzzeek | jaypipes: yep | |
| 13:40:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NUMATopologyLimits.obj_from_db_obj' https://review.openstack.org/537412 | |
| 13:40:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_to_dict' functions https://review.openstack.org/537413 | |
| 13:40:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.openstack.org/537414 | |
| 13:58:02 | artom | dansmith, is there a trick in func tests to start a compute service with a specific version? I *could* mock object.Service.get_by_compute_host, which is what I want to return an "older" Service, but then it messes up other stuff that calls it | |
| 14:11:40 | kashyap | Cany unit test experts comment on what I can do differenlty here: https://review.openstack.org/#/c/565242/5/nova/tests/unit/virt/libvirt/test_driver.py@6623 | |
| 14:12:03 | kashyap | When I assert that, I get a: "AssertionError: Expected 'warning' to be called once. Called 4 times." | |
| 14:13:31 | dansmith | artom: yeah, that'd be failure-prone.. I'd just start it and then update its record manually | |
| 14:13:40 | dansmith | also, mocks in functional tests aren't good | |
| 14:14:08 | artom | dansmith, yeah... | |
| 14:14:22 | artom | dansmith, if you're up for it I can WIP-up what I got and you can give early feedback? | |
| 14:14:39 | dansmith | artom: okay | |
| 14:14:44 | artom | I'm basing it on existing tests, so... | |
| 14:15:06 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Service version check for NUMA live migration https://review.openstack.org/566723 | |
| 14:15:11 | artom | dansmith, ^^ | |
| 14:17:09 | dansmith | artom: you want me to comment about the mock then? | |
| 14:17:29 | artom | I want you to be happy :) | |
| 14:17:51 | artom | dansmith, in seriousness though, just... if I'm way off base, let me know so I can adjust my approach right away, instead of going down this rabbit hole | |
| 14:19:27 | dansmith | artom: just commented what I said above but with pseudocode | |
| 14:19:38 | dansmith | does that make sense? | |
| 14:20:00 | kashyap | Can anyone remind me again, mentioning text like these in Config file help is OK, right? | |
| 14:20:03 | kashyap | "Note that support for Intel CMT events (`cmt`, `mbmbt`, `mbml`) is deprecated in Nova, and will be removed in "Stein" release." | |
| 14:20:17 | kashyap | Because the config file help text is per release, it is okay... | |
| 14:20:20 | openstackgerrit | Julia Kreger proposed openstack/nova master: ironic: add instance_uuid before any other spawn activity https://review.openstack.org/563722 | |
| 14:20:27 | artom | dansmith, ah, yeah, that's probably smarter. Cheers! | |
| 14:21:43 | artom | kashyap, you can iterate through the calls to see what they were, maybe something else is logging at warn level that you haven't considered? | |
| 14:22:34 | kashyap | artom: This is purely help text. All I am asking is, is it okay to call out future release names like what I noted above is okay in the help text | |
| 14:22:54 | artom | kashyap, I was answered your earlier question about the calls assetion :) | |
| 14:23:03 | artom | *answering | |
| 14:23:05 | artom | *asserting | |
| 14:23:14 | kashyap | artom: Aaah, darn. My memory is like a gold fish | |
| 14:23:35 | artom | kashyap, https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.mock_calls | |
| 14:23:46 | kashyap | artom: It is the specific warning: https://review.openstack.org/#/c/565242/5/nova/tests/unit/virt/libvirt/test_driver.py@6623 | |
| 14:23:49 | kashyap | On that line | |
| 14:24:17 | artom | kashyap, yeah, and as a debugging aid I'm suggesting you examine what the calls were | |
| 14:24:25 | kashyap | Which should log this: https://review.openstack.org/#/c/565242/5/nova/virt/libvirt/driver.py@4799 | |
| 14:24:45 | kashyap | artom: Yep, digging...Thx for the (non-null) pointer | |
| 14:24:48 | jroll | kashyap: if there's other warn calls happening, you could also do mock_warn.assert_has_calls([mock.call('Monitoring...')]) | |
| 14:25:22 | artom | kashyap, I'm thinking something else called LOG.warning somewhere along that test's execution | |
| 14:25:38 | kashyap | jroll: I don't think it's other warn calls, because I was calling it with a specific warning message | |
| 14:25:46 | kashyap | jroll: But let me try your trick. | |
| 14:25:46 | artom | If those were legit calls, you can adjust your tests to only assert on the call you care about | |
| 14:25:52 | jroll | kashyap: what artom said :) | |
| 14:25:55 | artom | If they weren't legit, you fix your code :) | |
| 14:26:14 | jroll | ++ | |
| 14:26:25 | kashyap | Thx for the comments, folks | |
| 14:27:06 | artom | Btw, asserting on *log messages* is horrible testing practive | |
| 14:27:08 | artom | *practice | |
| 14:27:20 | artom | I know Nova is side-effect land, so we don't have much choice | |
| 14:27:38 | artom | But in an ideal world, we'd be asserting stuff on output, given a certain input | |
| 14:27:44 | kashyap | artom: I was actually asked to do it. I firt did the self.assertTrue(mock_warn.called) | |
| 14:28:17 | kashyap | And the libvirt driver at least doesn't do a lot of those. Only 3 or so of them -- asserting on log message. | |
| 14:28:39 | kashyap | And if it's any consolation, this warning will be removed in the next release. | |
| 14:28:39 | artom | kashyap, yeah, I fully understand the reality of what we're working on | |
| 14:34:18 | openstackgerrit | Merged openstack/nova master: Changing scheduler sync event from INFO to DEBUG https://review.openstack.org/566392 | |
| 14:58:31 | jmccarthy | mriedem: Thanks again for the help, this experimental review did the trick for me https://review.openstack.org/#/c/566367 | |
| 15:05:22 | mriedem | jmccarthy: np, thanks for verifying it | |
| 15:06:28 | kashyap | jroll: Your trick tells me my assertion was not even in the call list. ("not all found in call list"). | |
| 15:06:52 | jroll | interesting | |
| 15:21:23 | mriedem | jmccarthy: just tested mel's suggestion in there, i'll get that patch updated later today | |
| 15:22:37 | jmccarthy | mriedem: Ah ok cool - I can give that a whirl also to confirm (but probably not till later) | |
| 15:46:33 | openstackgerrit | Eric Fried proposed openstack/nova master: Use query_parameters in granular gabbit https://review.openstack.org/566927 | |
| 15:48:54 | openstackgerrit | Aditya Vaja proposed openstack/nova master: remove IVS plug/unplug as they're moved to separate plugin https://review.openstack.org/534371 | |
| 15:52:06 | stephenfin | jaypipes, bauzas, sean-k-mooney: Fancy taking another swing at https://review.openstack.org/#/c/541290/ today? | |
| 15:53:32 | jaypipes | stephenfin: ack, will do. sorry, was swamped with other stuff yesterday and didn't get to that. | |
| 15:54:04 | stephenfin | no worries :) /me is going through bandwidth-resource-provider again atm | |
| 15:55:43 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add notification support for trusted_certs https://review.openstack.org/563269 | |
| 16:01:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Wait for network-vif-plugged before starting live migration https://review.openstack.org/558001 | |
| 16:01:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add check if neutron "binding-extended" extension is available https://review.openstack.org/523548 | |
| 16:01:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add "bind_ports_to_host" neutron API method https://review.openstack.org/523604 | |
| 16:01:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add "delete_port_binding" network API method https://review.openstack.org/552170 | |