| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-08 | |||
| 12:31:49 | artom | But... the more coverage the better | |
| 12:31:50 | mriedem | melwitt: fyi i've marked https://blueprints.launchpad.net/nova/+spec/convert-consoles-to-objects complete | |
| 12:31:58 | mriedem | artom: sure | |
| 12:32:21 | mriedem | artom: also, a new test would get by the intel 3rd party ci blacklist which is currently based on test names | |
| 12:32:53 | artom | mriedem, oh, hah, ineed. Sneaky :D | |
| 12:33:20 | mriedem | so you'll probably have to do something like have your nova series, and then have a DNM nova patch on top that depends on the tempest change, | |
| 12:33:29 | mriedem | because the intel CI runs on nova changes, but probably not tempest changes | |
| 12:33:52 | sean-k-mooney | i have to run to a meeting but ill be back in an hour or 2 | |
| 12:34:13 | artom | mriedem, yep, and a patch to the intel CI plugin that does stuff like check instance XML | |
| 12:34:29 | artom | mriedem, Or. Or! A patch to nova that adds what I need to the diagnostics API | |
| 12:34:43 | artom | Not sure what would be simpler. | |
| 12:34:56 | mriedem | hypervisor-specific stuff in tempest sucks, | |
| 12:35:04 | mriedem | which is why i suggested adding a new field to the diagnostics api | |
| 12:35:29 | mriedem | alternatively, | |
| 12:35:37 | mriedem | does any of this numa stuff for the guest get modeled in placement? | |
| 12:35:45 | mriedem | as a consumed resource? | |
| 12:35:52 | alex_xu | + | |
| 12:35:57 | artom | mriedem, some of it, I think? But allocations are still on the compute via resource tracker, I believe | |
| 12:35:59 | mriedem | alex likes it | |
| 12:36:16 | mriedem | the numa resource allocations would be on numa resource providers in the compute node provider tree | |
| 12:36:40 | mriedem | but given an instance (consumer) uuid, you can get it's resource class allocations against which providers in placement | |
| 12:36:53 | mriedem | so your test could assert that the instance has NUMA resource class allocations | |
| 12:36:56 | alex_xu | mriedem: my daugther just smash my keyboard... | |
| 12:37:04 | efried | nice find tetsuro | |
| 12:37:04 | mriedem | ha, lot of that going on today | |
| 12:37:22 | artom | alex_xu, she clearly didn't smash hard enough since we can all read what you're typing | |
| 12:37:46 | mriedem | artom: but i don't think the numa stuff is done, or close(?) | |
| 12:37:54 | artom | mriedem, I'm not sure placement would be enough, since we would ideally check specific CPUs, not just quantities | |
| 12:38:03 | artom | And pinning can't be checked at all | |
| 12:38:36 | mriedem | artom: i'm not sure if there is a placement solution in the works for that yet or not, but in that case you could just hack up the diagnostics api | |
| 12:38:41 | artom | mriedem, placement is a pool I swim in, but I still breath through a snorkel, so I don't know what liquid surrounds me | |
| 12:39:12 | artom | At some point I will need to grow gills to breath through the placement pool fluid | |
| 12:39:15 | alex_xu | artom: yea, a little hulk | |
| 12:44:47 | openstackgerrit | Takahito Hirose proposed openstack/python-novaclient master: api_version decorator becomes an error in Python 3.5.0. https://review.openstack.org/564702 | |
| 13:16:59 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Deprecate support for monitoring Intel CMT `perf` events https://review.openstack.org/565242 | |
| 13:17:56 | kashyap | mriedem: When you get a minute, I read the scrollback from yesterday here, and went with the: "deprecate in Rocky and hard-fail in Stein" | |
| 13:19:22 | kashyap | I don't think I got the "assert_called_once_with" quite right here: https://review.openstack.org/#/c/565242/5/nova/tests/unit/virt/libvirt/test_driver.py@6623 | |
| 13:20:16 | wznoinsk | mriedem, hi | |
| 13:20:42 | zzzeek | jaypipes: what would cause lock wait timeout exceeded for an INSERT? | |
| 13:21:29 | jaypipes | zzzeek: another thread executing LOCK TABLES <table>? | |
| 13:21:50 | zzzeek | jaypipes: just that? nothing more subtle? ceilometer is doing it | |
| 13:22:32 | jaypipes | zzzeek: got a log output or something more for me? :) | |
| 13:22:58 | zzzeek | jaypipes: i have the error message and the query id have to spend time looking for the logs. | |
| 13:23:10 | zzzeek | jaypipes: but there's nothing like, the "auto increment" feature or somethign locks | |
| 13:23:14 | jaypipes | zzzeek: the only other thing I can think of would be threads attempting to execute huge transactions. | |
| 13:23:26 | jaypipes | zzzeek: all concurrently | |
| 13:23:37 | zzzeek | jaypipes: right and then innodb locks ...a set of potential rows? | |
| 13:24:31 | jaypipes | zzzeek: no, autoinc won't produce that lock wait timeout generally, unless like I said, you have multiple threads simultaneously attempting to commit huge transactions (with thousands or tens of thousands of data modifications in each trx) | |
| 13:25:05 | jaypipes | zzzeek: yes, innodb will do its gap locks if the PK isn't autoinc. | |
| 13:25:11 | zzzeek | jaypipes: ok but in that csae, what is the lock that the INSERT is waiting for? OK gap locks. got it | |
| 13:25:25 | jaypipes | zzzeek: but again... you need some serious concurrency and huge trx to see this impact IME | |
| 13:25:38 | zzzeek | jaypipes: this is a load test | |
| 13:25:39 | BlackDex | Hello there. Does queens support active/active rw in multiple instance using ceph storage and the correct kvm version | |
| 13:25:41 | BlackDex | ? | |
| 13:26:18 | jaypipes | zzzeek: my guess would be ceilometer is attempting to commit batches of record changes. maybe try reducing the length of time between those commits? | |
| 13:26:39 | zzzeek | jaypipes: I dont even know wehre ceilometer's database code is | |
| 13:26:50 | jaypipes | zzzeek: what version? | |
| 13:26:55 | zzzeek | master | |
| 13:27:09 | jaypipes | zzzeek: lemme grep and see. | |
| 13:27:21 | jaypipes | zzzeek: been a very long time since I looked at ceilometer. | |
| 13:27:34 | zzzeek | jaypipes: $ find ceilometer/ -name "*.py" -exec grep -l sql {} \; | |
| 13:27:34 | zzzeek | [classic@photon2 ceilometer]$ | |
| 13:27:36 | zzzeek | zero | |
| 13:27:45 | zzzeek | they've hidden it | |
| 13:28:16 | jaypipes | zzzeek: gnocchi is now the backend data storage for meters, though, right? | |
| 13:28:19 | zzzeek | that's pretty impressive the string "sql" does not appear in their source base at all | |
| 13:28:24 | jaypipes | zzzeek: ceilometer is just the polling thing right? | |
| 13:28:39 | zzzeek | jaypipes: right. but the log is the "ceilometer agent-notification" | |
| 13:28:41 | jaypipes | zzzeek: https://github.com/openstack/ceilometer/blob/master/ceilometer/gnocchi_client.py | |
| 13:30:11 | zzzeek | jaypipes: table name is "event" | |
| 13:30:19 | zzzeek | jaypipes: isn't that the old mysql driver? | |
| 13:30:29 | jaypipes | zzzeek: no idea :( | |
| 13:30:32 | zzzeek | jaypipes: ok | |
| 13:33:07 | jaypipes | zzzeek: is this happening in like a tempest run or something? or is this in a prod env? | |
| 13:33:31 | jaypipes | zzzeek: https://github.com/openstack/ceilometer/blob/master/ceilometer/polling/manager.py#L46 <-- maybe try setting that to False and seeing if lock wait timeouts go down (due to smaller trx sizes) | |
| 13:33:43 | zzzeek | jaypipes: top seekrit :) | |
| 13:34:00 | zzzeek | jaypipes: dont worry, you've been a great help :) | |
| 13:34:20 | zzzeek | jaypipes: the issue here is writing to an "event" table and I think that is the ancient mysql backend | |
| 13:34:48 | jaypipes | zzzeek: yeah, sounds like it. | |
| 13:36:30 | jaypipes | zzzeek: you sure this is master? https://github.com/openstack/ceilometer/commit/9323f07f977f320882f8b536c3b54835274826fc | |
| 13:37:05 | zzzeek | jaypipes: in the error I'm seeing? it is purportedly at least queens | |
| 13:39:10 | jaypipes | zzzeek: I think you may need to reach out to jd__. | |
| 13:39:18 | zzzeek | jaypipes: yep | |
| 13:40:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NUMATopologyLimits.obj_from_db_obj' https://review.openstack.org/537412 | |
| 13:40:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_to_dict' functions https://review.openstack.org/537413 | |
| 13:40:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.openstack.org/537414 | |
| 13:58:02 | artom | dansmith, is there a trick in func tests to start a compute service with a specific version? I *could* mock object.Service.get_by_compute_host, which is what I want to return an "older" Service, but then it messes up other stuff that calls it | |
| 14:11:40 | kashyap | Cany unit test experts comment on what I can do differenlty here: https://review.openstack.org/#/c/565242/5/nova/tests/unit/virt/libvirt/test_driver.py@6623 | |
| 14:12:03 | kashyap | When I assert that, I get a: "AssertionError: Expected 'warning' to be called once. Called 4 times." | |
| 14:13:31 | dansmith | artom: yeah, that'd be failure-prone.. I'd just start it and then update its record manually | |
| 14:13:40 | dansmith | also, mocks in functional tests aren't good | |
| 14:14:08 | artom | dansmith, yeah... | |
| 14:14:22 | artom | dansmith, if you're up for it I can WIP-up what I got and you can give early feedback? | |
| 14:14:39 | dansmith | artom: okay | |
| 14:14:44 | artom | I'm basing it on existing tests, so... | |
| 14:15:06 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Service version check for NUMA live migration https://review.openstack.org/566723 | |
| 14:15:11 | artom | dansmith, ^^ | |
| 14:17:09 | dansmith | artom: you want me to comment about the mock then? | |
| 14:17:29 | artom | I want you to be happy :) | |
| 14:17:51 | artom | dansmith, in seriousness though, just... if I'm way off base, let me know so I can adjust my approach right away, instead of going down this rabbit hole | |