| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-25 | |||
| 20:26:42 | jroll | mriedem: I'm liking nova/tests/functional/compute/test_resource_tracker.py | |
| 20:26:43 | mriedem | and using a placement api client to assert things | |
| 20:26:49 | jroll | has a bunch of RT + ironic things already | |
| 20:27:04 | mriedem | this doesn't necessarily need to be an ironic virt driver functional test | |
| 20:27:19 | mriedem | just a fake driver (see nova/virt/fake.py) that returns a standard and custom trait | |
| 20:27:30 | jroll | true | |
| 20:27:38 | mriedem | test_resource_tracker.py is probably fine as a new test class | |
| 20:27:57 | mriedem | *with a new test class | |
| 20:28:03 | jroll | yeah, I'll poke around, thanks | |
| 20:43:29 | openstackgerrit | Claudiu Belu proposed openstack/nova master: tests: fixes mock autospec usage https://review.openstack.org/447505 | |
| 20:43:36 | mriedem | fyi, the cinder backup bug is still in the gate | |
| 20:43:39 | mriedem | there is another thing going on | |
| 20:43:52 | dansmith | still in the gate as in, unmerged? | |
| 20:43:52 | mriedem | swiftclient is logging within the thread | |
| 20:43:55 | mriedem | no, | |
| 20:43:59 | mriedem | another issue, same bug | |
| 20:44:06 | mriedem | http://logs.openstack.org/05/266605/33/check/neutron-tempest-dvr/8a5dd5f/logs/screen-c-bak.txt.gz#_Jan_25_19_26_01_896832 | |
| 20:44:16 | dansmith | ah | |
| 20:44:17 | mriedem | swiftclient is logging while their are in a thread and it switches | |
| 20:44:23 | mriedem | *they are | |
| 20:44:54 | efried | shitballs, does this mean stuff that's been sitting in the gate for 18h is now going to fail?? | |
| 20:44:57 | mriedem | seeing if i can patch cinder-backup to not do debug logging for swiftclient if using that backup driver | |
| 20:45:06 | mriedem | efried: it means it has a good chance of failing | |
| 20:45:07 | mriedem | :) | |
| 20:46:06 | efried | (That longish pause was me vocalizing my opinion of the situation) | |
| 20:46:24 | mriedem | i'm pushing a cinder patch | |
| 20:46:33 | efried | mriedem, thanks for tracking it down. | |
| 20:47:23 | efried | Sonofa... three of my patches were all green & blue, alllmost through the gate, and then stalled again. | |
| 20:49:35 | dansmith | heh | |
| 20:51:34 | efried | at least they didn't outright fail, and have to start 18+h cycle over again | |
| 20:52:22 | mriedem | efried: dansmith: https://review.openstack.org/538027 | |
| 20:53:49 | efried | mriedem What happens if swiftclient=SOMETHING_ELSE was already in the string? Does last win? | |
| 20:54:22 | mriedem | don't know | |
| 20:54:29 | mriedem | i know it's not in the default list of log levels in oslo.log | |
| 20:56:05 | efried | but if it's explicitly in whatever conf (on customer site), we don't want to override it. | |
| 20:59:52 | mriedem | that's why it's default_log_levels yeah? | |
| 21:00:30 | efried | Could be, could be. I don't know. I left comments, but +0 - smarter people can decide whether those things are really an issue. | |
| 21:01:40 | mriedem | look efried, do you want your 20 patches to merge or not | |
| 21:01:41 | mriedem | :) | |
| 21:02:11 | efried | I have NO IDEA what's actually best for my karma at this point. Walkin on eggshells, here. | |
| 21:03:48 | mriedem | yeah, i'm about to just call it a day | |
| 21:03:56 | mriedem | not really much that can be done about anything | |
| 21:04:36 | efried | No, you have to stick around to +2+W stuff so us peons can sit around rechecking them through the night. | |
| 21:04:57 | mriedem | i was rechecking things until about 11:30 last night | |
| 21:05:00 | mriedem | i've done my job | |
| 21:06:28 | efried | BTW, I was tooling around on stackalytics the other day - how did you manage to merge 18KLOC while pushing PTL paperwork 18 hours a day? | |
| 21:06:46 | efried | Impressive. Most impressive. | |
| 21:08:11 | Roamer` | hm, so here's a stupid question about stackalytics: if I put my name and e-mail address in default_data.json, will it pick up the patches that have already been merged, or does it just go forward? | |
| 21:09:46 | efried | Roamer` me no idea. Someone in -infra might know, but they're frying great big fish at the moment. | |
| 21:13:12 | Roamer` | efried, thanks, and yeah, I'm following the discussions there | |
| 21:14:18 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Reduce complexity of _from_db_object https://review.openstack.org/537728 | |
| 21:14:19 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408 | |
| 21:14:19 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_certs to instance_extra https://review.openstack.org/537897 | |
| 21:14:20 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 21:14:20 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 21:15:41 | openstackgerrit | Merged openstack/nova master: Update VMWare vSphere link address https://review.openstack.org/535244 | |
| 21:17:38 | mriedem | efried: don't know | |
| 21:17:50 | mriedem | amphetamines | |
| 21:18:06 | mriedem | likely api changes with lots of api samples | |
| 21:18:07 | efried | Tell ya, wait til you're my age. You'll never have the stamina. | |
| 21:18:20 | mriedem | that's why i take that thing that frank thomas is selling | |
| 21:18:32 | mriedem | nugenix | |
| 21:18:55 | mriedem | https://www.ispot.tv/ad/wZA8/nugenix-fan-featuring-frank-thomas | |
| 21:19:13 | mriedem | "stronger and leaner" | |
| 21:19:18 | efried | a powerful, man-boosting innovation | |
| 21:19:19 | mriedem | the guy has been like 300lb for 20 years | |
| 21:19:32 | mriedem | man boosting testafen | |
| 21:19:42 | mriedem | my passion levels will be through the roof! | |
| 21:27:01 | edleafe | It's cute watching you young 'uns talking about being old. | |
| 21:29:16 | dansmith | hush, dad's home! | |
| 21:29:34 | melwitt | hah :D | |
| 21:29:54 | mriedem | i'll grab the hearin' cone | |
| 21:30:25 | dansmith | that said, I'm quite sure edleafe remembers being 36 and feeling like life was over | |
| 21:33:07 | efried | Any time I think of 36... https://youtu.be/q6zhOdroJCE?t=1347 | |
| 21:33:49 | mriedem | umm, yikes | |
| 21:38:12 | melwitt | for the native thread logging related gate issues, will there always be a message somewhere in the logs that says "cannot switch to a different thread"? | |
| 21:38:26 | mriedem | yeah | |
| 21:39:35 | melwitt | okay. I was wondering if it could be related to the stable branch cells v1 job 100% fails. libvirt is getting "connection broke" as though something is holding up the tpool and never letting it continue | |
| 21:39:55 | melwitt | (I noticed all the libvirt calls go through eventlet.tpool) | |
| 21:40:12 | melwitt | but the "cannot switch to a different thread" isn't in the logs anywhere | |
| 21:41:11 | melwitt | I've been investigating it but haven't figured out anything yet | |
| 21:41:58 | dansmith | the libvirt driver uses native threads properly | |
| 21:42:08 | dansmith | so it should be fine unless something has regressed (or was never right) | |
| 21:42:40 | dansmith | the connection issue with libvirt almost always means libvirtd crashed, | |
| 21:42:51 | dansmith | which happens a lot more than it should | |
| 21:43:59 | melwitt | yeah. in this case it hasn't crashed. it's as if something held it up so long that the connection dropped from not getting keepalive, or something like that | |
| 21:44:19 | Roamer` | melwitt, what OS is this happening on? Could it be the Ubuntu py27/eventlet problem? http://lists.openstack.org/pipermail/openstack-dev/2018-January/126580.html | |
| 21:44:34 | melwitt | I was wondering if any other library also using eventlet could have made it so it never switches back to the libvirt driver | |
| 21:44:46 | dansmith | melwitt: why do you say it hasn't crashed? | |
| 21:45:25 | melwitt | dansmith: because from syslog and libvirtd I couldn't find any crashing. unless I totally don't know what I'm looking at, which is possible | |
| 21:45:45 | dansmith | melwitt: link? | |
| 21:45:46 | melwitt | Roamer`: it's ubuntu xenial. and that's what I was wondering, but I'm still grasping at straws at this point | |
| 21:46:34 | melwitt | dansmith: here's an example from recently http://logs.openstack.org/04/536904/3/check/legacy-tempest-dsvm-cells/13aacfc/logs/screen-n-cpu.txt.gz#_2018-01-25_19_06_13_271 | |
| 21:50:24 | dansmith | hmm, empty journal on that one? | |
| 21:50:25 | dansmith | that's weird | |
| 21:51:26 | melwitt | I feel like I've seen that before | |
| 21:52:56 | dansmith | yeah, libvirtd doesn't look like it was restarted across that connection failure boundary | |
| 21:52:58 | mriedem | interestingly, it's the first guest to be created it looks like http://logs.openstack.org/04/536904/3/check/legacy-tempest-dsvm-cells/13aacfc/logs/libvirt/qemu/instance-0000000a.txt.gz | |
| 21:53:17 | dansmith | this would be the first time I've ever seen that symptom not associated with a crash | |
| 21:53:34 | dansmith | so I still feel a little doubtful, but it's certainly not as glaring as normal | |