| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 18:24:36 | dansmith | sean-k-mooney: dude, you need to cut yourself off :) | |
| 18:25:20 | dansmith | https://review.opendev.org/#/c/742961/ | |
| 18:27:43 | dansmith | I think the nova team needs to have the keys to sean-k-mooney's irc bouncer so we can turn it off when it's time for him to sleep | |
| 18:28:10 | sean-k-mooney | hehe i dont use one i just dont trun my laptop off :P | |
| 18:28:11 | dansmith | like giving car keys to the bartender | |
| 18:28:50 | melwitt | sean-k-mooney laptop and dev box permanently ON | |
| 18:29:20 | dansmith | sean-k-mooney: well, then an ssh account to your laptop I guess | |
| 18:29:49 | sean-k-mooney | melwitt: yes they more or less are. | |
| 18:30:00 | dansmith | melwitt: more like sean-k-mooney permanently ON | |
| 18:30:02 | melwitt | man, what was I doing earlier | |
| 18:30:10 | melwitt | true | |
| 18:30:20 | dansmith | laptop sleep timer be like "jesus when is he going to go to bed, I'm exhausted" | |
| 18:30:29 | melwitt | haha yeah | |
| 18:31:23 | artom | dansmith, I think we'll need remote access to his fuse box... | |
| 18:31:38 | artom | First we'll need to invent an SSHable fuse box... | |
| 18:31:44 | dansmith | artom: he's a property owner now, so we can't go to the landlord | |
| 18:32:27 | dansmith | artom: no need to cut the power just reset his luks key | |
| 18:33:04 | artom | But then they'd have a crappy web UI with 'password' hardcoded as the admin password | |
| 18:33:07 | artom | So maybe not | |
| 18:33:17 | melwitt | yeah, I would not be surprised | |
| 18:34:32 | sean-k-mooney | dansmith: oh your mean resting the luks key on my laptop would be a pain to fix | |
| 18:34:47 | dansmith | sean-k-mooney: no, we can reset it back to the one you know when you should be online | |
| 18:34:54 | sean-k-mooney | ah ok | |
| 18:35:00 | dansmith | hah | |
| 18:35:02 | sean-k-mooney | speaking of which o/ | |
| 18:35:07 | dansmith | good :) | |
| 18:35:27 | melwitt | wait, weren't you on pto today too? what the heck | |
| 18:35:56 | sean-k-mooney | yesterday | |
| 18:36:04 | sean-k-mooney | well untill today | |
| 18:36:08 | dansmith | maybe /melwitt/ needs the sleep | |
| 18:36:08 | sean-k-mooney | i was at a funeral | |
| 18:36:36 | sean-k-mooney | so ya back for a "shortish" day of not very stressful things | |
| 18:36:52 | sean-k-mooney | i planned to leave after teh bug call but got distraced | |
| 18:36:56 | sean-k-mooney | anyway food | |
| 18:37:00 | sean-k-mooney | o/ | |
| 18:37:04 | melwitt | well we swapped running the bug call today so I was like why is sean here | |
| 18:37:11 | melwitt | have a nice weekend o/ | |
| 18:37:26 | artom | melwitt, he was on PTO wednesday, so unable to send out the email | |
| 18:37:36 | artom | And usual email + run the call go hand in hand | |
| 18:37:43 | melwitt | no, I swapped with him and I was supposed to send the email | |
| 18:37:58 | melwitt | I just forgot to | |
| 18:37:58 | artom | Right, so swap means you get to run the call as well | |
| 18:38:06 | melwitt | I know, and I did | |
| 18:38:08 | artom | But... he's allowed to be there for the call | |
| 18:38:14 | melwitt | I know he's allowed to be there lol | |
| 18:38:24 | artom | WELL WHAT THE HELL ARE WE ARGUING ABOUT | |
| 18:38:27 | melwitt | I just thought if we swapped cause he was out, I was surprised when he was there | |
| 18:38:37 | melwitt | I DONT KNOW | |
| 18:38:41 | artom | WHY ARE WE YELLING | |
| 18:39:02 | melwitt | WE ARE HAVING TROUBLE CONTROLLING THE VOLUME OF OUR VOICE | |
| 18:39:06 | artom | OH RIGHT I STARTED IT IM SO SORRY | |
| 18:40:08 | melwitt | s/TROUBLE/DIFFICULTY/ | |
| 18:40:15 | artom | Umm, how about we do real work for a bit? Is there a way to see stats for a particular job? nova-ceph-multistore just failed twice on me | |
| 18:40:26 | dansmith | lol | |
| 18:40:26 | dansmith | dude | |
| 18:40:34 | artom | mriedem would have hacked up a logtash query in seconds | |
| 18:40:37 | dansmith | have you like paid attention to the last three hours in here at all? | |
| 18:40:38 | melwitt | omg | |
| 18:40:49 | dansmith | and also, logstash is still fubar I think | |
| 18:40:55 | melwitt | no u di'nt | |
| 18:41:21 | artom | dansmith, me? Pay attention? lol u cray cray | |
| 18:41:27 | dansmith | apparently ;) | |
| 19:22:53 | mriedem | i only do logdna queries these days now anyway | |
| 19:43:25 | melwitt | dansmith: I opened https://bugs.launchpad.net/nova/+bug/1888895 for the gate failure. going to mail the ML now | |
| 19:43:25 | openstack | Launchpad bug 1888895 in devstack-plugin-ceph "nova-ceph-multistore job fails often with 'No valid host was found. There are not enough hosts available.'" [Undecided,In progress] | |
| 19:43:44 | dansmith | cool | |
| 19:44:15 | melwitt | great, the WIP patch just failed | |
| 19:44:25 | melwitt | [errno 110] error connecting to the cluster wtf | |
| 19:46:27 | dansmith | maybe that config made it fail to start? | |
| 19:47:05 | dansmith | no ceph logs | |
| 19:47:54 | dansmith | hrm, don't even see it set the ini, | |
| 19:47:59 | dansmith | so maybe it didn't even get that far | |
| 19:50:44 | melwitt | yeah must be unless it's a fluke coincidence that ceph totally bombed this time | |
| 19:51:41 | dansmith | I think it must be a fluke bombing because it didn't run that line | |
| 19:53:01 | dansmith | ah here we go: 2020-07-24 18:32:36.045 | /opt/stack/devstack-plugin-ceph/devstack/lib/ceph: line 429: (24G: value too great for base (error token is "24G") | |
| 19:55:46 | melwitt | ah | |
| 19:55:59 | dansmith | hacky fix | |
| 20:05:20 | dansmith | melwitt: oops, should have put the bug on that, sorry | |
| 20:05:24 | dansmith | but it'll need cleanup | |
| 20:05:35 | melwitt | ah yeah | |
| 20:05:54 | melwitt | I was just thinking, I think all ceph jobs in openstack are broken over this, not just ours | |
| 20:06:50 | dansmith | potentially, but like I said, my job may be running slower or have different behaviors | |
| 20:07:26 | dansmith | actually | |
| 20:07:40 | melwitt | yeah. I was thinking of replying to mention other ceph jobs may be affected too. I see the older version 14.2.2 being pulled in openstack/tempest, for example | |
| 20:07:44 | dansmith | heh, I just saw a glance failure on the plain ceph job which is the same novalidhost | |
| 20:07:48 | dansmith | so I think yeah. | |
| 20:07:57 | dansmith | yeah | |
| 20:08:14 | dansmith | this is from one of my glance patches: https://e81e6b81331830d4903c-5acdef5dc10478cee5291df1596ec66a.ssl.cf1.rackcdn.com/742065/9/check/devstack-plugin-ceph-tempest-py3/292fd21/testr_results.html | |
| 20:09:17 | melwitt | ah yeah. and this is the tempest job failure I was looking at https://1ca2ee7583d21788b1d8-42b9b3ca9891e58d539431fcfb5b799d.ssl.cf2.rackcdn.com/742836/2/check/devstack-plugin-ceph-tempest-py3/547fab7/testr_results.html | |
| 20:09:27 | melwitt | (I picked a recent patch proposed to the tempest repo) | |
| 20:09:37 | dansmith | cool | |
| 20:09:56 | dansmith | that's awesome because of two things: | |
| 20:10:02 | dansmith | 1. I didn't break stuff | |
| 20:10:13 | dansmith | 2. I get to steal the credit from sean-k-mooney for fixing more people! | |
| 20:10:31 | melwitt | lol awww | |
| 20:10:45 | dansmith | dang, now I better give him honorable mention in the commit message ;P | |
| 20:11:08 | melwitt | hey, I found the default config value too | |
| 20:11:42 | melwitt | after that he blew past me finding how/where it was used | |
| 20:12:11 | dansmith | heh | |
| 20:12:51 | dansmith | he gave me a line to copy/paste, so... | |