| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-31 | |||
| 14:29:42 | dansmith | oh jeez, today is halloween | |
| 14:29:48 | dansmith | totes forgot | |
| 14:43:18 | mdbooth | sean-k-mooney: My concern with the 'Don't clean anything up because we want to backport it' philosophy is that it means nothing ever gets cleaned up, which means we just end up accumulating duplicated crap. You could move the generic parts of NUMAServersTestNew to a reusable location and it would | |
| 14:43:18 | mdbooth | not impact the backportability of this patch at all. | |
| 14:43:55 | mriedem | dansmith: i don't want to alarm you, but you should probably get to your panic room in the basement soon | |
| 14:43:59 | mriedem | the kids start earlier and earlier | |
| 14:44:01 | mdbooth | At some point you have to clean stuff up. That impacts backports. The alternative is never clean anything up. | |
| 14:44:25 | mriedem | mdbooth: it's not that black and white, | |
| 14:44:51 | mriedem | if you've got 3 refactors leading up to a bug fix that is going to be backported, try to minimize change to make the fix backportable, and stack the refactors on top | |
| 14:44:53 | dansmith | mriedem: remember, my house automatically shuts off all front-facing illumination on 10/31 | |
| 14:44:58 | dansmith | it remembered, I didn't | |
| 14:45:17 | mdbooth | dansmith: Nice :) | |
| 14:45:56 | mriedem | the cirros image we're using in the gate (0.4.0) is ~2 years old... | |
| 14:46:01 | mriedem | but there isn't anything newer | |
| 14:46:27 | efried | mriedem: here's that rbd thing http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22RuntimeError%3A%20rbd%20python%20libraries%20not%20found%5C%22 | |
| 14:46:35 | sean-k-mooney | mdbooth: halloween is an irish holiday | |
| 14:46:53 | sean-k-mooney | we exported it to the us and they may have exported it to the uk | |
| 14:46:54 | mdbooth | sean-k-mooney: What are you doing at work, then? | |
| 14:47:06 | efried | mriedem: am I reading that correctly, 80 hits in like 12 minutes? | |
| 14:47:20 | mdbooth | Ok, I may have claimed to have no beef with Ireland in the past, but now I'm mad. | |
| 14:47:20 | sean-k-mooney | we had the bankholiday on monday | |
| 14:47:43 | sean-k-mooney | trick or treating was not part of the irish holiday by the way | |
| 14:48:04 | sean-k-mooney | that was added in the us | |
| 14:48:19 | mdbooth | mriedem: Sure, but in the case of sean-k-mooney's patch the 'refactor' is just putting the generic stuff somewhere else. | |
| 14:48:24 | sean-k-mooney | it orgially was a pagan festival with games and music and stuff | |
| 14:49:01 | mdbooth | And if you invert the order, it's still going to hit the backportability of the next thing | |
| 14:49:14 | mdbooth | So if the cost is low, meh | |
| 14:49:14 | sean-k-mooney | mdbooth: mriedem i have no issue with wriing another patch to refactor my test on top of mdbooth refactor by the way | |
| 14:49:28 | mdbooth | sean-k-mooney: My thing is gone | |
| 14:49:39 | mdbooth | sean-k-mooney: I'm not using libvirt for that functional test any more | |
| 14:49:41 | sean-k-mooney | oh ok did not know that | |
| 14:49:56 | sean-k-mooney | well i dont need libvirt in this case either | |
| 14:50:07 | sean-k-mooney | its just the only numa test we currently have | |
| 14:50:38 | mdbooth | sean-k-mooney: I was just advocating that rather than essentially rewriting a new libvirt functional base class for 1 test, you could actually write one and then use it | |
| 14:50:54 | mdbooth | Do it in the same patch if required. | |
| 14:51:10 | mdbooth | mriedem: How strongly do you feel about the CastAsCall thing, btw? | |
| 14:51:30 | mdbooth | mriedem: That was a significant change when I rewrote my test. I see the benefit, though. | |
| 14:51:31 | sean-k-mooney | well i coudl write a generic numa functional base class and use that | |
| 14:51:49 | sean-k-mooney | and then later refactor the libvir one to inherit form it | |
| 14:52:31 | mdbooth | sean-k-mooney: If you went down the route of doing something like mriedem's new base class, but adding libvirtiness, you'd be losing CastAsCall. | |
| 14:52:55 | sean-k-mooney | we dont want the castAsCall stuff right? | |
| 14:52:56 | mdbooth | Which I agree with in principal, especially if we're writing a new base class, as it's quite different. | |
| 14:53:10 | mdbooth | sean-k-mooney: Right, but your base class here will be using it. | |
| 14:53:20 | mdbooth | So I'm wondering what the strength of feeling is there. | |
| 14:53:22 | sean-k-mooney | yes but may tests does not care | |
| 14:54:11 | mdbooth | sean-k-mooney: Ah, crap. I forgot to get coffee before your meeting ;) | |
| 14:54:29 | sean-k-mooney | i need the messaging to work so the service can comunicate but i dont depend on the castascall behavior | |
| 14:54:38 | sean-k-mooney | yep im about to do the same | |
| 14:55:33 | mriedem | efried: logstash isn't showing me hits for that | |
| 14:55:36 | mriedem | in 15 minutes anyway | |
| 14:55:56 | mriedem | efried: it's also only a couple of changes in the last 2 days | |
| 14:56:12 | mriedem | mdbooth: i don't know what sean-k-mooney patch you're referring to | |
| 14:56:52 | mriedem | mdbooth: i feel pretty strongly about not building new tests on anything that uses CastAsCall, | |
| 14:57:05 | mriedem | since it's bogus for testing realistic API behavior which is the point of functional tests | |
| 14:57:18 | mriedem | i'd like to nuke the castascall fixture unless we have a good reason to use it | |
| 14:58:10 | dansmith | mriedem: I agree it creates unrealistic tests, but don't we use it in a crapton of places currently? | |
| 14:58:19 | sean-k-mooney | mriedem: its the numa rebuild patch. which we discussed before as actully not beeing libvirt specific | |
| 14:58:29 | sean-k-mooney | mriedem: and it does not need castascall | |
| 14:58:34 | mriedem | dansmith: it's used in one of the parent classes for the API samples tests | |
| 14:58:51 | mriedem | dansmith: so lots of functional tests inherit it unknowingly | |
| 14:59:07 | dansmith | yeah | |
| 14:59:14 | sean-k-mooney | yep most proably would work fine without it | |
| 14:59:24 | sean-k-mooney | well provied the messageing worked | |
| 15:03:54 | dansmith | mriedem: so that mysql 16m packet patch of yours | |
| 15:04:22 | dansmith | oh, I see you saw the problem even on that eh? | |
| 15:05:17 | mriedem | not sure, i haven't dug into the grenade jobs on that one - unless i did and commented | |
| 15:05:22 | mriedem | i did see the aborted connections though | |
| 15:05:26 | mriedem | not the grenade failure though i don't think | |
| 15:05:47 | mriedem | but yeah the point is that is supposed to avoid the aborted connections | |
| 15:06:04 | dansmith | the mysql side of the error you mean | |
| 15:06:08 | mriedem | right | |
| 15:06:21 | dansmith | I was going to say, let's just merge that and see, | |
| 15:06:28 | dansmith | but maybe not worth it if you still saw those | |
| 15:06:35 | mriedem | right | |
| 15:06:48 | mriedem | kind of would like zzzeek to validate that kind of thing | |
| 15:07:04 | dansmith | those aborted connections definitely seem like they have to be a bad thing | |
| 15:07:08 | mriedem | the change on top that enables query logging - the gd mysql slow log file is so big you can't open it | |
| 15:08:00 | dansmith | mriedem: we don't capture etc/my.cnf it looks like so we can't validate that you set it properly right? | |
| 15:18:50 | mriedem | yeah that's a todo | |
| 15:19:10 | dansmith | mriedem: did you look for other evidence that it was set right? I dunno if it logs anything useful | |
| 15:20:04 | dansmith | looks like we only get error logs | |
| 15:21:04 | dansmith | I see the sudo log with the sed command at least | |
| 15:24:00 | dansmith | mriedem: ah, that message can come from a client just dropping the connection, which could maybe be happening due to connection pooling or something less problematic | |
| 15:24:05 | dansmith | the mysql-side error message I mean | |
| 15:24:39 | mriedem | ok | |
| 15:25:58 | mriedem | so this mostly hits in master patches, so grenade master == stable/train old and ussuri new, i'll check some stable/stein patches to see if their grenade logs have the same mysql aborted connection messages | |
| 15:26:17 | dansmith | ack | |
| 15:27:00 | mriedem | yeah this is a stein patch that merged on 10/26, seeing the same stuff in here https://zuul.opendev.org/t/openstack/build/7fd7bdb35ade439c81023b0fb0846ee8/log/logs/mysql/error.txt.gz | |
| 15:27:46 | mriedem | i wonder if there is some mysql diagnostics command we can run post-upgrade after calling verify_instance (which hits cell1) but before running smoke tests | |
| 15:28:44 | dansmith | well, I'm kinda thinking that this is just fallout from things we don't control, like the db connection pooling (or even api workers under uwsgi never getting scheduled) | |
| 15:28:51 | dansmith | unrelated to the scheduler thing | |
| 15:29:13 | dansmith | also, hmm | |
| 15:29:15 | mriedem | but something changed in train/ussuri obviously | |
| 15:29:33 | dansmith | mriedem: those errors say the user is "root".. aren't the services using actual users to connect to mysql? | |
| 15:31:43 | mriedem | connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_api?charset=utf8 | |
| 15:31:46 | mriedem | ... | |
| 15:31:47 | mriedem | i guess not | |
| 15:31:51 | mriedem | https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/688419/2/check/grenade-py3/8ec542b/logs/etc/nova/nova.conf.txt.gz | |
| 15:31:53 | dansmith | huh. | |
| 15:32:32 | mriedem | same in stein for a non-grenade job https://zuul.opendev.org/t/openstack/build/6cd13330e99346e4b688f35ec2a7ed36/log/controller/logs/etc/nova/nova_conf.txt.gz | |