| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-31 | |||
| 14:44:01 | mdbooth | At some point you have to clean stuff up. That impacts backports. The alternative is never clean anything up. | |
| 14:44:25 | mriedem | mdbooth: it's not that black and white, | |
| 14:44:51 | mriedem | if you've got 3 refactors leading up to a bug fix that is going to be backported, try to minimize change to make the fix backportable, and stack the refactors on top | |
| 14:44:53 | dansmith | mriedem: remember, my house automatically shuts off all front-facing illumination on 10/31 | |
| 14:44:58 | dansmith | it remembered, I didn't | |
| 14:45:17 | mdbooth | dansmith: Nice :) | |
| 14:45:56 | mriedem | the cirros image we're using in the gate (0.4.0) is ~2 years old... | |
| 14:46:01 | mriedem | but there isn't anything newer | |
| 14:46:27 | efried | mriedem: here's that rbd thing http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22RuntimeError%3A%20rbd%20python%20libraries%20not%20found%5C%22 | |
| 14:46:35 | sean-k-mooney | mdbooth: halloween is an irish holiday | |
| 14:46:53 | sean-k-mooney | we exported it to the us and they may have exported it to the uk | |
| 14:46:54 | mdbooth | sean-k-mooney: What are you doing at work, then? | |
| 14:47:06 | efried | mriedem: am I reading that correctly, 80 hits in like 12 minutes? | |
| 14:47:20 | mdbooth | Ok, I may have claimed to have no beef with Ireland in the past, but now I'm mad. | |
| 14:47:20 | sean-k-mooney | we had the bankholiday on monday | |
| 14:47:43 | sean-k-mooney | trick or treating was not part of the irish holiday by the way | |
| 14:48:04 | sean-k-mooney | that was added in the us | |
| 14:48:19 | mdbooth | mriedem: Sure, but in the case of sean-k-mooney's patch the 'refactor' is just putting the generic stuff somewhere else. | |
| 14:48:24 | sean-k-mooney | it orgially was a pagan festival with games and music and stuff | |
| 14:49:01 | mdbooth | And if you invert the order, it's still going to hit the backportability of the next thing | |
| 14:49:14 | mdbooth | So if the cost is low, meh | |
| 14:49:14 | sean-k-mooney | mdbooth: mriedem i have no issue with wriing another patch to refactor my test on top of mdbooth refactor by the way | |
| 14:49:28 | mdbooth | sean-k-mooney: My thing is gone | |
| 14:49:39 | mdbooth | sean-k-mooney: I'm not using libvirt for that functional test any more | |
| 14:49:41 | sean-k-mooney | oh ok did not know that | |
| 14:49:56 | sean-k-mooney | well i dont need libvirt in this case either | |
| 14:50:07 | sean-k-mooney | its just the only numa test we currently have | |
| 14:50:38 | mdbooth | sean-k-mooney: I was just advocating that rather than essentially rewriting a new libvirt functional base class for 1 test, you could actually write one and then use it | |
| 14:50:54 | mdbooth | Do it in the same patch if required. | |
| 14:51:10 | mdbooth | mriedem: How strongly do you feel about the CastAsCall thing, btw? | |
| 14:51:30 | mdbooth | mriedem: That was a significant change when I rewrote my test. I see the benefit, though. | |
| 14:51:31 | sean-k-mooney | well i coudl write a generic numa functional base class and use that | |
| 14:51:49 | sean-k-mooney | and then later refactor the libvir one to inherit form it | |
| 14:52:31 | mdbooth | sean-k-mooney: If you went down the route of doing something like mriedem's new base class, but adding libvirtiness, you'd be losing CastAsCall. | |
| 14:52:55 | sean-k-mooney | we dont want the castAsCall stuff right? | |
| 14:52:56 | mdbooth | Which I agree with in principal, especially if we're writing a new base class, as it's quite different. | |
| 14:53:10 | mdbooth | sean-k-mooney: Right, but your base class here will be using it. | |
| 14:53:20 | mdbooth | So I'm wondering what the strength of feeling is there. | |
| 14:53:22 | sean-k-mooney | yes but may tests does not care | |
| 14:54:11 | mdbooth | sean-k-mooney: Ah, crap. I forgot to get coffee before your meeting ;) | |
| 14:54:29 | sean-k-mooney | i need the messaging to work so the service can comunicate but i dont depend on the castascall behavior | |
| 14:54:38 | sean-k-mooney | yep im about to do the same | |
| 14:55:33 | mriedem | efried: logstash isn't showing me hits for that | |
| 14:55:36 | mriedem | in 15 minutes anyway | |
| 14:55:56 | mriedem | efried: it's also only a couple of changes in the last 2 days | |
| 14:56:12 | mriedem | mdbooth: i don't know what sean-k-mooney patch you're referring to | |
| 14:56:52 | mriedem | mdbooth: i feel pretty strongly about not building new tests on anything that uses CastAsCall, | |
| 14:57:05 | mriedem | since it's bogus for testing realistic API behavior which is the point of functional tests | |
| 14:57:18 | mriedem | i'd like to nuke the castascall fixture unless we have a good reason to use it | |
| 14:58:10 | dansmith | mriedem: I agree it creates unrealistic tests, but don't we use it in a crapton of places currently? | |
| 14:58:19 | sean-k-mooney | mriedem: its the numa rebuild patch. which we discussed before as actully not beeing libvirt specific | |
| 14:58:29 | sean-k-mooney | mriedem: and it does not need castascall | |
| 14:58:34 | mriedem | dansmith: it's used in one of the parent classes for the API samples tests | |
| 14:58:51 | mriedem | dansmith: so lots of functional tests inherit it unknowingly | |
| 14:59:07 | dansmith | yeah | |
| 14:59:14 | sean-k-mooney | yep most proably would work fine without it | |
| 14:59:24 | sean-k-mooney | well provied the messageing worked | |
| 15:03:54 | dansmith | mriedem: so that mysql 16m packet patch of yours | |
| 15:04:22 | dansmith | oh, I see you saw the problem even on that eh? | |
| 15:05:17 | mriedem | not sure, i haven't dug into the grenade jobs on that one - unless i did and commented | |
| 15:05:22 | mriedem | i did see the aborted connections though | |
| 15:05:26 | mriedem | not the grenade failure though i don't think | |
| 15:05:47 | mriedem | but yeah the point is that is supposed to avoid the aborted connections | |
| 15:06:04 | dansmith | the mysql side of the error you mean | |
| 15:06:08 | mriedem | right | |
| 15:06:21 | dansmith | I was going to say, let's just merge that and see, | |
| 15:06:28 | dansmith | but maybe not worth it if you still saw those | |
| 15:06:35 | mriedem | right | |
| 15:06:48 | mriedem | kind of would like zzzeek to validate that kind of thing | |
| 15:07:04 | dansmith | those aborted connections definitely seem like they have to be a bad thing | |
| 15:07:08 | mriedem | the change on top that enables query logging - the gd mysql slow log file is so big you can't open it | |
| 15:08:00 | dansmith | mriedem: we don't capture etc/my.cnf it looks like so we can't validate that you set it properly right? | |
| 15:18:50 | mriedem | yeah that's a todo | |
| 15:19:10 | dansmith | mriedem: did you look for other evidence that it was set right? I dunno if it logs anything useful | |
| 15:20:04 | dansmith | looks like we only get error logs | |
| 15:21:04 | dansmith | I see the sudo log with the sed command at least | |
| 15:24:00 | dansmith | mriedem: ah, that message can come from a client just dropping the connection, which could maybe be happening due to connection pooling or something less problematic | |
| 15:24:05 | dansmith | the mysql-side error message I mean | |
| 15:24:39 | mriedem | ok | |
| 15:25:58 | mriedem | so this mostly hits in master patches, so grenade master == stable/train old and ussuri new, i'll check some stable/stein patches to see if their grenade logs have the same mysql aborted connection messages | |
| 15:26:17 | dansmith | ack | |
| 15:27:00 | mriedem | yeah this is a stein patch that merged on 10/26, seeing the same stuff in here https://zuul.opendev.org/t/openstack/build/7fd7bdb35ade439c81023b0fb0846ee8/log/logs/mysql/error.txt.gz | |
| 15:27:46 | mriedem | i wonder if there is some mysql diagnostics command we can run post-upgrade after calling verify_instance (which hits cell1) but before running smoke tests | |
| 15:28:44 | dansmith | well, I'm kinda thinking that this is just fallout from things we don't control, like the db connection pooling (or even api workers under uwsgi never getting scheduled) | |
| 15:28:51 | dansmith | unrelated to the scheduler thing | |
| 15:29:13 | dansmith | also, hmm | |
| 15:29:15 | mriedem | but something changed in train/ussuri obviously | |
| 15:29:33 | dansmith | mriedem: those errors say the user is "root".. aren't the services using actual users to connect to mysql? | |
| 15:31:43 | mriedem | connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_api?charset=utf8 | |
| 15:31:46 | mriedem | ... | |
| 15:31:47 | mriedem | i guess not | |
| 15:31:51 | mriedem | https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/688419/2/check/grenade-py3/8ec542b/logs/etc/nova/nova.conf.txt.gz | |
| 15:31:53 | dansmith | huh. | |
| 15:32:32 | mriedem | same in stein for a non-grenade job https://zuul.opendev.org/t/openstack/build/6cd13330e99346e4b688f35ec2a7ed36/log/controller/logs/etc/nova/nova_conf.txt.gz | |
| 15:34:01 | mriedem | yeah DATABASE_USER in devstack defaults to MYSQL_USER which defaults to root | |
| 15:35:51 | dansmith | okay always thought it was stack or nova or something | |
| 15:35:55 | mriedem | https://docs.openstack.org/nova/train/configuration/config.html#database.connection_recycle_time defaults to 1 hour, i wonder if we should drop that to like 10 minutes or something - not sure it would help anything | |
| 15:37:43 | dansmith | ah, like max_requests | |
| 15:37:53 | dansmith | the failure happens really early in the run though right? | |
| 15:38:06 | mriedem | post-upgrade? | |