| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-28 | |||
| 14:27:06 | artom | stephenfin, thanks :) | |
| 14:27:08 | gmann | exactly | |
| 14:27:16 | artom | dansmith, https://review.opendev.org/#/c/672595/73 pretty please? When your queue gets to it | |
| 14:27:35 | dansmith | stephenfin: I think that's more obscure for people that aren't already looking for a sample file, but maybe easier to get themselves out of the hole once they realize they've deployed and screwed themselves up | |
| 14:28:04 | dansmith | artom: ack | |
| 14:31:17 | gmann | gibi: I will update the patch for reno comments and will keep bug open. and we can discuss the best possible approach about policy file usage in cross project sessions in PTG what bnemec added in oslo etherpad and i linked in nova ptg ethrepad too. | |
| 14:32:10 | gmann | gibi: zigo that works for you ^^ ? | |
| 14:35:41 | dansmith | stephenfin: oh right, yeah I'm not going to approve those completely inconsistent style things in the middle of a file, but I'm sure someone else will | |
| 14:36:06 | stephenfin | huh? | |
| 14:37:03 | dansmith | the "black says this ugly style is cool, so I'll just break from the rest of nova conventions here for the new code I'm adding" thing in your tests | |
| 14:37:39 | stephenfin | um, those are entirely new tests? | |
| 14:37:56 | dansmith | in a file with a style, in a project with a style | |
| 14:38:29 | stephenfin | really? | |
| 14:38:47 | dansmith | you're not even consistent within those tests | |
| 14:39:10 | stephenfin | we're inconsistent all over the place | |
| 14:39:17 | stephenfin | you're not going to hammer me for style, surely? | |
| 14:43:26 | artom | stephenfin, dansmith, hey, I may be late to the party, but those .contains("nova_object.name") searches... | |
| 14:43:38 | artom | This kind of substring search is notoriously slow in SQL, no? | |
| 14:43:52 | artom | For the compute nodes, it's probably fine, since there aren't that many of them, but for instances... | |
| 14:43:58 | dansmith | artom: they're not indexed, so yes | |
| 14:45:00 | artom | So we're just accepting that? | |
| 14:45:46 | artom | I dunno if that's been talked about before, as I said, I'm late to the party | |
| 14:46:05 | stephenfin | I don't think we've a choice. There's no other heuristic we can use to differentiate the legacy entries from the non-legacy ones | |
| 14:46:17 | dansmith | I guess if we did it in python, despite being slower, it's load on the machine running the nova-manage, instead of the DB server itself, which is probably better | |
| 14:47:39 | dansmith | although maybe we'd have no filter at that point and basically return all the instances | |
| 14:47:43 | dansmith | so yeah I dunno | |
| 14:49:21 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Clarify the policy new defaults upgrade notes https://review.opendev.org/723645 | |
| 14:49:40 | gmann | dansmith: gibi zigo updated , please check - https://review.opendev.org/#/c/723645/ | |
| 14:50:30 | artom | dansmith, stephenfin, I guess there's no way to do "pagination" when selecting instance_extra? | |
| 14:50:48 | dansmith | artom: what would that help? | |
| 14:51:29 | artom | dansmith, select all instance_extra in steps of 100 or whatever, do the filtering in Python, find the instance UUIDs that way, then with those go back to the database? | |
| 14:52:58 | artom | Apparently you can do stuff with LIMIT: https://stackoverflow.com/questions/3799193/mysql-data-best-way-to-implement-paging | |
| 14:53:02 | dansmith | artom: but we'd still end up selecting every instance out of the database every time right? | |
| 14:53:11 | dansmith | every instance, in groups of 100 | |
| 14:53:26 | artom | Yeah, but at least it's spread out over time? | |
| 14:53:46 | artom | You're not filtering on all X thousands instances in a single query | |
| 14:53:55 | dansmith | artom: well, the migrations are supposed to be idempotent, so ideally you can run this over and over and have less and less impact | |
| 14:54:04 | gmann | melwitt: all setup for devstack and grenade now. we can approve this - https://review.opendev.org/#/c/704364/5 | |
| 14:54:08 | gmann | and then backport | |
| 14:54:26 | dansmith | artom: I think I'd rather the string filter in the query than just continually select every instance out of the database on every run, only to find that none are needed | |
| 14:54:28 | gmann | *done now | |
| 14:54:41 | dansmith | artom: I'd have to iterate every instance in the DB every time you asked me "are you done?" | |
| 14:55:06 | artom | dansmith, but isn't that what the string filter in the query is doing anyways, behind the scenes? | |
| 14:55:15 | artom | Looking at every instance in the database? | |
| 14:55:40 | dansmith | artom: no, it's looking at every value in a table | |
| 14:56:16 | dansmith | which is massively less expensive than selecting multiple tables, feeding those to python, constructing tons and tons of objects, for us to go through and do...that same string comparison | |
| 14:56:51 | artom | dansmith, right sorry - but what's what I was suggesting we do in python - select instance_uuid, numa_topology from instance_extra <in steps of 100>, fitler in python, use those instance_uuids for the actual update | |
| 14:57:17 | artom | But maybe ^^^ is impossible with our models/sqlalchemy? | |
| 14:57:31 | dansmith | you mean a direct query and not an ORM one? that's better, but you're still sending the results of all those queries across the network every time | |
| 14:57:52 | dansmith | no, we can do it without an ORM query, we do that in places | |
| 14:58:46 | artom | dansmith, yeah :/ | |
| 14:59:19 | artom | I guess we'd need metrics of how long the string filter would take, as a function of # of instances | |
| 14:59:47 | artom | To see if we're making a mountain our of a molehill | |
| 14:59:59 | dansmith | it'll definitely be faster for the db server to do it, the only benefit I can see is you offload the cpu processing to another machine, but you pay the penalty of increasing latency and definitely network traffic | |
| 15:00:34 | dansmith | besides, stephenfin will want to nuke this migration in a cycle or two anyway, so it won't be run forever | |
| 15:01:33 | gibi | away | |
| 15:02:29 | artom | dansmith, I suppose asking CERN to test this for us is not an option ;) | |
| 15:03:15 | dansmith | artom: sure you can ask them for their opinion on the pain | |
| 15:03:23 | dansmith | I dunno what the alternative is though | |
| 15:03:31 | artom | Yeah, valid point | |
| 15:03:44 | dansmith | especially in cern's case, the impact of sending their massive amount of data over the network is not likely better | |
| 15:03:52 | dansmith | and, they have upgrade windows where they can handle this kind of thing | |
| 15:04:10 | dansmith | artom: well, one alternative is to avoid doing this, which is valid | |
| 15:04:21 | dansmith | artom: this is cleanup that we don't need to do, other than to address stephenfin's OCD | |
| 15:04:36 | artom | I tend to like stephenfin OCD ;) | |
| 15:04:38 | dansmith | which while this seems like a good one to do, because it's data at rest, it's not *really* costing us much to maintain the compatibility | |
| 15:05:02 | artom | dansmith, I don't suppose doing it on the fly is an option? | |
| 15:05:12 | dansmith | artom: we're already doing it on the fly, that's my point | |
| 15:05:19 | artom | Like, every time we call those legacy dict methods, update the DB? | |
| 15:05:28 | artom | Seriously? | |
| 15:05:40 | artom | Yeeeaaahhh | |
| 15:05:49 | artom | I have to say, that sways me the other way around | |
| 15:05:50 | dansmith | we're doing it on read, and if we save it for some reason (like a migration) then we update it | |
| 15:06:07 | dansmith | we could do update-on-read too, which we've done in other places | |
| 15:06:23 | artom | Update on read would be nice | |
| 15:06:33 | dansmith | I'd be cool with that | |
| 15:06:52 | dansmith | despite this looking nicer, it really isn't *necessary* to update this.. these compat routines have been here for ages | |
| 15:07:11 | artom | I'd be less worried if the hit to the DB was less | |
| 15:07:28 | artom | If jaypipes was still around he could set us straight | |
| 15:07:39 | artom | Maybe I'm paranoid for nothing | |
| 15:07:50 | dansmith | yeah, I think it's a good insight.. I honestly hadn't really considered it, but I think you've got a good point | |
| 15:08:08 | dansmith | if it was something we really had to do, then I'd be for this approach, let the DB server do the hard bit, but... | |
| 15:08:33 | dansmith | the other thing about this, is for instances created since like 2018, this is just pain for no reason | |
| 15:08:41 | dansmith | or whenever we moved to the new format | |
| 15:08:54 | dansmith | this is really just looking for super ancient instances that have been nursed for a long time, | |
| 15:09:00 | dansmith | which definitely puts it into perspective for me | |
| 15:09:12 | stephenfin | 2014 | |
| 15:09:17 | dansmith | hah | |
| 15:12:27 | artom | stephenfin, your call I guess - looks like update on read might be a more acceptable solution, though with it we'd never be 100% sure we got all of those ancient instances | |
| 15:12:39 | stephenfin | And this isn't OCD. I've to maintain the hardware.py module and uglier corners of libvirt driver code, and I'm continuously trying to burn down tech debt inflicted upon me by others to make that job easier | |
| 15:13:39 | openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.opendev.org/722644 | |
| 15:14:12 | stephenfin | artom: personally, I was for deleting all the compat code to handle instances that hadn't been _touched_ since 2014, but dansmith said that wasn't good enough so I did this | |
| 15:14:23 | stephenfin | and have been working on it for two years | |
| 15:14:41 | artom | stephenfin, sorry dude, didn't mean to sabotage it :( | |
| 15:14:57 | artom | But it would have been dishonest to silence my concern | |
| 15:15:37 | artom | belmoreira, around? We're having a DB performance discussion, and figured CERN's scale might shed some light | |
| 15:17:21 | belmoreira | artom I just need to leave now. If you can I will be around tomorrow | |
| 15:17:47 | artom | belmoreira, ok, I'll ping you again earlier tomorrow morning (I'm on EDT) | |
| 15:20:27 | artom | I'm trying to read up on LIKE performance, which I *assume* is what sqlalchemy boils down .contains() to... | |
| 15:21:08 | dansmith | like is a little more expensive than a strstr() I think, although it might collapse to that if mysql notices there are no pattern characters | |