| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-28 | |||
| 14:24:00 | dansmith | or at least, not in time | |
| 14:25:17 | gmann | dansmith: yeah that was my understanding but gibi opinion is to do that if we can. | |
| 14:25:28 | bnemec | I'm not a big fan of adding a feature to support an anti-pattern in deploying OpenStack. | |
| 14:26:03 | stephenfin | Could we just handwrite a policy.json file and include it in our sdist for this release? | |
| 14:26:17 | dansmith | gmann: the problem I see is that the oslo tool will not do that by default, but our default in nova is to need the deprecated rules, | |
| 14:26:25 | stephenfin | zigo could consume that instead of using the oslopolicy tool | |
| 14:26:29 | gmann | we have to carefully things of way for operator to move to new defaults was only overwrite the rule in policy file which is this case we taken as broken | |
| 14:26:34 | dansmith | so we're requiring a lot of people to know to disable one default and override another, to get a consistent set of defaults that will work | |
| 14:27:06 | artom | stephenfin, thanks :) | |
| 14:27:08 | gmann | exactly | |
| 14:27:16 | artom | dansmith, https://review.opendev.org/#/c/672595/73 pretty please? When your queue gets to it | |
| 14:27:35 | dansmith | stephenfin: I think that's more obscure for people that aren't already looking for a sample file, but maybe easier to get themselves out of the hole once they realize they've deployed and screwed themselves up | |
| 14:28:04 | dansmith | artom: ack | |
| 14:31:17 | gmann | gibi: I will update the patch for reno comments and will keep bug open. and we can discuss the best possible approach about policy file usage in cross project sessions in PTG what bnemec added in oslo etherpad and i linked in nova ptg ethrepad too. | |
| 14:32:10 | gmann | gibi: zigo that works for you ^^ ? | |
| 14:35:41 | dansmith | stephenfin: oh right, yeah I'm not going to approve those completely inconsistent style things in the middle of a file, but I'm sure someone else will | |
| 14:36:06 | stephenfin | huh? | |
| 14:37:03 | dansmith | the "black says this ugly style is cool, so I'll just break from the rest of nova conventions here for the new code I'm adding" thing in your tests | |
| 14:37:39 | stephenfin | um, those are entirely new tests? | |
| 14:37:56 | dansmith | in a file with a style, in a project with a style | |
| 14:38:29 | stephenfin | really? | |
| 14:38:47 | dansmith | you're not even consistent within those tests | |
| 14:39:10 | stephenfin | we're inconsistent all over the place | |
| 14:39:17 | stephenfin | you're not going to hammer me for style, surely? | |
| 14:43:26 | artom | stephenfin, dansmith, hey, I may be late to the party, but those .contains("nova_object.name") searches... | |
| 14:43:38 | artom | This kind of substring search is notoriously slow in SQL, no? | |
| 14:43:52 | artom | For the compute nodes, it's probably fine, since there aren't that many of them, but for instances... | |
| 14:43:58 | dansmith | artom: they're not indexed, so yes | |
| 14:45:00 | artom | So we're just accepting that? | |
| 14:45:46 | artom | I dunno if that's been talked about before, as I said, I'm late to the party | |
| 14:46:05 | stephenfin | I don't think we've a choice. There's no other heuristic we can use to differentiate the legacy entries from the non-legacy ones | |
| 14:46:17 | dansmith | I guess if we did it in python, despite being slower, it's load on the machine running the nova-manage, instead of the DB server itself, which is probably better | |
| 14:47:39 | dansmith | although maybe we'd have no filter at that point and basically return all the instances | |
| 14:47:43 | dansmith | so yeah I dunno | |
| 14:49:21 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Clarify the policy new defaults upgrade notes https://review.opendev.org/723645 | |
| 14:49:40 | gmann | dansmith: gibi zigo updated , please check - https://review.opendev.org/#/c/723645/ | |
| 14:50:30 | artom | dansmith, stephenfin, I guess there's no way to do "pagination" when selecting instance_extra? | |
| 14:50:48 | dansmith | artom: what would that help? | |
| 14:51:29 | artom | dansmith, select all instance_extra in steps of 100 or whatever, do the filtering in Python, find the instance UUIDs that way, then with those go back to the database? | |
| 14:52:58 | artom | Apparently you can do stuff with LIMIT: https://stackoverflow.com/questions/3799193/mysql-data-best-way-to-implement-paging | |
| 14:53:02 | dansmith | artom: but we'd still end up selecting every instance out of the database every time right? | |
| 14:53:11 | dansmith | every instance, in groups of 100 | |
| 14:53:26 | artom | Yeah, but at least it's spread out over time? | |
| 14:53:46 | artom | You're not filtering on all X thousands instances in a single query | |
| 14:53:55 | dansmith | artom: well, the migrations are supposed to be idempotent, so ideally you can run this over and over and have less and less impact | |
| 14:54:04 | gmann | melwitt: all setup for devstack and grenade now. we can approve this - https://review.opendev.org/#/c/704364/5 | |
| 14:54:08 | gmann | and then backport | |
| 14:54:26 | dansmith | artom: I think I'd rather the string filter in the query than just continually select every instance out of the database on every run, only to find that none are needed | |
| 14:54:28 | gmann | *done now | |
| 14:54:41 | dansmith | artom: I'd have to iterate every instance in the DB every time you asked me "are you done?" | |
| 14:55:06 | artom | dansmith, but isn't that what the string filter in the query is doing anyways, behind the scenes? | |
| 14:55:15 | artom | Looking at every instance in the database? | |
| 14:55:40 | dansmith | artom: no, it's looking at every value in a table | |
| 14:56:16 | dansmith | which is massively less expensive than selecting multiple tables, feeding those to python, constructing tons and tons of objects, for us to go through and do...that same string comparison | |
| 14:56:51 | artom | dansmith, right sorry - but what's what I was suggesting we do in python - select instance_uuid, numa_topology from instance_extra <in steps of 100>, fitler in python, use those instance_uuids for the actual update | |
| 14:57:17 | artom | But maybe ^^^ is impossible with our models/sqlalchemy? | |
| 14:57:31 | dansmith | you mean a direct query and not an ORM one? that's better, but you're still sending the results of all those queries across the network every time | |
| 14:57:52 | dansmith | no, we can do it without an ORM query, we do that in places | |
| 14:58:46 | artom | dansmith, yeah :/ | |
| 14:59:19 | artom | I guess we'd need metrics of how long the string filter would take, as a function of # of instances | |
| 14:59:47 | artom | To see if we're making a mountain our of a molehill | |
| 14:59:59 | dansmith | it'll definitely be faster for the db server to do it, the only benefit I can see is you offload the cpu processing to another machine, but you pay the penalty of increasing latency and definitely network traffic | |
| 15:00:34 | dansmith | besides, stephenfin will want to nuke this migration in a cycle or two anyway, so it won't be run forever | |
| 15:01:33 | gibi | away | |
| 15:02:29 | artom | dansmith, I suppose asking CERN to test this for us is not an option ;) | |
| 15:03:15 | dansmith | artom: sure you can ask them for their opinion on the pain | |
| 15:03:23 | dansmith | I dunno what the alternative is though | |
| 15:03:31 | artom | Yeah, valid point | |
| 15:03:44 | dansmith | especially in cern's case, the impact of sending their massive amount of data over the network is not likely better | |
| 15:03:52 | dansmith | and, they have upgrade windows where they can handle this kind of thing | |
| 15:04:10 | dansmith | artom: well, one alternative is to avoid doing this, which is valid | |
| 15:04:21 | dansmith | artom: this is cleanup that we don't need to do, other than to address stephenfin's OCD | |
| 15:04:36 | artom | I tend to like stephenfin OCD ;) | |
| 15:04:38 | dansmith | which while this seems like a good one to do, because it's data at rest, it's not *really* costing us much to maintain the compatibility | |
| 15:05:02 | artom | dansmith, I don't suppose doing it on the fly is an option? | |
| 15:05:12 | dansmith | artom: we're already doing it on the fly, that's my point | |
| 15:05:19 | artom | Like, every time we call those legacy dict methods, update the DB? | |
| 15:05:28 | artom | Seriously? | |
| 15:05:40 | artom | Yeeeaaahhh | |
| 15:05:49 | artom | I have to say, that sways me the other way around | |
| 15:05:50 | dansmith | we're doing it on read, and if we save it for some reason (like a migration) then we update it | |
| 15:06:07 | dansmith | we could do update-on-read too, which we've done in other places | |
| 15:06:23 | artom | Update on read would be nice | |
| 15:06:33 | dansmith | I'd be cool with that | |
| 15:06:52 | dansmith | despite this looking nicer, it really isn't *necessary* to update this.. these compat routines have been here for ages | |
| 15:07:11 | artom | I'd be less worried if the hit to the DB was less | |
| 15:07:28 | artom | If jaypipes was still around he could set us straight | |
| 15:07:39 | artom | Maybe I'm paranoid for nothing | |
| 15:07:50 | dansmith | yeah, I think it's a good insight.. I honestly hadn't really considered it, but I think you've got a good point | |
| 15:08:08 | dansmith | if it was something we really had to do, then I'd be for this approach, let the DB server do the hard bit, but... | |
| 15:08:33 | dansmith | the other thing about this, is for instances created since like 2018, this is just pain for no reason | |
| 15:08:41 | dansmith | or whenever we moved to the new format | |
| 15:08:54 | dansmith | this is really just looking for super ancient instances that have been nursed for a long time, | |
| 15:09:00 | dansmith | which definitely puts it into perspective for me | |
| 15:09:12 | stephenfin | 2014 | |
| 15:09:17 | dansmith | hah | |
| 15:12:27 | artom | stephenfin, your call I guess - looks like update on read might be a more acceptable solution, though with it we'd never be 100% sure we got all of those ancient instances | |
| 15:12:39 | stephenfin | And this isn't OCD. I've to maintain the hardware.py module and uglier corners of libvirt driver code, and I'm continuously trying to burn down tech debt inflicted upon me by others to make that job easier | |
| 15:13:39 | openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.opendev.org/722644 | |
| 15:14:12 | stephenfin | artom: personally, I was for deleting all the compat code to handle instances that hadn't been _touched_ since 2014, but dansmith said that wasn't good enough so I did this | |