Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-28
14:22:18 gmann which was the only way to move to new defaults and stop old token to pass via deprecated rule until we introduced new flag in oslo.policy 'oslo_policy.enforce_new_defaults'
14:22:21 stephenfin dansmith: Now that we've branched, could you take a look at https://review.opendev.org/#/c/537414/ and https://review.opendev.org/#/c/530905/ again?
14:23:12 gmann may be bnemec and stephenfin can input how we can accommodate both use case of 'oslopolicy-sample-generator' here.
14:23:39 gmann i mean adding deprecated based on request can be done but if it solve the zigo case.
14:23:43 dansmith stephenfin: ack
14:23:49 stephenfin dansmith: ta
14:23:57 dansmith gmann: I thought it was said that the generator wasn't going to get that new mode
14:24:00 dansmith or at least, not in time
14:25:17 gmann dansmith: yeah that was my understanding but gibi opinion is to do that if we can.
14:25:28 bnemec I'm not a big fan of adding a feature to support an anti-pattern in deploying OpenStack.
14:26:03 stephenfin Could we just handwrite a policy.json file and include it in our sdist for this release?
14:26:17 dansmith gmann: the problem I see is that the oslo tool will not do that by default, but our default in nova is to need the deprecated rules,
14:26:25 stephenfin zigo could consume that instead of using the oslopolicy tool
14:26:29 gmann we have to carefully things of way for operator to move to new defaults was only overwrite the rule in policy file which is this case we taken as broken
14:26:34 dansmith so we're requiring a lot of people to know to disable one default and override another, to get a consistent set of defaults that will work
14:27:06 artom stephenfin, thanks :)
14:27:08 gmann exactly
14:27:16 artom dansmith, https://review.opendev.org/#/c/672595/73 pretty please? When your queue gets to it
14:27:35 dansmith stephenfin: I think that's more obscure for people that aren't already looking for a sample file, but maybe easier to get themselves out of the hole once they realize they've deployed and screwed themselves up
14:28:04 dansmith artom: ack
14:31:17 gmann gibi: I will update the patch for reno comments and will keep bug open. and we can discuss the best possible approach about policy file usage in cross project sessions in PTG what bnemec added in oslo etherpad and i linked in nova ptg ethrepad too.
14:32:10 gmann gibi: zigo that works for you ^^ ?
14:35:41 dansmith stephenfin: oh right, yeah I'm not going to approve those completely inconsistent style things in the middle of a file, but I'm sure someone else will
14:36:06 stephenfin huh?
14:37:03 dansmith the "black says this ugly style is cool, so I'll just break from the rest of nova conventions here for the new code I'm adding" thing in your tests
14:37:39 stephenfin um, those are entirely new tests?
14:37:56 dansmith in a file with a style, in a project with a style
14:38:29 stephenfin really?
14:38:47 dansmith you're not even consistent within those tests
14:39:10 stephenfin we're inconsistent all over the place
14:39:17 stephenfin you're not going to hammer me for style, surely?
14:43:26 artom stephenfin, dansmith, hey, I may be late to the party, but those .contains("nova_object.name") searches...
14:43:38 artom This kind of substring search is notoriously slow in SQL, no?
14:43:52 artom For the compute nodes, it's probably fine, since there aren't that many of them, but for instances...
14:43:58 dansmith artom: they're not indexed, so yes
14:45:00 artom So we're just accepting that?
14:45:46 artom I dunno if that's been talked about before, as I said, I'm late to the party
14:46:05 stephenfin I don't think we've a choice. There's no other heuristic we can use to differentiate the legacy entries from the non-legacy ones
14:46:17 dansmith I guess if we did it in python, despite being slower, it's load on the machine running the nova-manage, instead of the DB server itself, which is probably better
14:47:39 dansmith although maybe we'd have no filter at that point and basically return all the instances
14:47:43 dansmith so yeah I dunno
14:49:21 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Clarify the policy new defaults upgrade notes https://review.opendev.org/723645
14:49:40 gmann dansmith: gibi zigo updated , please check - https://review.opendev.org/#/c/723645/
14:50:30 artom dansmith, stephenfin, I guess there's no way to do "pagination" when selecting instance_extra?
14:50:48 dansmith artom: what would that help?
14:51:29 artom dansmith, select all instance_extra in steps of 100 or whatever, do the filtering in Python, find the instance UUIDs that way, then with those go back to the database?
14:52:58 artom Apparently you can do stuff with LIMIT: https://stackoverflow.com/questions/3799193/mysql-data-best-way-to-implement-paging
14:53:02 dansmith artom: but we'd still end up selecting every instance out of the database every time right?
14:53:11 dansmith every instance, in groups of 100
14:53:26 artom Yeah, but at least it's spread out over time?
14:53:46 artom You're not filtering on all X thousands instances in a single query
14:53:55 dansmith artom: well, the migrations are supposed to be idempotent, so ideally you can run this over and over and have less and less impact
14:54:04 gmann melwitt: all setup for devstack and grenade now. we can approve this - https://review.opendev.org/#/c/704364/5
14:54:08 gmann and then backport
14:54:26 dansmith artom: I think I'd rather the string filter in the query than just continually select every instance out of the database on every run, only to find that none are needed
14:54:28 gmann *done now
14:54:41 dansmith artom: I'd have to iterate every instance in the DB every time you asked me "are you done?"
14:55:06 artom dansmith, but isn't that what the string filter in the query is doing anyways, behind the scenes?
14:55:15 artom Looking at every instance in the database?
14:55:40 dansmith artom: no, it's looking at every value in a table
14:56:16 dansmith which is massively less expensive than selecting multiple tables, feeding those to python, constructing tons and tons of objects, for us to go through and do...that same string comparison
14:56:51 artom dansmith, right sorry - but what's what I was suggesting we do in python - select instance_uuid, numa_topology from instance_extra <in steps of 100>, fitler in python, use those instance_uuids for the actual update
14:57:17 artom But maybe ^^^ is impossible with our models/sqlalchemy?
14:57:31 dansmith you mean a direct query and not an ORM one? that's better, but you're still sending the results of all those queries across the network every time
14:57:52 dansmith no, we can do it without an ORM query, we do that in places
14:58:46 artom dansmith, yeah :/
14:59:19 artom I guess we'd need metrics of how long the string filter would take, as a function of # of instances
14:59:47 artom To see if we're making a mountain our of a molehill
14:59:59 dansmith it'll definitely be faster for the db server to do it, the only benefit I can see is you offload the cpu processing to another machine, but you pay the penalty of increasing latency and definitely network traffic
15:00:34 dansmith besides, stephenfin will want to nuke this migration in a cycle or two anyway, so it won't be run forever
15:01:33 gibi away
15:02:29 artom dansmith, I suppose asking CERN to test this for us is not an option ;)
15:03:15 dansmith artom: sure you can ask them for their opinion on the pain
15:03:23 dansmith I dunno what the alternative is though
15:03:31 artom Yeah, valid point
15:03:44 dansmith especially in cern's case, the impact of sending their massive amount of data over the network is not likely better
15:03:52 dansmith and, they have upgrade windows where they can handle this kind of thing
15:04:10 dansmith artom: well, one alternative is to avoid doing this, which is valid
15:04:21 dansmith artom: this is cleanup that we don't need to do, other than to address stephenfin's OCD
15:04:36 artom I tend to like stephenfin OCD ;)
15:04:38 dansmith which while this seems like a good one to do, because it's data at rest, it's not *really* costing us much to maintain the compatibility
15:05:02 artom dansmith, I don't suppose doing it on the fly is an option?
15:05:12 dansmith artom: we're already doing it on the fly, that's my point
15:05:19 artom Like, every time we call those legacy dict methods, update the DB?
15:05:28 artom Seriously?
15:05:40 artom Yeeeaaahhh
15:05:49 artom I have to say, that sways me the other way around
15:05:50 dansmith we're doing it on read, and if we save it for some reason (like a migration) then we update it
15:06:07 dansmith we could do update-on-read too, which we've done in other places
15:06:23 artom Update on read would be nice
15:06:33 dansmith I'd be cool with that
15:06:52 dansmith despite this looking nicer, it really isn't *necessary* to update this.. these compat routines have been here for ages
15:07:11 artom I'd be less worried if the hit to the DB was less
15:07:28 artom If jaypipes was still around he could set us straight
15:07:39 artom Maybe I'm paranoid for nothing
15:07:50 dansmith yeah, I think it's a good insight.. I honestly hadn't really considered it, but I think you've got a good point
15:08:08 dansmith if it was something we really had to do, then I'd be for this approach, let the DB server do the hard bit, but...
15:08:33 dansmith the other thing about this, is for instances created since like 2018, this is just pain for no reason
15:08:41 dansmith or whenever we moved to the new format
15:08:54 dansmith this is really just looking for super ancient instances that have been nursed for a long time,

Earlier   Later