| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-29 | |||
| 13:20:53 | gibi | bauzas: good point, thanks | |
| 13:24:02 | gibi | gmann: have you tried the new policy upgrade check in a devstack? I'm trying it but I see that the enforcer is None here https://review.opendev.org/#/c/723645/7/nova/cmd/status.py@378 | |
| 13:29:27 | gibi | gmann: linked printouts in the review | |
| 13:31:14 | sean-k-mooney | gibi: printouts as in paper? | |
| 13:31:24 | gibi | sean-k-mooney: :) | |
| 13:31:34 | gibi | sean-k-mooney: printouts as the stuff my debugger printed | |
| 13:31:41 | sean-k-mooney | ah | |
| 13:32:24 | sean-k-mooney | i have actully done some spec reviews in paper form when i needed to compare and contrast 3 interrealted specs and ran out of monitor space on 3 monitors.. | |
| 13:32:42 | sean-k-mooney | it works but its a pain and should be avoided unless you hate trees | |
| 13:34:03 | bauzas | gibi: I need to work on a next devstack :) | |
| 13:34:14 | bauzas | I have some hardware, I should try to use it | |
| 13:35:36 | gibi | sean-k-mooney: I did that last time when reading heavy telco specifications | |
| 13:42:23 | artom | belmoreira, hello again, have more time this morning (well, afternoon, for you) | |
| 13:42:52 | belmoreira | hi artom, tell me | |
| 13:43:53 | artom | belmoreira, so, stephenfin has proposed an online data migration here: https://review.opendev.org/#/c/537414/26/nova/objects/compute_node.py@533 to get rid of really old dict JSON blobs in instance_extra.numa_topology and replace them with ovo JSON blobs | |
| 13:45:28 | artom | The proposal is to use SQL string filtering, which is slow - so for deployments with a large number of instances (anything over 1000, based on what zzzeek was saying yesterday), this might be a painful migration to run | |
| 13:46:03 | artom | belmoreira, CERN's probably the largest operator (that we know of), so I was wondering if you could have any input on that | |
| 13:46:16 | artom | Like, maybe online data migrations taking forever isn't such a big deal? | |
| 13:46:27 | belmoreira | let me have a look | |
| 13:47:08 | artom | Thanks :) | |
| 13:52:08 | belmoreira | for what I see it should generate something like 'select * from compute_nodes where deleted = 0 and numa_topology like "%nova_object.name%";' | |
| 13:52:36 | belmoreira | being "compute_nodes" table, this will be done per cell | |
| 13:53:41 | stephenfin | belmoreira: pretty much | |
| 13:53:43 | dansmith | belmoreira: he linked you to the wrong one.. we're doing compute nodes, but we're also doing all instances in the next file | |
| 13:53:43 | stephenfin | though | |
| 13:54:14 | artom | belmoreira, oh, sorry, yeah, the compute nodes one isn't the big deal, there are rarely over a thousand of those | |
| 13:54:20 | stephenfin | 'select * from instance_extra where deleted = 0 and numa_topology is not null and numa_topology like "%nova_object.name%";' | |
| 13:54:20 | artom | The instances are the open question | |
| 13:54:36 | belmoreira | We have ~200 nodes per cell. If that is the correct query (I'm expecting that sql alchemy introduces much more stuff) executing it in a cell of ~200 is fast enough (39 ms) | |
| 13:55:29 | artom | stephenfin, did you find out about the filtering order? Like, if we put deleted=0 first, does it reduce the amount of instances the string filter has to work over? | |
| 13:55:51 | stephenfin | artom: the docs that I read said that the SQL engine would do that optimization itself | |
| 13:56:05 | belmoreira | stephenfin artom are we talking about instance_extra as well? | |
| 13:56:10 | artom | stephenfin, ok, that's reassuring | |
| 13:56:13 | artom | belmoreira, we are | |
| 13:56:18 | stephenfin | belmoreira: yes, both | |
| 13:57:33 | belmoreira | ok, I started with the compute_node.py :) | |
| 13:58:04 | artom | belmoreira, that was my bad, I linked the wrong thing | |
| 13:59:24 | belmoreira | I'm checking how much time takes the query that stephenfin mentioned | |
| 14:00:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-audit: Use cliff instead of homegrown argparse bleh https://review.opendev.org/724332 | |
| 14:00:30 | artom | I have to say, this shows the power of operator involvement upstream | |
| 14:00:46 | dansmith | artom: man, I really wish this had just been done as migrate-on-load | |
| 14:01:07 | artom | Us devs get "access" to actual large deployment, and the operators get devs not being stupid ;) | |
| 14:01:53 | artom | dansmith, I guess stephenfin's response to that would be "but then we'll never know for sure that we can remove the compat routines" | |
| 14:02:18 | stephenfin | 'zactly | |
| 14:02:30 | dansmith | artom: I think we'd load this whenever we do update available resource, so it'd be migrated within one cycle | |
| 14:02:51 | artom | dansmith, for all instances? | |
| 14:02:54 | dansmith | and if not, we could have just added it to the preloads and forced it to happen | |
| 14:02:56 | gmann | bauzas: gibi oh, i checked locally but i might need to init the policy explicitly depends on when cmd are run. let me test those scenario also. I will do after my internal meeting. | |
| 14:03:00 | dansmith | artom: yes, parallelized by compute node | |
| 14:03:36 | stephenfin | dansmith: have we an in-tree example of this migrate-on-load pattern? | |
| 14:04:02 | dansmith | I can point you at one for sure, but let me see if we have any in current master | |
| 14:04:52 | artom | The fact that belmoreira appears to have RIP'ed running that query is worrisome ;) | |
| 14:06:23 | sean-k-mooney | stephenfin: one thing we could do if this was expensive is to not do it as a normal online data migration in the sense of a global change | |
| 14:06:37 | dansmith | stephenfin: https://github.com/openstack/nova/blob/58aaffade9f78c5fdc6b0d26ec26b70908b7a6aa/nova/objects/migration.py#L89-L89 | |
| 14:06:41 | sean-k-mooney | but instead have a nova manage command that would allow you do it per host | |
| 14:06:52 | dansmith | stephenfin: we generate uuid for migration on load if not present | |
| 14:07:04 | sean-k-mooney | so you could slowly space our the migration host by host until you are done | |
| 14:07:36 | dansmith | sean-k-mooney: we already have batch limits in place, so that doesn't help | |
| 14:07:41 | dansmith | other than create more debt in the form of a new command | |
| 14:07:51 | belmoreira | I was running these in our DBs. Again this is per cell. Some have much more than 1000 instances but for those we don't have numa topology defined. The ones that have numa_topology defined only run batch processing so they are large instances. Meaning that we have x4 or x8 times the number of compute nodes. | |
| 14:08:09 | belmoreira | in both cases queries take few ms | |
| 14:08:21 | sean-k-mooney | well ya that basically what i was thinkign of just limiting the rows it touches but if we can already do that then sure | |
| 14:08:40 | stephenfin | belmoreira++ excellent, thanks for checking that up | |
| 14:08:53 | artom | belmoreira, yep, that's very helpful, thanks! | |
| 14:09:38 | belmoreira | artom stephenfin thanks for ping me on this | |
| 14:09:42 | sean-k-mooney | well there are two parts first the query to get all the instance that need to be migrated then we need to lock the db and update all recoreds | |
| 14:10:01 | sean-k-mooney | you can actully do that in one update query | |
| 14:10:18 | sean-k-mooney | in general bug not in this case | |
| 14:10:24 | artom | belmoreira, just to make sure I understand, you're saying that your numa_topology-having instances are only about 4 to 8 instances per compute host, right? | |
| 14:10:25 | gibi | gmann: ack, thanks | |
| 14:10:25 | dansmith | belmoreira: I'm not sure I got what you said.. you don't have a db with tons of instances with a numa topology to test this on right? | |
| 14:11:49 | sean-k-mooney | if i parsed it correctly the cells with numa instance only run large vms so they have fewer isntances then the other cells | |
| 14:12:01 | dansmith | right | |
| 14:12:01 | belmoreira | artom yes | |
| 14:12:46 | artom | belmoreira, aha, ok - well, it tells us that the sql engine (or sqlalchemy itself?) is smart and checks the more restrictive stuff first - iow having a numa_topology at all | |
| 14:13:03 | artom | But it doesn't tell us what happens if there are thousands of numa_topology-having instances... :( | |
| 14:13:06 | sean-k-mooney | to be fair i would suspect that is likly common of many deployments. e.g. most nuam instance will be large flavors and therefore there will be less of them then standard instances | |
| 14:13:27 | artom | sean-k-mooney, that does seem very likely... | |
| 14:13:47 | dansmith | stephenfin: without the migrate on load, any idea what I can do to tickle those instances to rewrite them? | |
| 14:13:47 | belmoreira | dansmith you're right. I don't have a cell with a lot of instances. <2000 instances only | |
| 14:13:54 | dansmith | belmoreira: gotcha, thanks | |
| 14:14:12 | sean-k-mooney | belmoreira: could you do a count(*) on instance that have a numa topology so we can get a feel for the amount you have | |
| 14:14:32 | belmoreira | sean-k-mooney yes, give a sec | |
| 14:14:53 | sean-k-mooney | just wondering if its in the 100s or roughly what you were messuring | |
| 14:15:53 | sean-k-mooney | artom: i mean we can always fake it if we need too | |
| 14:16:08 | artom | sean-k-mooney, fake what? | |
| 14:16:24 | sean-k-mooney | a db with 1000s of numa instances | |
| 14:16:39 | artom | True | |
| 14:16:49 | belmoreira | our typical use case: | |
| 14:16:52 | belmoreira | select count(*) from instance_extra where deleted = 0; #711 | |
| 14:16:52 | belmoreira | select count(*) from compute_nodes nodes; #176 | |
| 14:16:53 | belmoreira | select * from instance_extra where deleted = 0 and numa_topology is not null and numa_topology like "%nova_object.name%"; #took 38.2 ms | |
| 14:17:10 | sean-k-mooney | ok so 700 ish is not bad | |
| 14:17:27 | sean-k-mooney | its not 2000 but its not 10 | |
| 14:18:07 | artom | So based on those number it might actually be OK to go ahead as is... | |
| 14:18:11 | sean-k-mooney | if we were to assume this was liniar then i think it would be accpetable | |
| 14:18:13 | artom | *numbers | |
| 14:19:09 | dansmith | so, if we just convert to migrate-on-load, we don't take any overhead for string searching at all, migrate only the instances that need it, and are assured that they have migrated all instances by the end of the cycle, right? | |
| 14:19:22 | dansmith | even with nova-manage, we can't make them run it without a blocker migration (which is also expensive) | |
| 14:19:48 | belmoreira | in this case nova_topology is not defined, but may help in the analysis: | |