Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-29
08:20:59 brinzhang_ gibi: hi, https://review.opendev.org/#/c/723884/2/api-guide/source/accelerator-support.rst@69 what do you mean?
08:22:47 brinzhang_ gibi: Or other words, I seems not understand this sentence "I would make this cyborg specific in this doc.", and Line 76
08:51:41 brinzhang_ gibi: I think I get your comment, will be update
08:53:17 openstackgerrit Jiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD https://review.opendev.org/572805
09:08:13 openstackgerrit xuyuanhao proposed openstack/nova master: the vms can not be force deleted when vm_status is soft-delete and task-state=deleting https://review.opendev.org/724260
09:17:37 openstackgerrit Brin Zhang proposed openstack/nova master: Add nested resource providers limit for multi create https://review.opendev.org/723884
09:17:44 brinzhang_ gibi: updated, although the sentence is somewhat redundant, but it is easier to understand, please review again, thanks
09:33:16 stephenfin gibi, bauzas: Could I get you folks to take a look at these patches for me, one of which has been around for a loooong time https://review.opendev.org/#/c/706013/ https://review.opendev.org/#/c/530905/
10:15:28 openstackgerrit Brin Zhang proposed openstack/nova master: Add nested resource providers limit for multi create https://review.opendev.org/723884
10:16:19 openstackgerrit Stephen Finucane proposed openstack/nova master: Fix 500 error while passing 4-byte unicode data https://review.opendev.org/407514
10:59:36 openstackgerrit Merged openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772
10:59:43 openstackgerrit Merged openstack/nova master: Feature matrix: update AArch64 information https://review.opendev.org/715979
11:28:02 openstackgerrit jayaditya gupta proposed openstack/nova master: Support for --force flag for nova-manage placement heal_allocations command use this flag to forcefully call heal allocation for a specific instance https://review.opendev.org/715395
12:22:57 gibi stephenfin: did a review on both
12:23:05 stephenfin gibi++ ta
12:29:38 bauzas (from PTO morning)
12:36:05 gibi gmann: I left feedback on https://review.opendev.org/#/c/723645
13:09:50 openstackgerrit Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301
13:15:41 bauzas gibi: gmann: fwiw, we need to merge https://review.opendev.org/#/c/723645/ by a RC2 :(
13:16:04 bauzas I mean, by merging it back to Ussuri for a next RC
13:16:40 bauzas because if not, nova-status for Ussuri won't check it
13:17:46 gibi bauzas: yes, we have to do that
13:18:46 bauzas gibi: then I'm adding an ussuri-rc-potential tag to the bug
13:20:53 gibi bauzas: good point, thanks
13:24:02 gibi gmann: have you tried the new policy upgrade check in a devstack? I'm trying it but I see that the enforcer is None here https://review.opendev.org/#/c/723645/7/nova/cmd/status.py@378
13:29:27 gibi gmann: linked printouts in the review
13:31:14 sean-k-mooney gibi: printouts as in paper?
13:31:24 gibi sean-k-mooney: :)
13:31:34 gibi sean-k-mooney: printouts as the stuff my debugger printed
13:31:41 sean-k-mooney ah
13:32:24 sean-k-mooney i have actully done some spec reviews in paper form when i needed to compare and contrast 3 interrealted specs and ran out of monitor space on 3 monitors..
13:32:42 sean-k-mooney it works but its a pain and should be avoided unless you hate trees
13:34:03 bauzas gibi: I need to work on a next devstack :)
13:34:14 bauzas I have some hardware, I should try to use it
13:35:36 gibi sean-k-mooney: I did that last time when reading heavy telco specifications
13:42:23 artom belmoreira, hello again, have more time this morning (well, afternoon, for you)
13:42:52 belmoreira hi artom, tell me
13:43:53 artom belmoreira, so, stephenfin has proposed an online data migration here: https://review.opendev.org/#/c/537414/26/nova/objects/compute_node.py@533 to get rid of really old dict JSON blobs in instance_extra.numa_topology and replace them with ovo JSON blobs
13:45:28 artom The proposal is to use SQL string filtering, which is slow - so for deployments with a large number of instances (anything over 1000, based on what zzzeek was saying yesterday), this might be a painful migration to run
13:46:03 artom belmoreira, CERN's probably the largest operator (that we know of), so I was wondering if you could have any input on that
13:46:16 artom Like, maybe online data migrations taking forever isn't such a big deal?
13:46:27 belmoreira let me have a look
13:47:08 artom Thanks :)
13:52:08 belmoreira for what I see it should generate something like 'select * from compute_nodes where deleted = 0 and numa_topology like "%nova_object.name%";'
13:52:36 belmoreira being "compute_nodes" table, this will be done per cell
13:53:41 stephenfin belmoreira: pretty much
13:53:43 stephenfin though
13:53:43 dansmith belmoreira: he linked you to the wrong one.. we're doing compute nodes, but we're also doing all instances in the next file
13:54:14 artom belmoreira, oh, sorry, yeah, the compute nodes one isn't the big deal, there are rarely over a thousand of those
13:54:20 artom The instances are the open question
13:54:20 stephenfin 'select * from instance_extra where deleted = 0 and numa_topology is not null and numa_topology like "%nova_object.name%";'
13:54:36 belmoreira We have ~200 nodes per cell. If that is the correct query (I'm expecting that sql alchemy introduces much more stuff) executing it in a cell of ~200 is fast enough (39 ms)
13:55:29 artom stephenfin, did you find out about the filtering order? Like, if we put deleted=0 first, does it reduce the amount of instances the string filter has to work over?
13:55:51 stephenfin artom: the docs that I read said that the SQL engine would do that optimization itself
13:56:05 belmoreira stephenfin artom are we talking about instance_extra as well?
13:56:10 artom stephenfin, ok, that's reassuring
13:56:13 artom belmoreira, we are
13:56:18 stephenfin belmoreira: yes, both
13:57:33 belmoreira ok, I started with the compute_node.py :)
13:58:04 artom belmoreira, that was my bad, I linked the wrong thing
13:59:24 belmoreira I'm checking how much time takes the query that stephenfin mentioned
14:00:22 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-audit: Use cliff instead of homegrown argparse bleh https://review.opendev.org/724332
14:00:30 artom I have to say, this shows the power of operator involvement upstream
14:00:46 dansmith artom: man, I really wish this had just been done as migrate-on-load
14:01:07 artom Us devs get "access" to actual large deployment, and the operators get devs not being stupid ;)
14:01:53 artom dansmith, I guess stephenfin's response to that would be "but then we'll never know for sure that we can remove the compat routines"
14:02:18 stephenfin 'zactly
14:02:30 dansmith artom: I think we'd load this whenever we do update available resource, so it'd be migrated within one cycle
14:02:51 artom dansmith, for all instances?
14:02:54 dansmith and if not, we could have just added it to the preloads and forced it to happen
14:02:56 gmann bauzas: gibi oh, i checked locally but i might need to init the policy explicitly depends on when cmd are run. let me test those scenario also. I will do after my internal meeting.
14:03:00 dansmith artom: yes, parallelized by compute node
14:03:36 stephenfin dansmith: have we an in-tree example of this migrate-on-load pattern?
14:04:02 dansmith I can point you at one for sure, but let me see if we have any in current master
14:04:52 artom The fact that belmoreira appears to have RIP'ed running that query is worrisome ;)
14:06:23 sean-k-mooney stephenfin: one thing we could do if this was expensive is to not do it as a normal online data migration in the sense of a global change
14:06:37 dansmith stephenfin: https://github.com/openstack/nova/blob/58aaffade9f78c5fdc6b0d26ec26b70908b7a6aa/nova/objects/migration.py#L89-L89
14:06:41 sean-k-mooney but instead have a nova manage command that would allow you do it per host
14:06:52 dansmith stephenfin: we generate uuid for migration on load if not present
14:07:04 sean-k-mooney so you could slowly space our the migration host by host until you are done
14:07:36 dansmith sean-k-mooney: we already have batch limits in place, so that doesn't help
14:07:41 dansmith other than create more debt in the form of a new command
14:07:51 belmoreira I was running these in our DBs. Again this is per cell. Some have much more than 1000 instances but for those we don't have numa topology defined. The ones that have numa_topology defined only run batch processing so they are large instances. Meaning that we have x4 or x8 times the number of compute nodes.
14:08:09 belmoreira in both cases queries take few ms
14:08:21 sean-k-mooney well ya that basically what i was thinkign of just limiting the rows it touches but if we can already do that then sure
14:08:40 stephenfin belmoreira++ excellent, thanks for checking that up
14:08:53 artom belmoreira, yep, that's very helpful, thanks!
14:09:38 belmoreira artom stephenfin thanks for ping me on this
14:09:42 sean-k-mooney well there are two parts first the query to get all the instance that need to be migrated then we need to lock the db and update all recoreds
14:10:01 sean-k-mooney you can actully do that in one update query
14:10:18 sean-k-mooney in general bug not in this case
14:10:24 artom belmoreira, just to make sure I understand, you're saying that your numa_topology-having instances are only about 4 to 8 instances per compute host, right?
14:10:25 dansmith belmoreira: I'm not sure I got what you said.. you don't have a db with tons of instances with a numa topology to test this on right?
14:10:25 gibi gmann: ack, thanks
14:11:49 sean-k-mooney if i parsed it correctly the cells with numa instance only run large vms so they have fewer isntances then the other cells
14:12:01 belmoreira artom yes
14:12:01 dansmith right
14:12:46 artom belmoreira, aha, ok - well, it tells us that the sql engine (or sqlalchemy itself?) is smart and checks the more restrictive stuff first - iow having a numa_topology at all
14:13:03 artom But it doesn't tell us what happens if there are thousands of numa_topology-having instances... :(
14:13:06 sean-k-mooney to be fair i would suspect that is likly common of many deployments. e.g. most nuam instance will be large flavors and therefore there will be less of them then standard instances

Earlier   Later