Russ,

This is a matter of great concern to me too, and I posted a question and got

some interesting responses not too long ago on the subject. I have a monthly

job that imports a few million records spread across almost a thousand

tables (ZIP4 tables for the entire US). Because the data has to be massaged

on the way into tables, I originally setup the import process to (low level

I/O) read the ASCII fixed fields table, do the work and then APPEND BLANK to

add a record to the database.

I sped up the process a little by APPENDing blank records in 1,000 record

"blocks" and then replacing the blank fields of each until the 1,000 empty

records was filled and repeated the process again.

Working this way, the import takes something like 2 days on a P133 machine.

A faster machine probably would help, but relegating the job to a 'spare'

machine that just chugs away seems to be the least bothersome. One thing I

did do is design the process so multiple machines could participate in a

time crunch, and that does work very well when needed.

Suggestions I received for speeding up the process included using APPEND

TYPE SDF, which would bring in each table en masse, but that would have to

be followed with another step to do the massaging needed to prepare the

tables for use, since UDFs can't be used on individual fields with APPEND

TYPE SDF (but that would only partially address the data massaging

requirements anyway). Another suggestion, by Anders, that seems to have a

lot of merit, but I didn't try it yet, is to use APPEND TYPE SDF but with

rules in the target DBC (has to be VFP and target has to be DBC, tho) ...

I'm anxious to try this, but can't get to it yet.

In general, processing each input record and then adding it to a table using

APPEND BLANK is very slow compared to a bulk-add using APPEND TYPE SDF.

As for FPD versus VFP in terms of speed, I know some people think that FPD

if faster, but my experience so far with VFP shows that VFP (6, anyway, with

SP3 if that means anything) it's actually faster than equivalent FPD apps. I

attribute the difference to better memory management in VFP, but that's only

my gut feeling (wish I had better measurement tools; I like the Profiler,

but can't find a way to track I/O utilization). I have not yet tested the

bulk-import under VFP, the tests I'm referring to are different, but they

are long running processor and I/O intensive jobs.

Bill

-----Original Message-----

From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

Sent: Sunday, August 29, 1999 9:57 PM

To: Multiple recipients of ProFox

Subject: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

processing

Hey, all:

I have a chance to investigate a data conversion job. This would involve

importing from a flat file into FP tables, formatting/ massaging it, then

outputting it to another set of flat files. The company uses FPD (not sure

which version) instead of VFP because "FPD is faster". To give you an idea

of the size of the jobs involved, they take from ~1 hour to multiple days.

This causes me to ask some questions:

Would FPD really be faster for data conversion if you compared VFP under

Win9x/NT to FPD running on the same machine booted to DOS only? Does the

version of DOS make a significant difference? Does anyone have any hard

data, educated guesses or gut feelings about this?

Are there considerations besides speed (reliablility for example) to take

into consideration?

On a related note, has anyone clocked APPEND FROM... and COPY TO... against

low-level file commands?

I don't have to talk to these people for a couple of days and was curious

if anyone else had looked into these issues.

Any ideas?

Thanks:

Russ Scott

russsct@ix.netcom.com