Thanks for the input, Bill:

Since the data would probably require a couple of steps' worth of massaging

I was planning on a blanket APPEND FROM ... TYPE SDF then continuing from

there (I'm not sure, not having seen the whole project) unless I learn that

low-level file functions are significantly faster. Unless they want to

change from FPD, I don't get any .DBC capabilities, either, so a middle

step of massaging seems likely anyway.

Russ Scott

russsct@ix.netcom.com

====================================

At 10:17 PM 8/29/99 -0400, you wrote:

>Russ,

>

>This is a matter of great concern to me too, and I posted a question and got

>some interesting responses not too long ago on the subject. I have a monthly

>job that imports a few million records spread across almost a thousand

>tables (ZIP4 tables for the entire US). Because the data has to be massaged

>on the way into tables, I originally setup the import process to (low level

>I/O) read the ASCII fixed fields table, do the work and then APPEND BLANK to

>add a record to the database.

>

>I sped up the process a little by APPENDing blank records in 1,000 record

>"blocks" and then replacing the blank fields of each until the 1,000 empty

>records was filled and repeated the process again.

>

>Working this way, the import takes something like 2 days on a P133 machine.

>A faster machine probably would help, but relegating the job to a 'spare'

>machine that just chugs away seems to be the least bothersome. One thing I

>did do is design the process so multiple machines could participate in a

>time crunch, and that does work very well when needed.

>

>Suggestions I received for speeding up the process included using APPEND

>TYPE SDF, which would bring in each table en masse, but that would have to

>be followed with another step to do the massaging needed to prepare the

>tables for use, since UDFs can't be used on individual fields with APPEND

>TYPE SDF (but that would only partially address the data massaging

>requirements anyway). Another suggestion, by Anders, that seems to have a

>lot of merit, but I didn't try it yet, is to use APPEND TYPE SDF but with

>rules in the target DBC (has to be VFP and target has to be DBC, tho) ...

>I'm anxious to try this, but can't get to it yet.

>

>In general, processing each input record and then adding it to a table using

>APPEND BLANK is very slow compared to a bulk-add using APPEND TYPE SDF.

>

>As for FPD versus VFP in terms of speed, I know some people think that FPD

>if faster, but my experience so far with VFP shows that VFP (6, anyway, with

>SP3 if that means anything) it's actually faster than equivalent FPD apps. I

>attribute the difference to better memory management in VFP, but that's only

>my gut feeling (wish I had better measurement tools; I like the Profiler,

>but can't find a way to track I/O utilization). I have not yet tested the

>bulk-import under VFP, the tests I'm referring to are different, but they

>are long running processor and I/O intensive jobs.

>

>Bill

>

>

>-----Original Message-----

>From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

>Sent: Sunday, August 29, 1999 9:57 PM

>To: Multiple recipients of ProFox

>Subject: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

>processing

>

>

>

>Hey, all:

>

>I have a chance to investigate a data conversion job. This would involve

>importing from a flat file into FP tables, formatting/ massaging it, then

>outputting it to another set of flat files. The company uses FPD (not sure

>which version) instead of VFP because "FPD is faster". To give you an idea

>of the size of the jobs involved, they take from ~1 hour to multiple days.

>

>This causes me to ask some questions:

>

>Would FPD really be faster for data conversion if you compared VFP under

>Win9x/NT to FPD running on the same machine booted to DOS only? Does the

>version of DOS make a significant difference? Does anyone have any hard

>data, educated guesses or gut feelings about this?

>

>Are there considerations besides speed (reliablility for example) to take

>into consideration?

>

>On a related note, has anyone clocked APPEND FROM... and COPY TO... against

>low-level file commands?

>

>I don't have to talk to these people for a couple of days and was curious

>if anyone else had looked into these issues.

>

>Any ideas?

>

>

>Thanks:

>

>Russ Scott

>russsct@ix.netcom.com

>

>

>

>

>

>

>

>