Index
1999-08-17 22:09Steve Kramer : Getting up to speed? Learning curve?
1999-08-18 00:59Javed Aslam : Re: Getting up to speed? Learning curve?
1999-08-29 18:56Russ Scott, russsct@ix.netcom.com: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing
1999-08-29 19:17Bill Arnold, bill@wjarnold.com: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing
1999-08-29 22:19Russ Scott, russsct@ix.netcom.com: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing
1999-08-29 23:27Bill Arnold, bill@wjarnold.com: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing
1999-08-29 23:39Matthew S. Jarvis, mjarvis@peak.org: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing
1999-08-30 00:05Matthew S. Jarvis, mjarvis@peak.org: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing
1999-08-30 03:40Anders Altberg, anders.altberg@swipnet.se: Re: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing
1999-08-30 03:49Anders Altberg, anders.altberg@swipnet.se: Re: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing
Back to top
Getting up to speed? Learning curve?

Author: Steve Kramer

Posted: 1999-08-17 22:09:46   Link

Today is my first day working with Codebook after downloading all the files from the Flash web site. What's the quickest way to learn Codebook and what's the learning curve like on average? I am starting work on a new contract that appears to use the Codebook framework. The faster I can learn it and get up to speed using it, the better. I have been reading the Word docs for a few hours and studying the classes for a few hours today. I think my first task will be to add a program or form that reads an unformatted text file into a specific table format that is already part of a project that has codebook framework. Any thoughts, opinions and ideas would be greatly appreciated.

Steve Kramer, Kramer & Kramer Design.

VFP Programmer/Analyst and Cartoonist.

Home of "GO Cartoons". Over 250 and counting award-winning, original, off-the-wall cartoons. E-mail - gocartoons@aol.com

Web site - http://members.aol.com/gocartoons

©1999 Steve Kramer
Back to top
Re: Getting up to speed? Learning curve?

Author: Javed Aslam

Posted: 1999-08-18 00:59:29   Link

Hi Steve,

I have been using the CodeBook for a couple of years, and in my judgement, the best way to learn it is to simply dive in by creating a new project. Read about the QStart app in the documentation, and jump in. Make sure you install your VFP on the c: drive, and the CodeBook in c:VFPCodeBook. Create your own projects in the CodeBook directory where the QStart will put them in their own folders.

After creating your new project, creat a table with a few fields. Then subclass the iBizObjForm class to create a Form. Add the iControls library to the Form Controls tool bar to make it easy to place some data entry controls on the form. The class that requires the most intense study is cBizObj(in cBizness). Read everything in its CopyRight method. Once you understand how to subclass and use iBizObj, you are about half way up the learning curve. These are the classes, along with the cDataBehavior class (in cbhavior) that you will use for most of your data updates.

Allow yourself 6-12 months. I trust you have the CodeBook book. It is dated but still quite useful in learning the CodeBook "philosophy". This reading of textfile data into a table that you are talking about is hardly elementary in the context of CodeBook learning.

Hope this helps.

Regards

Javed

<I><FONT COLOR="#663300">Today is my first day working with Codebook after downloading all the files from the Flash web site. What's the quickest way to learn Codebook and what's the learning curve like on average? I am starting work on a new contract that appears to use the Codebook framework. The faster I can learn it and get up to speed using it, the better. I have been reading the Word docs for a few hours and studying the classes for a few hours today. I think my first task will be to add a program or form that reads an unformatted text file into a specific table format that is already part of a project that has codebook framework. Any thoughts, opinions and ideas would be greatly appreciated.</FONT></I>

©1999 Javed Aslam
Back to top
FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing

Author: Russ Scott, russsct@ix.netcom.com

Posted: 1999-08-29 18:56:36   Link

Hey, all:

I have a chance to investigate a data conversion job. This would involve

importing from a flat file into FP tables, formatting/ massaging it, then

outputting it to another set of flat files. The company uses FPD (not sure

which version) instead of VFP because "FPD is faster". To give you an idea

of the size of the jobs involved, they take from ~1 hour to multiple days.

This causes me to ask some questions:

Would FPD really be faster for data conversion if you compared VFP under

Win9x/NT to FPD running on the same machine booted to DOS only? Does the

version of DOS make a significant difference? Does anyone have any hard

data, educated guesses or gut feelings about this?

Are there considerations besides speed (reliablility for example) to take

into consideration?

On a related note, has anyone clocked APPEND FROM... and COPY TO... against

low-level file commands?

I don't have to talk to these people for a couple of days and was curious

if anyone else had looked into these issues.

Any ideas?

Thanks:

Russ Scott

russsct@ix.netcom.com

©1999 Russ Scott, russsct@ix.netcom.com
Back to top
RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing

Author: Bill Arnold, bill@wjarnold.com

Posted: 1999-08-29 19:17:07   Link

Russ,

This is a matter of great concern to me too, and I posted a question and got

some interesting responses not too long ago on the subject. I have a monthly

job that imports a few million records spread across almost a thousand

tables (ZIP4 tables for the entire US). Because the data has to be massaged

on the way into tables, I originally setup the import process to (low level

I/O) read the ASCII fixed fields table, do the work and then APPEND BLANK to

add a record to the database.

I sped up the process a little by APPENDing blank records in 1,000 record

"blocks" and then replacing the blank fields of each until the 1,000 empty

records was filled and repeated the process again.

Working this way, the import takes something like 2 days on a P133 machine.

A faster machine probably would help, but relegating the job to a 'spare'

machine that just chugs away seems to be the least bothersome. One thing I

did do is design the process so multiple machines could participate in a

time crunch, and that does work very well when needed.

Suggestions I received for speeding up the process included using APPEND

TYPE SDF, which would bring in each table en masse, but that would have to

be followed with another step to do the massaging needed to prepare the

tables for use, since UDFs can't be used on individual fields with APPEND

TYPE SDF (but that would only partially address the data massaging

requirements anyway). Another suggestion, by Anders, that seems to have a

lot of merit, but I didn't try it yet, is to use APPEND TYPE SDF but with

rules in the target DBC (has to be VFP and target has to be DBC, tho) ...

I'm anxious to try this, but can't get to it yet.

In general, processing each input record and then adding it to a table using

APPEND BLANK is very slow compared to a bulk-add using APPEND TYPE SDF.

As for FPD versus VFP in terms of speed, I know some people think that FPD

if faster, but my experience so far with VFP shows that VFP (6, anyway, with

SP3 if that means anything) it's actually faster than equivalent FPD apps. I

attribute the difference to better memory management in VFP, but that's only

my gut feeling (wish I had better measurement tools; I like the Profiler,

but can't find a way to track I/O utilization). I have not yet tested the

bulk-import under VFP, the tests I'm referring to are different, but they

are long running processor and I/O intensive jobs.

Bill

-----Original Message-----

From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

Sent: Sunday, August 29, 1999 9:57 PM

To: Multiple recipients of ProFox

Subject: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

processing

Hey, all:

I have a chance to investigate a data conversion job. This would involve

importing from a flat file into FP tables, formatting/ massaging it, then

outputting it to another set of flat files. The company uses FPD (not sure

which version) instead of VFP because "FPD is faster". To give you an idea

of the size of the jobs involved, they take from ~1 hour to multiple days.

This causes me to ask some questions:

Would FPD really be faster for data conversion if you compared VFP under

Win9x/NT to FPD running on the same machine booted to DOS only? Does the

version of DOS make a significant difference? Does anyone have any hard

data, educated guesses or gut feelings about this?

Are there considerations besides speed (reliablility for example) to take

into consideration?

On a related note, has anyone clocked APPEND FROM... and COPY TO... against

low-level file commands?

I don't have to talk to these people for a couple of days and was curious

if anyone else had looked into these issues.

Any ideas?

Thanks:

Russ Scott

russsct@ix.netcom.com

©1999 Bill Arnold, bill@wjarnold.com
Back to top
RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing

Author: Russ Scott, russsct@ix.netcom.com

Posted: 1999-08-29 22:19:53   Link

Thanks for the input, Bill:

Since the data would probably require a couple of steps' worth of massaging

I was planning on a blanket APPEND FROM ... TYPE SDF then continuing from

there (I'm not sure, not having seen the whole project) unless I learn that

low-level file functions are significantly faster. Unless they want to

change from FPD, I don't get any .DBC capabilities, either, so a middle

step of massaging seems likely anyway.

Russ Scott

russsct@ix.netcom.com

====================================

At 10:17 PM 8/29/99 -0400, you wrote:

>Russ,

>

>This is a matter of great concern to me too, and I posted a question and got

>some interesting responses not too long ago on the subject. I have a monthly

>job that imports a few million records spread across almost a thousand

>tables (ZIP4 tables for the entire US). Because the data has to be massaged

>on the way into tables, I originally setup the import process to (low level

>I/O) read the ASCII fixed fields table, do the work and then APPEND BLANK to

>add a record to the database.

>

>I sped up the process a little by APPENDing blank records in 1,000 record

>"blocks" and then replacing the blank fields of each until the 1,000 empty

>records was filled and repeated the process again.

>

>Working this way, the import takes something like 2 days on a P133 machine.

>A faster machine probably would help, but relegating the job to a 'spare'

>machine that just chugs away seems to be the least bothersome. One thing I

>did do is design the process so multiple machines could participate in a

>time crunch, and that does work very well when needed.

>

>Suggestions I received for speeding up the process included using APPEND

>TYPE SDF, which would bring in each table en masse, but that would have to

>be followed with another step to do the massaging needed to prepare the

>tables for use, since UDFs can't be used on individual fields with APPEND

>TYPE SDF (but that would only partially address the data massaging

>requirements anyway). Another suggestion, by Anders, that seems to have a

>lot of merit, but I didn't try it yet, is to use APPEND TYPE SDF but with

>rules in the target DBC (has to be VFP and target has to be DBC, tho) ...

>I'm anxious to try this, but can't get to it yet.

>

>In general, processing each input record and then adding it to a table using

>APPEND BLANK is very slow compared to a bulk-add using APPEND TYPE SDF.

>

>As for FPD versus VFP in terms of speed, I know some people think that FPD

>if faster, but my experience so far with VFP shows that VFP (6, anyway, with

>SP3 if that means anything) it's actually faster than equivalent FPD apps. I

>attribute the difference to better memory management in VFP, but that's only

>my gut feeling (wish I had better measurement tools; I like the Profiler,

>but can't find a way to track I/O utilization). I have not yet tested the

>bulk-import under VFP, the tests I'm referring to are different, but they

>are long running processor and I/O intensive jobs.

>

>Bill

>

>

>-----Original Message-----

>From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

>Sent: Sunday, August 29, 1999 9:57 PM

>To: Multiple recipients of ProFox

>Subject: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

>processing

>

>

>

>Hey, all:

>

>I have a chance to investigate a data conversion job. This would involve

>importing from a flat file into FP tables, formatting/ massaging it, then

>outputting it to another set of flat files. The company uses FPD (not sure

>which version) instead of VFP because "FPD is faster". To give you an idea

>of the size of the jobs involved, they take from ~1 hour to multiple days.

>

>This causes me to ask some questions:

>

>Would FPD really be faster for data conversion if you compared VFP under

>Win9x/NT to FPD running on the same machine booted to DOS only? Does the

>version of DOS make a significant difference? Does anyone have any hard

>data, educated guesses or gut feelings about this?

>

>Are there considerations besides speed (reliablility for example) to take

>into consideration?

>

>On a related note, has anyone clocked APPEND FROM... and COPY TO... against

>low-level file commands?

>

>I don't have to talk to these people for a couple of days and was curious

>if anyone else had looked into these issues.

>

>Any ideas?

>

>

>Thanks:

>

>Russ Scott

>russsct@ix.netcom.com

>

>

>

>

>

>

>

>

©1999 Russ Scott, russsct@ix.netcom.com
Back to top
RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing

Author: Bill Arnold, bill@wjarnold.com

Posted: 1999-08-29 23:27:13   Link

Russ, the low level file functions would speed things up if they could

target a table, but having to do APPEND BLANKs kills it, for some reason

that I don't understand. Even without indexes, the operation crawls. I don't

remember where I saw it, or perhaps I'm just making it up, but there

shouldn't be any reason why a 'low level' routine (maybe this would be a

good use for C) can't load a .dbf table (mimicking header and stuff).

Imagine loading tables with the speed of, say, PKZIP?

Bill

-----Original Message-----

From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

Sent: Monday, August 30, 1999 1:20 AM

To: Multiple recipients of ProFox

Subject: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

processing

Thanks for the input, Bill:

Since the data would probably require a couple of steps' worth of massaging

I was planning on a blanket APPEND FROM ... TYPE SDF then continuing from

there (I'm not sure, not having seen the whole project) unless I learn that

low-level file functions are significantly faster. Unless they want to

change from FPD, I don't get any .DBC capabilities, either, so a middle

step of massaging seems likely anyway.

Russ Scott

russsct@ix.netcom.com

©1999 Bill Arnold, bill@wjarnold.com
Back to top
RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing

Author: Matthew S. Jarvis, mjarvis@peak.org

Posted: 1999-08-29 23:39:28   Link

VFP (5) exports to FPW, but will VFP export to FPD? If so, maybe the VFP6

with DBC rules might be an interesting combined solution.

mjar

At 10:19 PM 8/29/99 -0700, you wrote:

>

>Thanks for the input, Bill:

>

>Since the data would probably require a couple of steps' worth of massaging

>I was planning on a blanket APPEND FROM ... TYPE SDF then continuing from

>there (I'm not sure, not having seen the whole project) unless I learn that

>low-level file functions are significantly faster. Unless they want to

>change from FPD, I don't get any .DBC capabilities, either, so a middle

>step of massaging seems likely anyway.

>

>Russ Scott

>russsct@ix.netcom.com

>

>====================================

>At 10:17 PM 8/29/99 -0400, you wrote:

>>Russ,

>>

>>This is a matter of great concern to me too, and I posted a question and got

>>some interesting responses not too long ago on the subject. I have a monthly

>>job that imports a few million records spread across almost a thousand

>>tables (ZIP4 tables for the entire US). Because the data has to be massaged

>>on the way into tables, I originally setup the import process to (low level

>>I/O) read the ASCII fixed fields table, do the work and then APPEND BLANK to

>>add a record to the database.

>>

>>I sped up the process a little by APPENDing blank records in 1,000 record

>>"blocks" and then replacing the blank fields of each until the 1,000 empty

>>records was filled and repeated the process again.

>>

>>Working this way, the import takes something like 2 days on a P133 machine.

>>A faster machine probably would help, but relegating the job to a 'spare'

>>machine that just chugs away seems to be the least bothersome. One thing I

>>did do is design the process so multiple machines could participate in a

>>time crunch, and that does work very well when needed.

>>

>>Suggestions I received for speeding up the process included using APPEND

>>TYPE SDF, which would bring in each table en masse, but that would have to

>>be followed with another step to do the massaging needed to prepare the

>>tables for use, since UDFs can't be used on individual fields with APPEND

>>TYPE SDF (but that would only partially address the data massaging

>>requirements anyway). Another suggestion, by Anders, that seems to have a

>>lot of merit, but I didn't try it yet, is to use APPEND TYPE SDF but with

>>rules in the target DBC (has to be VFP and target has to be DBC, tho) ...

>>I'm anxious to try this, but can't get to it yet.

>>

>>In general, processing each input record and then adding it to a table using

>>APPEND BLANK is very slow compared to a bulk-add using APPEND TYPE SDF.

>>

>>As for FPD versus VFP in terms of speed, I know some people think that FPD

>>if faster, but my experience so far with VFP shows that VFP (6, anyway, with

>>SP3 if that means anything) it's actually faster than equivalent FPD apps. I

>>attribute the difference to better memory management in VFP, but that's only

>>my gut feeling (wish I had better measurement tools; I like the Profiler,

>>but can't find a way to track I/O utilization). I have not yet tested the

>>bulk-import under VFP, the tests I'm referring to are different, but they

>>are long running processor and I/O intensive jobs.

>>

>>Bill

>>

>>

>>-----Original Message-----

>>From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

>>Sent: Sunday, August 29, 1999 9:57 PM

>>To: Multiple recipients of ProFox

>>Subject: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

>>processing

>>

>>

>>

>>Hey, all:

>>

>>I have a chance to investigate a data conversion job. This would involve

>>importing from a flat file into FP tables, formatting/ massaging it, then

>>outputting it to another set of flat files. The company uses FPD (not sure

>>which version) instead of VFP because "FPD is faster". To give you an idea

>>of the size of the jobs involved, they take from ~1 hour to multiple days.

>>

>>This causes me to ask some questions:

>>

>>Would FPD really be faster for data conversion if you compared VFP under

>>Win9x/NT to FPD running on the same machine booted to DOS only? Does the

>>version of DOS make a significant difference? Does anyone have any hard

>>data, educated guesses or gut feelings about this?

>>

>>Are there considerations besides speed (reliablility for example) to take

>>into consideration?

>>

>>On a related note, has anyone clocked APPEND FROM... and COPY TO... against

>>low-level file commands?

>>

>>I don't have to talk to these people for a couple of days and was curious

>>if anyone else had looked into these issues.

>>

>>Any ideas?

>>

>>

>>Thanks:

>>

>>Russ Scott

>>russsct@ix.netcom.com

>>

>>

>>

>>

>>

>>

>>

>>

>

>

>

>

©1999 Matthew S. Jarvis, mjarvis@peak.org
Back to top
RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing

Author: Matthew S. Jarvis, mjarvis@peak.org

Posted: 1999-08-30 00:05:39   Link

Also, I would have thought that the later versions of FP would have

enhanced Rushmore performance - or would that not apply because you are

most likely going without indexes and therefor non-optimizable? Also, maybe

the enhanced performance is eaten away by the Windows overhead?

Just some thoughts....

mjar

At 11:39 PM 8/29/99 -0700, you wrote:

>VFP (5) exports to FPW, but will VFP export to FPD? If so, maybe the VFP6

>with DBC rules might be an interesting combined solution.

>

>mjar

>

>

>At 10:19 PM 8/29/99 -0700, you wrote:

>>

>>Thanks for the input, Bill:

>>

>>Since the data would probably require a couple of steps' worth of massaging

>>I was planning on a blanket APPEND FROM ... TYPE SDF then continuing from

>>there (I'm not sure, not having seen the whole project) unless I learn that

>>low-level file functions are significantly faster. Unless they want to

>>change from FPD, I don't get any .DBC capabilities, either, so a middle

>>step of massaging seems likely anyway.

>>

>>Russ Scott

>>russsct@ix.netcom.com

>>

>>====================================

>>At 10:17 PM 8/29/99 -0400, you wrote:

>>>Russ,

>>>

>>>This is a matter of great concern to me too, and I posted a question and

got

>>>some interesting responses not too long ago on the subject. I have a

monthly

>>>job that imports a few million records spread across almost a thousand

>>>tables (ZIP4 tables for the entire US). Because the data has to be massaged

>>>on the way into tables, I originally setup the import process to (low level

>>>I/O) read the ASCII fixed fields table, do the work and then APPEND

BLANK to

>>>add a record to the database.

>>>

>>>I sped up the process a little by APPENDing blank records in 1,000 record

>>>"blocks" and then replacing the blank fields of each until the 1,000 empty

>>>records was filled and repeated the process again.

>>>

>>>Working this way, the import takes something like 2 days on a P133 machine.

>>>A faster machine probably would help, but relegating the job to a 'spare'

>>>machine that just chugs away seems to be the least bothersome. One thing I

>>>did do is design the process so multiple machines could participate in a

>>>time crunch, and that does work very well when needed.

>>>

>>>Suggestions I received for speeding up the process included using APPEND

>>>TYPE SDF, which would bring in each table en masse, but that would have to

>>>be followed with another step to do the massaging needed to prepare the

>>>tables for use, since UDFs can't be used on individual fields with APPEND

>>>TYPE SDF (but that would only partially address the data massaging

>>>requirements anyway). Another suggestion, by Anders, that seems to have a

>>>lot of merit, but I didn't try it yet, is to use APPEND TYPE SDF but with

>>>rules in the target DBC (has to be VFP and target has to be DBC, tho) ...

>>>I'm anxious to try this, but can't get to it yet.

>>>

>>>In general, processing each input record and then adding it to a table

using

>>>APPEND BLANK is very slow compared to a bulk-add using APPEND TYPE SDF.

>>>

>>>As for FPD versus VFP in terms of speed, I know some people think that FPD

>>>if faster, but my experience so far with VFP shows that VFP (6, anyway,

with

>>>SP3 if that means anything) it's actually faster than equivalent FPD

apps. I

>>>attribute the difference to better memory management in VFP, but that's

only

>>>my gut feeling (wish I had better measurement tools; I like the Profiler,

>>>but can't find a way to track I/O utilization). I have not yet tested the

>>>bulk-import under VFP, the tests I'm referring to are different, but they

>>>are long running processor and I/O intensive jobs.

>>>

>>>Bill

>>>

>>>

>>>-----Original Message-----

>>>From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

>>>Sent: Sunday, August 29, 1999 9:57 PM

>>>To: Multiple recipients of ProFox

>>>Subject: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

>>>processing

>>>

>>>

>>>

>>>Hey, all:

>>>

>>>I have a chance to investigate a data conversion job. This would involve

>>>importing from a flat file into FP tables, formatting/ massaging it, then

>>>outputting it to another set of flat files. The company uses FPD (not sure

>>>which version) instead of VFP because "FPD is faster". To give you an idea

>>>of the size of the jobs involved, they take from ~1 hour to multiple days.

>>>

>>>This causes me to ask some questions:

>>>

>>>Would FPD really be faster for data conversion if you compared VFP under

>>>Win9x/NT to FPD running on the same machine booted to DOS only? Does the

>>>version of DOS make a significant difference? Does anyone have any hard

>>>data, educated guesses or gut feelings about this?

>>>

>>>Are there considerations besides speed (reliablility for example) to take

>>>into consideration?

>>>

>>>On a related note, has anyone clocked APPEND FROM... and COPY TO... against

>>>low-level file commands?

>>>

>>>I don't have to talk to these people for a couple of days and was curious

>>>if anyone else had looked into these issues.

>>>

>>>Any ideas?

>>>

>>>

>>>Thanks:

>>>

>>>Russ Scott

>>>russsct@ix.netcom.com

>>>

>>>

>>>

>>>

>>>

>>>

>>>

>>>

>>

>>

>>

>>

>

>

>

>

©1999 Matthew S. Jarvis, mjarvis@peak.org
Back to top
Re: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing

Author: Anders Altberg, anders.altberg@swipnet.se

Posted: 1999-08-30 03:40:33   Link

INSERT IN TO is faster, fewer locks for one thing, and INSERT .. FROM ARRAY

would probably be faster still.

-Anders

----- Original Message -----

From: Bill Arnold <bill@wjarnold.com>

To: Multiple recipients of ProFox <profox@leafe.com>

Sent: mÃ

’ndag den 30 augusti 1999 08:27

Subject: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

processing

| Russ, the low level file functions would speed things up if they could

| target a table, but having to do APPEND BLANKs kills it, for some reason

| that I don't understand. Even without indexes, the operation crawls. I

don't

| remember where I saw it, or perhaps I'm just making it up, but there

| shouldn't be any reason why a 'low level' routine (maybe this would be a

| good use for C) can't load a .dbf table (mimicking header and stuff).

| Imagine loading tables with the speed of, say, PKZIP?

|

| Bill

|

|

| -----Original Message-----

| From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

| Sent: Monday, August 30, 1999 1:20 AM

| To: Multiple recipients of ProFox

| Subject: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

| processing

|

|

|

| Thanks for the input, Bill:

|

| Since the data would probably require a couple of steps' worth of

massaging

| I was planning on a blanket APPEND FROM ... TYPE SDF then continuing from

| there (I'm not sure, not having seen the whole project) unless I learn

that

| low-level file functions are significantly faster. Unless they want to

| change from FPD, I don't get any .DBC capabilities, either, so a middle

| step of massaging seems likely anyway.

|

| Russ Scott

| russsct@ix.netcom.com

|

|

|

|

©1999 Anders Altberg, anders.altberg@swipnet.se
Back to top
Re: FoxPro DOS v. Visual Foxpro : Speed of data conversion / processing

Author: Anders Altberg, anders.altberg@swipnet.se

Posted: 1999-08-30 03:49:11   Link

You can change a FPD table to a VFP format table as simply as ADD TABLE x

-Anders

----- Original Message -----

From: Russ Scott <russsct@ix.netcom.com>

To: Multiple recipients of ProFox <profox@leafe.com>

Sent: mÃ

’ndag den 30 augusti 1999 07:19

Subject: RE: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

processing

|

| Thanks for the input, Bill:

|

| Since the data would probably require a couple of steps' worth of

massaging

| I was planning on a blanket APPEND FROM ... TYPE SDF then continuing from

| there (I'm not sure, not having seen the whole project) unless I learn

that

| low-level file functions are significantly faster. Unless they want to

| change from FPD, I don't get any .DBC capabilities, either, so a middle

| step of massaging seems likely anyway.

|

| Russ Scott

| russsct@ix.netcom.com

|

| ====================================

| At 10:17 PM 8/29/99 -0400, you wrote:

| >Russ,

| >

| >This is a matter of great concern to me too, and I posted a question and

got

| >some interesting responses not too long ago on the subject. I have a

monthly

| >job that imports a few million records spread across almost a thousand

| >tables (ZIP4 tables for the entire US). Because the data has to be

massaged

| >on the way into tables, I originally setup the import process to (low

level

| >I/O) read the ASCII fixed fields table, do the work and then APPEND BLANK

to

| >add a record to the database.

| >

| >I sped up the process a little by APPENDing blank records in 1,000 record

| >"blocks" and then replacing the blank fields of each until the 1,000

empty

| >records was filled and repeated the process again.

| >

| >Working this way, the import takes something like 2 days on a P133

machine.

| >A faster machine probably would help, but relegating the job to a 'spare'

| >machine that just chugs away seems to be the least bothersome. One thing

I

| >did do is design the process so multiple machines could participate in a

| >time crunch, and that does work very well when needed.

| >

| >Suggestions I received for speeding up the process included using APPEND

| >TYPE SDF, which would bring in each table en masse, but that would have

to

| >be followed with another step to do the massaging needed to prepare the

| >tables for use, since UDFs can't be used on individual fields with APPEND

| >TYPE SDF (but that would only partially address the data massaging

| >requirements anyway). Another suggestion, by Anders, that seems to have a

| >lot of merit, but I didn't try it yet, is to use APPEND TYPE SDF but with

| >rules in the target DBC (has to be VFP and target has to be DBC, tho) ...

| >I'm anxious to try this, but can't get to it yet.

| >

| >In general, processing each input record and then adding it to a table

using

| >APPEND BLANK is very slow compared to a bulk-add using APPEND TYPE SDF.

| >

| >As for FPD versus VFP in terms of speed, I know some people think that

FPD

| >if faster, but my experience so far with VFP shows that VFP (6, anyway,

with

| >SP3 if that means anything) it's actually faster than equivalent FPD

apps. I

| >attribute the difference to better memory management in VFP, but that's

only

| >my gut feeling (wish I had better measurement tools; I like the Profiler,

| >but can't find a way to track I/O utilization). I have not yet tested the

| >bulk-import under VFP, the tests I'm referring to are different, but they

| >are long running processor and I/O intensive jobs.

| >

| >Bill

| >

| >

| >-----Original Message-----

| >From: profox@leafe.com [mailto:profox@leafe.com]On Behalf Of Russ Scott

| >Sent: Sunday, August 29, 1999 9:57 PM

| >To: Multiple recipients of ProFox

| >Subject: FoxPro DOS v. Visual Foxpro : Speed of data conversion /

| >processing

| >

| >

| >

| >Hey, all:

| >

| >I have a chance to investigate a data conversion job. This would involve

| >importing from a flat file into FP tables, formatting/ massaging it, then

| >outputting it to another set of flat files. The company uses FPD (not

sure

| >which version) instead of VFP because "FPD is faster". To give you an

idea

| >of the size of the jobs involved, they take from ~1 hour to multiple

days.

| >

| >This causes me to ask some questions:

| >

| >Would FPD really be faster for data conversion if you compared VFP under

| >Win9x/NT to FPD running on the same machine booted to DOS only? Does the

| >version of DOS make a significant difference? Does anyone have any hard

| >data, educated guesses or gut feelings about this?

| >

| >Are there considerations besides speed (reliablility for example) to take

| >into consideration?

| >

| >On a related note, has anyone clocked APPEND FROM... and COPY TO...

against

| >low-level file commands?

| >

| >I don't have to talk to these people for a couple of days and was curious

| >if anyone else had looked into these issues.

| >

| >Any ideas?

| >

| >

| >Thanks:

| >

| >Russ Scott

| >russsct@ix.netcom.com

| >

| >

| >

| >

| >

| >

| >

| >

|

|

|

©1999 Anders Altberg, anders.altberg@swipnet.se