Author: MB Software Solutions, LLC
Posted: 2011-08-09 14:40:37 Link
How do I determine the # of rows in a fixed-width flat file? I want to
provide a progress update in the loop so I know how much % of the file
has been processed. I'm trying to avoid reading the entire file in with
ALINES. (But I guess maybe I could do that but it seems like too much
overhead.)
Thanks!
--Mike
--
Mike Babcock, MCP
MB Software Solutions, LLC
President, Chief Software Architect
http://mbsoftwaresolutions.com
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E417F25.6060907@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: Jarvis, Matthew
Posted: 2011-08-09 14:55:12 Link
> How do I determine the # of rows in a fixed-width flat file? I want
to
> provide a progress update in the loop so I know how much % of the file
> has been processed. I'm trying to avoid reading the entire file in
with
> ALINES. (But I guess maybe I could do that but it seems like too much
> overhead.)
Oh the irony.... ALINES *should* be a perfect fit for this...
Does each line by chance have a marker of some sort i.e. a string like
"Begin:" or something?
If so maybe you could get away with using the OCCURS() function...
Thanks,
Matthew Jarvis || Business Systems Analyst
IT Department
McKenzie-Willamette Medical Center
1460 G Street, Springfield, OR 97477 || Ph: 541-744-6092 || Fax:
541-744-6145
--------------------------------------------------------------------------
Disclaimer: This electronic message may contain information that is
Proprietary, Confidential, or legally privileged or protected. It
is intended only for the use of the individual(s) and entity named
in the message. If you are not an intended recipient of this
message, please notify the sender immediately and delete the
material from your computer. Do not deliver, distribute or copy
this message and do not disclose its contents or take any action in
reliance on the information it contains.
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/69F310C05DD83C48A84BA3769CE1ECF8057C9EFE@TNTRIEXEVS02.triadhospitals.net
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: Tracy Pearson
Posted: 2011-08-09 14:57:23 Link
MB Software Solutions, LLC wrote on 2011-08-09:
> How do I determine the # of rows in a fixed-width flat file? I want to
> provide a progress update in the loop so I know how much % of the file
> has been processed. I'm trying to avoid reading the entire file in with
> ALINES. (But I guess maybe I could do that but it seems like too much
> overhead.)
>
> Thanks!
> --Mike
Mike,
It would be a two pass thing, or a best guess. I would go with the best
guess.
A tight loop getting the data and dumping it as you get a total count, then
return to the top and begin processing.
If you did this, since you have multiple types of data in the file, you
could get a count of each type. Then show the progress of each type as you
do the real processing.
The best guess, FSEEK() to the end returns the total number of bytes. You
can use this and add the bytes of each line to show the complete progress.
Tracy Pearson
PowerChurch Software
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/001501cc56c6$2ce15520$86a3ff60$@powerchurch.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: Mike Copeland
Posted: 2011-08-09 15:02:03 Link
MB Software Solutions, LLC wrote on 2011-08-09:
>> How do I determine the # of rows in a fixed-width flat file? I want to
>> provide a progress update in the loop so I know how much % of the file
>> has been processed. I'm trying to avoid reading the entire file in with
>> ALINES. (But I guess maybe I could do that but it seems like too much
>> overhead.)
>>
>> Thanks!
>> --Mike
> Mike,
>
> It would be a two pass thing, or a best guess. I would go with the best
> guess.
> A tight loop getting the data and dumping it as you get a total count, then
> return to the top and begin processing.
> If you did this, since you have multiple types of data in the file, you
> could get a count of each type. Then show the progress of each type as you
> do the real processing.
>
> The best guess, FSEEK() to the end returns the total number of bytes. You
> can use this and add the bytes of each line to show the complete progress.
>
>
> Tracy Pearson
> PowerChurch Software
Just test for FEOF() with each iteration. I've never had a problem doing
that to "find the end."
Mike Copeland
Genesis Group
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E41842B.6020002@ggisoft.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: Stephen Russell
Posted: 2011-08-09 15:02:44 Link
On Tue, Aug 9, 2011 at 1:40 PM, MB Software Solutions, LLC
<mbsoftwaresolutions@mbsoftwaresolutions.com> wrote:
> How do I determine the # of rows in a fixed-width flat file? I want to
> provide a progress update in the loop so I know how much % of the file
> has been processed. I'm trying to avoid reading the entire file in with
> ALINES. (But I guess maybe I could do that but it seems like too much
> overhead.)
You are possibly blowing the wad on the string that you are generating?
Loop of reading lines
Str = str + "contents of this new line"
end
Even though it is the same name you are duplicating in memory what you
had to make the appended string.
I don't know when your pointers get released in VFP strings but as a
WAG this is where your memory hole is.
Have played with 1.2 and 1.5 gig xml files in ram before with no real
problem on workstations that only had 3 gig of ram. Teh process that
did monthly reporting I expanded to annual. ;-> made a bit bigger
output didn't it.
--
Stephen Russell
Unified Health Services
60 Germantown Court
Suite 220
Cordova, TN 38018
Telephone: 888.510.2667
901.246-0159 cell
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/CAJidMYLj2XFu=bVDzF=A=uJM3LPpHNJb-fRAraZyxfbpc4yB=A@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: MB Software Solutions, LLC
Posted: 2011-08-09 15:08:22 Link
On 8/9/2011 2:55 PM, Jarvis, Matthew wrote:
> Oh the irony.... ALINES *should* be a perfect fit for this...
>
> Does each line by chance have a marker of some sort i.e. a string like
> "Begin:" or something?
>
> If so maybe you could get away with using the OCCURS() function...
No, no such marker.
--
Mike Babcock, MCP
MB Software Solutions, LLC
President, Chief Software Architect
http://mbsoftwaresolutions.com
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E4185A6.5040406@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: MB Software Solutions, LLC
Posted: 2011-08-09 15:23:31 Link
On 8/9/2011 3:02 PM, Mike Copeland wrote:
>> Mike,
>>
>> It would be a two pass thing, or a best guess. I would go with the best
>> guess.
>> A tight loop getting the data and dumping it as you get a total count, then
>> return to the top and begin processing.
>> If you did this, since you have multiple types of data in the file, you
>> could get a count of each type. Then show the progress of each type as you
>> do the real processing.
>>
>> The best guess, FSEEK() to the end returns the total number of bytes. You
>> can use this and add the bytes of each line to show the complete progress.
>>
>>
> Just test for FEOF() with each iteration. I've never had a problem doing
> that to "find the end."
Oh I know to use FEOF to test for the end of the loop, but I was
thinking that I could use FSIZE (with SET COMPATIBLE DB4) to get the
filesize in bytes, and then divide that by the record length, but
apparently my logic is flawed, because my loop was telling me
"Processing record 990123 of 636789" -- LOL
Doing the tight loop as Tracy mentioned should work, but it seems like a
kludge (but a necessary one!). I *do* like the idea of telling the user
how many records of each type ahead of time though!
Here's my logic so far (but the record count is off by seemingly twice
as much. Can you see why??!?!?
lcFileType = UPPER(RIGHT(JUSTSTEM(This.InputFile),2))
DO CASE
CASE lcFileType = "OP"
lnLength = 550
lnRecPosn = 54
* put other lengths in later after OP passes testing
OTHERWISE
SET STEP ON
ENDCASE
lnRows = lnFileSize / lnLength
liHandle = FOPEN(this.InputFile)
liCnt = 0
DO WHILE NOT FEOF(liHandle)
m.liCnt = m.liCnt + 1
m.LineOfText = FGETS(m.liHandle,m.lnLength) && according to VFP Help,
rec ptr is moved, so I don't need to use FSEEK
m.RecordType = SUBSTR(m.LineOfText,m.lnRecPosn,1)
WAIT WINDOW NOWAIT "Processing line " + ALLTRIM(STR(liCnt)) + " of " +
ALLTRIM(STR(lnRows))
DO CASE
CASE m.RecordType = "1"
This.ProcessType1(m.LineOfText)
CASE m.RecordType = "2"
This.ProcessType2(m.LineOfText)
CASE m.RecordType = "3"
This.ProcessType3(m.LineOfText)
CASE m.RecordType = "4"
This.ProcessType4(m.LineOfText)
OTHERWISE
* unknown type or EOF marker
IF LEN(m.LineOfText) > 1
SET STEP ON
This.LogIt("UT", "", m.LineOfText)
m.RogueCntr = m.RogueCntr + 1
This.ErrorCondition = .T.
*** mjb 2011/06/08 - changing so that if ANY errors occur,
program does not continue towards output file.
m.MsgTxt = "This file has an error. Processing will not
continue." + CHR(13)
m.MsgTxt = m.MsgTxt + "Please review DSHImportExport.LOG to see
the invalid data."
MESSAGEBOX(m.MsgTxt, 16, "Error: Invalid Data")
m.RetVal = .F.
EXIT
ENDIF
ENDCASE
ENDDO
FCLOSE(liHandle)
--
Mike Babcock, MCP
MB Software Solutions, LLC
President, Chief Software Architect
http://mbsoftwaresolutions.com
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E418933.5000909@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: MB Software Solutions, LLC
Posted: 2011-08-09 15:26:38 Link
On 8/9/2011 3:02 PM, Stephen Russell wrote:
> You are possibly blowing the wad on the string that you are generating?
>
> Loop of reading lines
> Str = str + "contents of this new line"
>
> end
>
> Even though it is the same name you are duplicating in memory what you
> had to make the appended string.
>
> I don't know when your pointers get released in VFP strings but as a
> WAG this is where your memory hole is.
Keep your money! :-) That's not it. But I appreciate the idea,
though, nonetheless.
>
> Have played with 1.2 and 1.5 gig xml files in ram before with no real
> problem on workstations that only had 3 gig of ram. Teh process that
> did monthly reporting I expanded to annual. ;-> made a bit bigger
> output didn't it.
Aren't XML files supposed to be small in size? lol
--
Mike Babcock, MCP
MB Software Solutions, LLC
President, Chief Software Architect
http://mbsoftwaresolutions.com
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E4189EE.6060107@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: Tracy Pearson
Posted: 2011-08-09 16:32:53 Link
Mike,
You know the record length of the file in question?
liHandle = FOPEN(this.InputFile)
if liHandle > 0
lnFileSize = fseek(liHandle, 0, 2) && get the file size
=fseek(liHandle, 0, 0) && put the pointer back at the begining
lnRows = lnFileSize / lnLength
DO WHILE NOT FEOF(liHandle)
&& ...
ENDDO
ENDIF
This won't work if the data is not a fixed width.
Otherwise you can use LEN() to add up the size of each returned line and
display a percentage of completion.
lnAmountRead = lnAmountRead + LEN(m.LineOfText)
lnPercent = lnAmountRead / lnFileSize
WAIT WINDOW NOWAIT " Processing " + trans(lnPercent * 100, "999.99%")
Tracy Pearson
PowerChurch Software
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/001f01cc56d3$83d19360$8b74ba20$@powerchurch.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Author: MB Software Solutions, LLC
Posted: 2011-08-09 16:46:39 Link
On 8/9/2011 4:32 PM, Tracy Pearson wrote:
> Mike,
>
> You know the record length of the file in question?
>
> liHandle = FOPEN(this.InputFile)
> if liHandle> 0
> lnFileSize = fseek(liHandle, 0, 2)&& get the file size
> =fseek(liHandle, 0, 0)&& put the pointer back at the begining
> lnRows = lnFileSize / lnLength
> DO WHILE NOT FEOF(liHandle)
> && ...
> ENDDO
> ENDIF
I had tried that with FSIZE but no luck. I'll try your way now. (I
read the FSEEK to just move the pointer...I didn't realize it would
return the length. It's been that kind of day I guess, as I know I read
the VFP Help screens.)
--
Mike Babcock, MCP
MB Software Solutions, LLC
President, Chief Software Architect
http://mbsoftwaresolutions.com
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/4E419CAF.9080809@mbsoftwaresolutions.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.