How to speed up reading a file ?

Charley,

I doubt it's the reading part - it's probably whatever you're doing in the ... bit. I just created a file with 1500000 lines, and then used:

procedure( QQ(file)
   prog( ()
      inPort=infile(file)
           while( gets(inLine inPort)
              t
          ) ;while
       inPort=nil
  ) ;prog
) ;procedure

I ran this reading the file (over the network, not local disk) through the profiler, and got:

Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)                 1.91     1.91
QQ                                    1.41     0.14
toplevel                              1.41     0.00
_gets                                 1.27     1.27
gc                                    0.49     0.49

So as you can see, a total of 2 seconds.

I then generated a much bigger (longer lines) file with 1800000 lines (95Mbytes), and the profile results were:

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)                 3.58     3.58
QQ                                    2.91     0.19
toplevel                              2.91     0.00
_gets                                 2.72     2.72
gc                                    0.67     0.67

I suggest you run the SKILL profiler to see where the bottleneck is.

Andrew.

Andrew,

I read a file which has 14844545 lines , It spend 7 mins , If case2 , it needs 7:50 mins , It through network to access. Not local disk.

If has another for loop and use parseString to remove \n , it needs more time. I can't do anything , just wait.

Q1:How to remove each line of \n  ? I use parseString , It needs 50 sec ,

Q2:Can I use another way yo read file ?( I only know this way)

Q3:Can I separate file to many files to read ?  (Not single file single line , Is multi-file single line)

procedure( QQ(file)
   prog( ()
      inPort=infile(file)
           while( gets(inLine inPort)
                   qq = parseString(inLine "\n")  ;; case2

                   ;t                                        ;;case1
          ) ;while
       inPort=nil
  ) ;prog
) ;procedure

Thank you very much,

Charley

Charley,

I'm a bit confused, because suddenly your file is 10 times bigger, and it still takes 7 minutes to read. I just tried reading one with 29097984 lines (i.e. roughly double yours ~ 29 million, and about 1.5Gbytes long), and in the version of QQ I used (without the parseString - so your case1) it took:

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               113.90   113.90
toplevel                             96.89     0.01
QQ                                   96.88     5.17
_gets                                91.71    91.71
gc                                   17.01    17.01

Less than 2 minutes to read such a huge file is not that bad. This is on my laptop (which is nearly 4 years old, so not state of the art, and on a 100Mbit/s ethernet link).

In your case2 (with the parseString) it takes:

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               148.23   148.23
QQ                                  127.00     5.79
toplevel                            127.00     0.00
parseString                          67.29    67.29
_gets                                53.92    53.92
gc                                   21.23    21.23

Interesting, the gets was quicker (not sure why), but parseString is taking a while. Even so, it's hardly slow.

I also tried this - similar:

procedure( QQ(file)
   prog( ()
      inPort=infile(file)
           while( gets(inLine inPort)
                   qq = substring(inLine 1 sub1(strlen(inLine)))

                   ;t                                        ;;case1
          ) ;while
       inPort=nil
  ) ;prog
) ;procedure

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               148.73   148.73
QQ                                  122.53    11.56
toplevel                            122.53     0.00
_gets                                52.88    52.88
substring                            52.15    52.15
gc                                   26.20    26.20
strlen                                3.40     3.40
sub1                                  2.54     2.54

Regards,

Andrew.

Charley,

The above figures were for IC5141. In IC615 your "case2" example was quicker still:

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               119.28   119.28
toplevel                            119.28     0.01
QQ                                  109.78    12.03
parseString                          57.11    57.11
_gets                                40.65    40.65
gc                                    9.47     9.47
parser                                0.01     0.01

Regards,

Andrew.

The formatting is all messed up, so it's hard for me to see what you're doing.

Also, why are you reading such enormous files? Seems an odd thing to be doing.

Regards,

Andrew.

Andrew,

I need to read maybe thousand or  ten thousand or G line from a  file to do something.

But only read file and use parseString or parseString twice , It seems hangs up.

Now I want to break down to check what is the buttleneck.

Thank you,

Charley

That's what the profiler is for. Anyway, one thing that can happen is if you end up creating a lot of garbage - it can spend a lot of time in garbage collection (gc) - so usually the trick is to avoid creating garbage in the first place.

Andrew.

Andrew,

(1)How to use  profiler  ?

(2) 

I remove a line with \n using parseString( nextLine "\n")  and parseString(nextLine " ")  to get data .

Is there any command is better to do it ?

Thank you,

Charley

Charley,

1. Read the SKILL documentation - we provide documentation for a good reason. In the Tools->SKILL Development Toolbox there's a "Profiler" and you can also use the profile(), profileReset() and profileSummary() functions to do this.
2. It rather depends on what you are trying to do with the results of the second parseString. I gave an alternative to using the first parseString (substring). Another possibility is to use the rex functions (rexCompile, rexReplace, rexSubstitute) or (in IC61) pcre functions. I'll leave it to you to determine what is the fastest for your application.

Regards,

Andrew.