How to speed up reading a file ?

Charley,

I'm a bit confused, because suddenly your file is 10 times bigger, and it still takes 7 minutes to read. I just tried reading one with 29097984 lines (i.e. roughly double yours ~ 29 million, and about 1.5Gbytes long), and in the version of QQ I used (without the parseString - so your case1) it took:

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               113.90   113.90
toplevel                             96.89     0.01
QQ                                   96.88     5.17
_gets                                91.71    91.71
gc                                   17.01    17.01

Less than 2 minutes to read such a huge file is not that bad. This is on my laptop (which is nearly 4 years old, so not state of the art, and on a 100Mbit/s ethernet link).

In your case2 (with the parseString) it takes:

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               148.23   148.23
QQ                                  127.00     5.79
toplevel                            127.00     0.00
parseString                          67.29    67.29
_gets                                53.92    53.92
gc                                   21.23    21.23

Interesting, the gets was quicker (not sure why), but parseString is taking a while. Even so, it's hardly slow.

I also tried this - similar:

procedure( QQ(file)
   prog( ()
      inPort=infile(file)
           while( gets(inLine inPort)
                   qq = substring(inLine 1 sub1(strlen(inLine)))

                   ;t                                        ;;case1
          ) ;while
       inPort=nil
  ) ;prog
) ;procedure

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               148.73   148.73
QQ                                  122.53    11.56
toplevel                            122.53     0.00
_gets                                52.88    52.88
substring                            52.15    52.15
gc                                   26.20    26.20
strlen                                3.40     3.40
sub1                                  2.54     2.54

Regards,

Andrew.

Charley,

I'm a bit confused, because suddenly your file is 10 times bigger, and it still takes 7 minutes to read. I just tried reading one with 29097984 lines (i.e. roughly double yours ~ 29 million, and about 1.5Gbytes long), and in the version of QQ I used (without the parseString - so your case1) it took:

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               113.90   113.90
toplevel                             96.89     0.01
QQ                                   96.88     5.17
_gets                                91.71    91.71
gc                                   17.01    17.01

Less than 2 minutes to read such a huge file is not that bad. This is on my laptop (which is nearly 4 years old, so not state of the art, and on a 100Mbit/s ethernet link).

In your case2 (with the parseString) it takes:

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               148.23   148.23
QQ                                  127.00     5.79
toplevel                            127.00     0.00
parseString                          67.29    67.29
_gets                                53.92    53.92
gc                                   21.23    21.23

Interesting, the gets was quicker (not sure why), but parseString is taking a while. Even so, it's hardly slow.

I also tried this - similar:

procedure( QQ(file)
   prog( ()
      inPort=infile(file)
           while( gets(inLine inPort)
                   qq = substring(inLine 1 sub1(strlen(inLine)))

                   ;t                                        ;;case1
          ) ;while
       inPort=nil
  ) ;prog
) ;procedure

 Function Name                        Total   Inside
-------------                        -----   ------
TOTAL CPU Time (secs)               148.73   148.73
QQ                                  122.53    11.56
toplevel                            122.53     0.00
_gets                                52.88    52.88
substring                            52.15    52.15
gc                                   26.20    26.20
strlen                                3.40     3.40
sub1                                  2.54     2.54

Regards,

Andrew.