Stats

  • Locked Locked
  • Replies 10
  • Subscribers 143
  • Views 16030
  • Members are here 0
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

How to speed up reading a file ?

Charley Chen

Hi All,

 I read a file which has 1386213 line  or more , only read each line  , it spend  7 minutes  and above.

How can it speed up ?

 

procedure( QQ(file)
   prog( ()
      inPort=infile(file)
           while( gets(inLine inPort)

               ........
           ) ;if
      ) ;while
       inPort=nil
  ) ;prog
) ;procedure

 

Thank you,

Charley

Parents
  • Andrew Beckett

    Charley,

    I doubt it's the reading part - it's probably whatever you're doing in the ... bit. I just created a file with 1500000 lines, and then used:

    procedure( QQ(file)
       prog( ()
          inPort=infile(file)
               while( gets(inLine inPort)
                  t
              ) ;while
           inPort=nil
      ) ;prog
    ) ;procedure

    I ran this reading the file (over the network, not local disk) through the profiler, and got:

    Function Name                        Total   Inside
    -------------                        -----   ------
    TOTAL CPU Time (secs)                 1.91     1.91
    QQ                                    1.41     0.14
    toplevel                              1.41     0.00
    _gets                                 1.27     1.27
    gc                                    0.49     0.49

    So as you can see, a total of 2 seconds.

    I then generated a much bigger (longer lines) file with 1800000 lines (95Mbytes), and the profile results were:

     Function Name                        Total   Inside
    -------------                        -----   ------
    TOTAL CPU Time (secs)                 3.58     3.58
    QQ                                    2.91     0.19
    toplevel                              2.91     0.00
    _gets                                 2.72     2.72
    gc                                    0.67     0.67

    I suggest you run the SKILL profiler to see where the bottleneck is.

    Andrew.

Reply
  • Andrew Beckett

    Charley,

    I doubt it's the reading part - it's probably whatever you're doing in the ... bit. I just created a file with 1500000 lines, and then used:

    procedure( QQ(file)
       prog( ()
          inPort=infile(file)
               while( gets(inLine inPort)
                  t
              ) ;while
           inPort=nil
      ) ;prog
    ) ;procedure

    I ran this reading the file (over the network, not local disk) through the profiler, and got:

    Function Name                        Total   Inside
    -------------                        -----   ------
    TOTAL CPU Time (secs)                 1.91     1.91
    QQ                                    1.41     0.14
    toplevel                              1.41     0.00
    _gets                                 1.27     1.27
    gc                                    0.49     0.49

    So as you can see, a total of 2 seconds.

    I then generated a much bigger (longer lines) file with 1800000 lines (95Mbytes), and the profile results were:

     Function Name                        Total   Inside
    -------------                        -----   ------
    TOTAL CPU Time (secs)                 3.58     3.58
    QQ                                    2.91     0.19
    toplevel                              2.91     0.00
    _gets                                 2.72     2.72
    gc                                    0.67     0.67

    I suggest you run the SKILL profiler to see where the bottleneck is.

    Andrew.

Children
No Data

Community Guidelines

The Cadence Design Communities support Cadence users and technologists interacting to exchange ideas, news, technical information, and best practices to solve problems and get the most from Cadence technology. The community is open to everyone, and to provide the most value, we require participants to follow our Community Guidelines that facilitate a quality exchange of ideas and information. By accessing, contributing, using or downloading any materials from the site, you agree to be bound by the full Community Guidelines.