Cannot read large file (2Gb) by datatable (error 132 occur)

[duybluemind1988]

Hi,

I try to read DAT file in my computer as below:

path='V04R-V04R-SQLData_Ric.dat'      # 2.2 GB
df2=dt.fread(path,sep='\t',header=None,fill=True)

An error occur :

IOError: Unable to obtain size of V04R-V04R-SQLData_Ric.dat: [errno 132] value too large

But when I read another DAT file with lower size (800 Mb) everything is fine. Could you help in this case ?

Thanks!

[oleksiyskononenko]

@duybluemind1988 could you please give us some more details? What is the .dat file you are trying to read and what is your platform/python/datatable version?

[duybluemind1988]

@oleksiyskononenko :
I try to read 2GB DAT file in jupyter lab as information below:

Datatable version: 0.11.0
Python version: 3.8.3

problem:

Here is the example of my DAT file (800 mb) read by Datatable (separate by tab):

Time to read this file by dt.fread: 14 min (Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz, 3300 Mhz, 2 Core(s), 4 Logical Processor(s); RAM 8 GB) --> very long ?

Another information:
When I try to read this 2GB DAT file in google colab: everything OK in first time, but when I read this file again today here is the problem:

Datatable version: 0.11.0
Python version: 3.6.9

[oleksiyskononenko]

Thanks, what is the operating system here? Linux/Mac/Windows?

[duybluemind1988]

The operation system in my company is Windows 10 . Could you please help ? Many thank!

[oleksiyskononenko]

This issue will be fixed in the new release 0.11.1 that we're preparing now.

[oleksiyskononenko]

Please, try the new released version of datatable and see if it fixes the problem.