Java Question

[Zllion]

Hi Guys,

IMDB Data set . I need to import this data set into a spring boot application, for further processing. there are millions of records with these file.
I tried read file and insert into in memory DB , but its taking a lot of time to complete. Suggest me a more optimal way of loading such kind of large data set.

 

[shnDexter]

Try to read the file line by line, without reading the whole file at once

 

[Zllion]

Try to read the file line by line, without reading the whole file at once

Tried that also. actually , I try to load a file, which has 9 M records. Its taking so much time

 

[shnDexter]

You can break the files into smaller chunks and read it and process it parallely, i haven't done this by java, but i have used this method in python.

 

[uditha88]

Try Multi-Threads

 

[kinkon]

බම්ප්