24 Million entries and I need to what?

Lonnie Olson lists at kittypee.com
Fri Dec 27 10:29:18 MST 2013


On Fri, Dec 27, 2013 at 10:26 AM, S. Dale Morrey <sdalemorrey at gmail.com> wrote:
> Sorting it was based on the premise that I could load it into a hashmap or
> dictionary type structure.  Unfortunately I keep running out of ram with
> every Java method I can think of to store this thing.  My initial thought
> was a HashSet, but that didn't work at all, due to running out of memory.
>
> Splitting it out and recreating it as a bunch of files just gives me a hack
> where I can check to see if the file exists.  In a RAM disk such as /tmp
> I'm guessing that this would no longer be bound to the spin rate of the
> disk.  I have 4GB of RAM, my OS is using 1.5GB of that.  I'm hoping that I
> won't suddenly deprive my system of RAM by dumping the contents to /tmp.

Try using some other data structure then.  Maybe something simpler
will fit into your available RAM.  Perl has Hashes, and those would
work fine.

Or just use some other binary indexing method.  MySQL, Postgres, SQLite, BDB.


More information about the PLUG mailing list