java.lang.OutOfMemoryError with mapping large file in FileChannel

[hamishmorgan]

When calling FileChannel.map on a large file, OutOfMemoryError is thrown.

Am unable to replicate this problem under on my own system.

[INFO] 
=== Running full thesaurus build === 

[INFO] 
Configuration:
 * Input instances file: /home/user1/Desktop/NLP/Data/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2
 * Output directory: /home/user1/Desktop/NLP/bybloStats/output1/thesauri (27.0GiB free)
 * Temporary directory: /home/user1/Desktop/NLP/bybloStats/output1/thesauri/tempdirtmp-8511953953287541189.tmp (27.0GiB free)
 * Character encoding: UTF-8
 * Num. Threads: 6
 * Memory used=5.5MiB, allocated=989.9MiB, available=1.9GiB
 * Java Spec: Java Platform API Specification 1.6, Sun Microsystems Inc.
 * Java VM: OpenJDK Client VM 20.0-b12, Sun Microsystems Inc.
 * Java Runtime: OpenJDK Runtime Environment 1.6.0_24-b24
 * OS: Linux 3.2.0-26-generic-pae i386
 * Start time: 22:59:04 o'clock BST Wednesday, 11 July 2012


[INFO] 
=== Stage 1 of 6: Enumerating Strings ===

[INFO] 
Configuration:
 * Input instances file: /home/user1/Desktop/NLP/Data/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2
 * Output enumerated instances file: /home/user1/Desktop/NLP/bybloStats/output1/thesauri/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2.enumerated
 * Output entry index: /home/user1/Desktop/NLP/bybloStats/output1/thesauri/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2.entry-index
 * Output feature index: /home/user1/Desktop/NLP/bybloStats/output1/thesauri/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2.feature-index
 * Start time: 22:59:05 o'clock BST Wednesday, 11 July 2012
 * Memory used=5.9MiB, allocated=989.9MiB, available=1.9GiB


Exception in thread "main" java.io.IOException: Map failed
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:803)
    at uk.ac.susx.mlcl.lib.io.CharFileChannel.insureMapped(CharFileChannel.java:402)
    at uk.ac.susx.mlcl.lib.io.CharFileChannel.read(CharFileChannel.java:316)
    at uk.ac.susx.mlcl.lib.io.Lexer.insureRemaining(Lexer.java:494)
    at uk.ac.susx.mlcl.lib.io.Lexer.advance0(Lexer.java:294)
    at uk.ac.susx.mlcl.lib.io.Lexer.advance(Lexer.java:279)
    at uk.ac.susx.mlcl.lib.io.TSV$Source.<init>(TSV.java:221)
    at uk.ac.susx.mlcl.byblo.io.TokenPairSource.open(TokenPairSource.java:102)
    at uk.ac.susx.mlcl.byblo.io.BybloIO.openInstancesSource(BybloIO.java:55)
    at uk.ac.susx.mlcl.byblo.commands.IndexingCommands$IndexInstances.openSource(IndexingCommands.java:240)
    at uk.ac.susx.mlcl.byblo.commands.AbstractCopyCommand.runCommand(AbstractCopyCommand.java:128)
    at uk.ac.susx.mlcl.byblo.commands.IndexingCommands.runCommand(IndexingCommands.java:76)
    at uk.ac.susx.mlcl.byblo.FullBuild.runIndex(FullBuild.java:405)
    at uk.ac.susx.mlcl.byblo.FullBuild.runCommand(FullBuild.java:294)
    at uk.ac.susx.mlcl.lib.commands.AbstractCommand.runCommand(AbstractCommand.java:163)
    at uk.ac.susx.mlcl.byblo.FullBuild.main(FullBuild.java:232)
Caused by: java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:800)
    ... 15 more
  Moved back to /home/user1/Desktop/NLP/bybloStats
   Byblo failed on input file: /home/user1/Desktop/NLP/Data/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55/medtest-tb-cb-ng-nl-nr-vhl-vhrpl-pr-cw-55-%0-2
   Fail Code: 1

[hamishmorgan]

Since this is an "Error", and as the documentation states a "reasonable application should not try to catch" these, it seem reasonable to conclude that this is caused by bug in the java platform. In this case it's OpenJDK which hasn't previously been tested.

There's a bunch of bug report for something that sounds similar, except they are all related to Oracle Implementation: here, here, here, here, and here. Sadly the problems only seems to occur on OpenJDK, so not sure if it's related.

[subhashlamba]

I am also facing same error when try to upload more then 200MB file.
Anyone have solution for this?