current community

your communities

more stack exchange communities

Stack Exchange
tour help
careers 2.0
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 4 down vote favorite
2

I'm trying to figure out why I am getting an OOM error even though the byte array I am initializing plus the currently used memory is less than the max heap size (1000MB). Right before the array is initialized I'm using 373MB with 117 free. When I try to initialize the array that takes up 371MB I get an error. The strange thing is that the error persists until I allocate 1.2G or more for the JVM.

373 + 371 is 744, I should still have 256MB free, this is driving me nuts. In a second case using 920mb with 117 free initializing a 918mb array takes at least 2800mb.

Is this somehow part of how java functions? If so is there a workaround so that something simple like an array copy operation can be done in less than 3n memory? (memory numbers are from Runtime and max heap size is set with -Xmx)

test.java:
byte iv[];
iv =new byte[32];
byte key[] = new byte[32];
new SecureRandom().nextBytes(iv);
new SecureRandom().nextBytes(key);
plaintext = FileUtils.readFileToByteArray(new File("sampleFile"));
EncryptionResult out = ExperimentalCrypto.doSHE(plaintext, key, iv);

ExperimentalCrypto.java:
public static byte[] ExperimentalCrypto(byte[] input ,byte[] key, byte[]iv){
if(input.length%32 != 0){
int length = input.length;
byte[] temp = null; 
System.out.println((input.length/32+1)*32 / (1024*1024));
temp=new byte[(input.length/32+1)*32]; // encounter error here
share|improve this question
1  
Have you tried to use a profiler to see what is using the memory. –  Juned Ahsan Oct 1 '13 at 6:25
1  
As you mentioned 117 MB was free and if you intialize array of 371 MB it will give error. you need to analyze why it was showing 117 free. Although your XMX is 1 GB, it might be getting utilised somewhere else in your program. Use a profiler to know where your memory is going. –  Vineet Kasat Oct 1 '13 at 6:31
5  
One possible reason is heap fragmentation. See also: stackoverflow.com/questions/9286934/… –  assylias Oct 1 '13 at 6:34
    
I have not used a profiler yet, I'll look into that. @VineetKasat 117 is the amount that is free whether xmx is 1gb(error) or 1.2gb(no error) and using -xms999 gives me 580 free, but I still get the error. Would Runtime.freeMemory() return a deceptive number somehow? I know 371mb of the used memory is an existing array, and the other two mb are mostly small primitives and a few 32byte arrays. –  kag0 Oct 1 '13 at 6:55
    
I suggest you to use visual vm to analyze the memory consumption. it will give you details about the instances and their memory consumption. As i mentioned earlier, memory could be used at other places at that time. just need to identify it. –  Vineet Kasat Oct 1 '13 at 7:04

1 Answer 1

up vote 4 down vote accepted

Typical JVM implementations split the Java heap into several parts dedicated to objects with a certain lifetime. Allocations of larger arrays typically bypass the stages for younger objects as these areas are usually smaller and to avoid unnecessary copying. So they will end up in the “old generation” space for which a size of ⅔ is not unusual. As you are using JVisualVM I recommend installing the plugin Visual GC which can show you a live view of the different memory areas and their fill state.

You can use the -XX:MaxPermSize=… and -XX:MaxNewSize=… startup options to reduce the sizes of the areas for the young and permanent generation and thus indirectly raise the fraction of the old generation’s area where your array will be allocated.

share|improve this answer
    
Makes sense, thanks.Is there a way to get those big arrays allocated without having to change the vm arguments? –  kag0 Oct 1 '13 at 22:01
    
Maybe you can use a direct ByteBuffer instead of the array. In a lot of places they can be used alternatively to a byte[]. docs.oracle.com/javase/7/docs/api/java/nio/… –  Holger Oct 1 '13 at 22:14
    
The method I'm working on is for encryption, and probably needs to return a byte[], so at some point it will use 2n memory for byte[]'s (original input in calling class and array to be returned). So I'm not sure how to avoid growing the vm to 3n in the process. Unless the buffer could somehow let me work on the original array from the calling class? –  kag0 Oct 1 '13 at 22:24
    
Two things come into my mind. First, when looking at the JFC, Cipher supports using ByteBuffer alternatively to byte[] docs.oracle.com/javase/7/docs/api/javax/crypto/Cipher.html Second, if you are implementing a block cipher you don’t need to process the entire file at once. Processing it in chunks of the block size or n*blocksize would work as well. Oh, and if that does not work for you: why not allocate the padded-sized array first and read the file directly into it? There’s no magic in readFileToByteArray. Then you don’t need two arrays at the same time. –  Holger Oct 1 '13 at 22:58
    
It's actually technically a stream cipher (block cipher using CTR mode), I do have methods already for processing block by block but what I'm working on now is a way to encrypt a given input in parallel, which is why I need the output array already initialized so when threads finish they can write to it and minimal memory usage. I'm just reading in from a file in order to have a large byte array to test on, the input might not always come from a file. –  kag0 Oct 1 '13 at 23:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.