up vote 0 down vote favorite

I wanted to take a file (text or binary) and fragment it into small pieces of a certain size (about 250-500kB), randomize the order of the fragments, and put it into another temporary fragmented file.

The un-fragmenting would then take the fragmented file, extract the pieces, put them in order and allow the original file to be intact.

This would be very easy for simple text-based ASCII files as you could use the C library functions (like sscanf) for formating/parsing the information. The one file could have a format then like 

(#### <fragment #> <fragment> ...)

However, I am not sure how one would do something like that with binary files.

I know one easy solution is to use separate files for the fragments like <.part1, .part2> files but this would be a bit ugly and wouldn't scale well to much larger files. It would be a lot better to just store it in one file.

Thanks a lot.

share|improve this question
Sounds like security through obscurity. Anyway you can continue to store your metadata as plaintext and use sscanf as before. You should look at fread (cplusplus.com/reference/cstdio/fread) for handling binary data directly. – Dave 19 hours ago
I figured. I'm just a bit worried about being able to extract the file fragments again to form the whole file in order. With text you can ensure that your labeling format will be consistent. With binary I'm not quite so sure. – Tim Haggard 19 hours ago
You need to add the size of each chunk: {#chunk ID, chunk size, data } .... – Bechir 19 hours ago
You'll have to clarify the problem. Usually binary is much more dependable than text. – Dave 19 hours ago
Can you share what you have done already? – Bechir 19 hours ago

2 Answers

up vote 1 down vote accepted

Try to use binary data only. In you fragmented file, follow the structure:

OFFSET SIZE  DESCRIPTION
     0    4  BLOCK NUMBER
     4    4  BLOCK SIZE IN BYTES
     8    ?  BLOCK DATA

Define a header structure:

typedef struct hdr
{
    uint32_t number;
    uint32_t size;
} hdr_t;

Code to work with it can look like:

void file_append(FILE *f, size_t block, size_t size, const void *data)
{
    hdr_t hdr;
    hdr.number = block;
    hdr.size = size;
    fwrite(&hdr, sizeof(hdr), 1, f);
    fwrite(data, size, 1, f);
}

And reading the data:

void file_read_chunk(FILE *f, size_t *block, size_t *size, void **data)
{
    hdr_t hdr;

    fread(&hdr, sizeof(hdr), f);
    *block = hdr.number;
    *size = hdr.size;
    *data = malloc(hdr.size);
    fread(*data, hdr.size, 1, f);
}
share|improve this answer
With a few modifications this worked perfectly. Thanks. – Tim Haggard 4 mins ago
up vote 2 down vote

Doing this with binary files is easiest of all, and also the fastest and most reliable. Your fragment files need a simple segment record that gives you the offset in the original file and length of the segment. The record might look like this:

typedef struct _Fragment
{
    unsigned long offset;
    unsigned long length;
} Fragment;

Writing your file would go like this:

Fragment fragment;
FILE *outFile;
unsigned long segmentOffset, segmentLength;
char segmentData[MAXSEGMENTLENGTH];

outFile = fopen(fileName, "wb");

while (ReadNextSegment(segmentData, &segmentOffset, &segmentLength))
{
    fragment.offset = segmentOffset;
    fragment.length = segmentLength;

    fwrite(header, sizeof(fragment), 1, outFile);
    fwrite(segmentData, 1, segmentLength, outFile);
}

fclose(outFile);

Reassembling the file is accomplished by reversing the process. Read each Fragment record, then read the following data with fread using fragment.length, then position to the correct offset in the target file using the fseek function and fragment.offset, and then write it using fwrite.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.