custom INputFormat, hadoop

Question

Please help, I have the following sample data:

-21.33,45.677,1234,1245,1267,1290,1212,1111,10000,1902
-21.34,45.677,1264,1645,1266,1260,1612,1611,16000,1602
-21.35,45.677,1244,1445,1467,1240,1242,1211,11000,1912
-21.36,45.677,1231,1215,1217,1210,1212,1111,10010,1902

I want that my hadoop mapreduce code should consider the first two float entries as the key(-21.33,45.677) and the remaining integer entries as value (1234,1245,1267,1290,1212,1111,10000,1902).

I am not sure if it can be done with the existing FileInputFormats. So how shall I go about this knowing that the value should be used as array not text.

Also how should I change the inputSplit such that I am able to get multiple records at the same time in the map for computation.

Answer 1

The easiest way is to use the TextInputFormat and have your mapper make the split between the key and the value. The output key and value of your mapper could then both be Text.

Answer 2

Any reason why you can't just use TextInputFormat's <LongWritable, Text> input types, and performs the extraction and transformation accordingly?

If that really isn't acceptable, then consider using the ChainMapper - use one map to do the extraction and then pass those results to another mapperthat's expecting the key/values required.

Answer 3

The easiest way is to split the record with the delimiter as ','. Then in your mapper just take the first two values and append them to make the key. You have to use Text because you need one value corresponding to your Key. Some computations will be required to convert the key back to numerical values.