hat are the configuration parameters in a “MapReduce” program ?

Mapreduce Example
  • The main configuration parameters in “MapReduce” framework are:
    • Input location of Jobs in the distributed file system
    • Output location of Jobs in the distributed file system
    • The input format of data
    • The output format of data
    • The class which contains the map function
    • The class which contains the reduce function
    • JAR file which contains the mapper, reducer and the driver classes
Mapreduce configuration parameters

Sample Code

public static void main(String [] args) throws Exception
{
    Configuration c=new Configuration();
    String[] files=new GenericOptionsParser(c,args).getRemainingArgs();
    Path input=new Path(files[0]);
    Path output=new Path(files[1]);
    Job j=new Job(c,"wordcount");
    j.setJarByClass(WordCount.class);
    j.setMapperClass(MapForWordCount.class);
    j.setReducerClass(ReduceForWordCount.class);
    j.setOutputKeyClass(Text.class);
    j.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(j, input);
    FileOutputFormat.setOutputPath(j, output);
    System.exit(j.waitForCompletion(true)?0:1);
}

Categorized in:

Big Data

Tagged in:

, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,