[Solved-1 Solution] Hadoop Pig: Passing Command Line Arguments ?

What is command line arguments ?

It is possible to pass arguments to programs when they are executed.
We can indicate the input parameter on the command line and use that when we are loading.

Example:

Command Line:

pig -f script.pig -param input=somefile.txt

Problem:

Is there a way to do this ? eg, pass the name of the file to be processed, etc ?

Solution 1:

In Command Line: We can use the following

If there are few parameters then use -param (-p)
If there are lot of parameters then use -param_file (-m)

We can use either approach depending on the nature of value of command line arguments, we can use -param when we developing and testing scripts. Once pig script is ready for batch processing or running thru crontab, we use -param_file so that if any change required, we can easily update the params.init file.

Pig will show all the available options.

-m, -param_file path to the parameter file
-p, -param key value pair of the form param=val

The sample code for student is given below

Employees .txt (input data)

001,Rajiv,Reddy,21,9848022337,Hyderabad
002,siddarth,Battacharya,22,9848022338,Kolkata
003,Rajesh,Khanna,22,9848022339,Delhi

Params.init (file to hold all parameters)

fileName='hdfs://horton/user/jgosalia/students.txt'
cityName='Chennai'

Filter.pig

students = LOAD '$fileName' USING PigStorage(',') AS (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray);
students = FILTER students BY city == '$cityName';
DUMP students;

1. Using params on command line (-param or -p) & Output

pig -param fileName='hdfs://horton/user/jgosalia/students.txt' -param cityName='Chennai' filter.pig

... Trimming the logs ...

(6,Archana,Mishra,23,9848022335,Chennai)

2. Using params file on command line (-param_file or -m)&Output

pig -param_file params.init filter.pig

... Trimming the logs ...

(6,Archana,Mishra,23,9848022335,Chennai)
(8,Bharathi,Nambiayar,24,9848022333,Chennai)