[Solved-4 Solutions] Error in pig while loading data ?
Problem
The following error came while loading data:
ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error whiletrying to run jobs.java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected.
How to handle this ? The following error came while loading data:
Solution 1:
- Apache Pig 0.12.0 expects an older version of Hadoop by default. we must recompile Pig for Hadoop 2.2.0 and replace two jars with new pig-0.12.1-SNAPSHOT.jar and pig-0.12.1-SNAPSHOT-withouthadoop.jar.
- For recompilation unpack the pig archive, go to the directory "pig-0.12.0 and just run:
ant clean jar-all -Dhadoopversion=23Solution 2:
- The same problem on CDH4.4 & Pig 0.11.0 when Pig script was invoking a UDF from java project which was compiled using Maven. we visited /usr/lib/pig/conf/build.properties file.
- Verified the versions mentioned against hadoop-core, hadoop-common, hadoop-mapreduce properties. Ensured that all these artifacts with same versions are included as dependencies in my java project's POM.xml file.
- After building my project's jar file with these POM settings, Pig script was able to invoke UDF without any problem.
Solution 3:
- Just building with command "ant clean jar-all - Dhadoopversion=23" is not enough if we are using maven dependencies in your project. Its need to install the jar created by this in local maven repo or use this dependency (notice "classifier" tag for hadoop2) in your pom.xml
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<classifier>h2</classifier>
<version>0.13.0</version>
Solution 4:
- Just caught the same exception, trying to run script with my UDF on Hadoop 2x.
- In latest release 0.14 and current trunk 0.15, ant target jar-all is missing. So if we are building from sources, we can use
jar-h12 Create pig for both Hadoop 1 and Hadoop 2
or
ant jar -Dhadoopversion=23