Pig Development Guide

1. Simple example

Upload data to the hdfs directory


[hadoop@uhadoop-******-master1 pig]$ hadoop fs -put /etc/passwd /user/hadoop/passwd

Start pig


[hadoop@uhadoop-******-master1 pig]$ pig

Load data


grunt> A = load 'passwd' using PigStorage(':');
grunt> dump A;

Display results:


(root,x,0,0,root,/root,/bin/bash) 
……

2. Use UDF

Prepare data

Content of the student file


any 9 5
bob 8 4

Upload the student file


hdfs dfs -put student /user/root/student

Sample code


package myudfs;
import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
 
public class UPPER extends EvalFunc<String>
{
    public String exec(Tuple input) throws IOException {
        if (input == null || input.size() == 0 || input.get(0) == null)
            return null;
        try{
            String str = (String)input.get(0);
            return str.toUpperCase();
        }catch(Exception e){
            throw new IOException("Caught exception processing input row ", e);
        }
    }
}

Compile


cd myudfs
javac -cp $ PIG_HOME/pig-0.12.0-cdh5.4.4.jar UPPER.java
cd ..
jar -cf myudfs.jar myudfs

Test script upper.pig


REGISTER myudfs.jar;
A = LOAD 'student' AS (name: chararray, age: int, gpa: float);
B = FOREACH A GENERATE myudfs.UPPER(name);
DUMP B;

Execute


pig upper.pig

Output result

(ANY 9 5) (BOB 8 4)