Pig UDF Example

Pig UDF Example

Background

This little UDF will convert the first 8 bytes of an HBase key into a long.  The Key that we had was a composite key made up of two 8 Byte longs and we needed to convert the first 8 bytes and then the second 8 bytes to get them separately. So here is how we did it.  

I have also left in the imports and you will need to find the jar files that contain these classes.  

Implementation

package allan.myudf;
import java.io.IOException;

import org.apache.hadoop.hbase.util.Bytes;
import org.apache.pig.EvalFunc;
import org.apache.pig.backend.hadoop.hbase.HBaseBinaryConverter;
import org.apache.pig.data.DataByteArray;
import org.apache.pig.data.Tuple;


public class ConvertFirst extends EvalFunc<Long> {
 public Long exec(Tuple input) throws IOException {
  if (input != null && input.size() == 1) {
   try {
    DataByteArray a = (DataByteArray) input.get(0);
    HBaseBinaryConverter b = new HBaseBinaryConverter();
    return Bytes.toLong(b.toBytes(a),0,8);
   

                        } catch (IllegalArgumentException e) {
    System.err.println("...");
   }
  }
  return null;
 }
}

Pig UDF - Converting HBase Key to Long

Pig UDF - Converting HBase Key to Long

Background

This little UDF will convert the first 8 bytes of an HBase key into a long.  The Key that we had was a composite key made up of two 8 Byte longs and we needed to convert the first 8 bytes and then the second 8 bytes to get them separately. So here is how we did it.  

I have also left in the imports and you will need to find the jar files that contain these classes.  

Implementation

package allan.myudf;
import java.io.IOException;

import org.apache.hadoop.hbase.util.Bytes;
import org.apache.pig.EvalFunc;
import org.apache.pig.backend.hadoop.hbase.HBaseBinaryConverter;
import org.apache.pig.data.DataByteArray;
import org.apache.pig.data.Tuple;


public class ConvertFirst extends EvalFunc<Long> {
 public Long exec(Tuple input) throws IOException {
  if (input != null && input.size() == 1) {
   try {
    DataByteArray a = (DataByteArray) input.get(0);
    HBaseBinaryConverter b = new HBaseBinaryConverter();
    return Bytes.toLong(b.toBytes(a),0,8);
   
                        } catch (IllegalArgumentException e) {
    System.err.println("...");
   }
  }
  return null;
 }
}

Retrieving Composite Key (HBase)

Retrieving Composite Key (HBase)

Its common to want to have the key of your HBase table to be made up more than one component, especially to ensure that your key stays unique.  There is a lack of documentation on how this can be done, but it is pretty easy. HBase orders its keys and allows you to filter by the key prior to loading that data in from region servers and therefore a composite key is a powerful tool.

Creating the Key

Creating your composite key is easy.  You can use longs, strings, integers, whatever as long as Apache supports it so check the supported methods. 

long1 = 11111111;
long2 = 22222222;
Bytes.add(Bytes.toLong(long1), Bytes.toLong(long2));

Retrieving the Key

You will need to understand how many bytes long your data is.  A long is 8 bytes in size so the key can be retrieved by first retrieving the first lot of 8 bytes, then retrieving the second lot of 8 bytes.

//res is a Result object
byte[] rowkey = res.getRow();
long key1 = Bytes.toLong(rowkey, 0, 8);
long key2 = Bytes.toLong(rowkey, 8, 8);