public abstract class SampleLoader extends LoadFunc
| Modifier and Type | Field and Description |
|---|---|
protected LoadFunc |
loader |
protected int |
numSamples |
| Constructor and Description |
|---|
SampleLoader(String funcSpec) |
| Modifier and Type | Method and Description |
|---|---|
void |
computeSamples(ArrayList<Pair<FileSpec,Boolean>> inputs,
PigContext pc) |
org.apache.hadoop.mapreduce.InputFormat<?,?> |
getInputFormat()
This will be called during planning on the front end.
|
LoadCaster |
getLoadCaster()
This will be called on both the front end and the back
end during execution.
|
int |
getNumSamples() |
void |
prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
PigSplit split)
Initializes LoadFunc for reading data.
|
String |
relativeToAbsolutePath(String location,
org.apache.hadoop.fs.Path curDir)
This method is called by the Pig runtime in the front end to convert the
input location to an absolute path if the location is relative.
|
void |
setLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the loader the location of the object(s) being loaded.
|
void |
setNumSamples(int n) |
void |
setUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
LoadFunc. |
boolean |
skipNext() |
getAbsolutePath, getCacheFiles, getNext, getPathStrings, getShipFiles, join, warnprotected int numSamples
protected LoadFunc loader
public SampleLoader(String funcSpec)
public void setNumSamples(int n)
public int getNumSamples()
public org.apache.hadoop.mapreduce.InputFormat<?,?> getInputFormat()
throws IOException
LoadFuncgetInputFormat in class LoadFuncIOException - if there is an exception during InputFormat
constructionpublic boolean skipNext()
throws IOException
IOExceptionpublic void computeSamples(ArrayList<Pair<FileSpec,Boolean>> inputs, PigContext pc) throws ExecException
ExecExceptionpublic LoadCaster getLoadCaster() throws IOException
LoadFuncgetLoadCaster in class LoadFuncLoadCaster associated with this loader. Returning null
indicates that casts from byte array are not supported for this loader.
constructionIOException - if there is an exception during LoadCasterpublic String relativeToAbsolutePath(String location, org.apache.hadoop.fs.Path curDir) throws IOException
LoadFuncrelativeToAbsolutePath in class LoadFunclocation - location as provided in the "load" statement of the scriptcurDir - the current working direction based on any "cd" statements
in the script before the "load" statement. If there are no "cd" statements
in the script, this would be the home directory -
/user/
IOException - if the conversion is not possiblepublic void prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
PigSplit split)
throws IOException
LoadFuncprepareToRead in class LoadFuncreader - RecordReader to be used by this instance of the LoadFuncsplit - The input PigSplit to processIOException - if there is an exception during initializationpublic void setLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadFuncLoadFunc.relativeToAbsolutePath(String, Path). Implementations
should use this method to communicate the location (and any other information)
to its underlying InputFormat through the Job object.
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.setLocation in class LoadFunclocation - Location as returned by
LoadFunc.relativeToAbsolutePath(String, Path)job - the Job object
store or retrieve earlier stored information from the UDFContextIOException - if the location is not valid.public void setUDFContextSignature(String signature)
LoadFuncLoadFunc. The signature can be used
to store into the UDFContext any information which the
LoadFunc needs to store between various method invocations in the
front end and back end. A use case is to store LoadPushDown.RequiredFieldList
passed to it in LoadPushDown.pushProjection(RequiredFieldList) for
use in the back end before returning tuples in LoadFunc.getNext().
This method will be call before other methods in LoadFuncsetUDFContextSignature in class LoadFuncsignature - a unique signature to identify this LoadFuncCopyright © 2007-2017 The Apache Software Foundation