@InterfaceAudience.Public @InterfaceStability.Stable public abstract class StoreFunc extends Object implements StoreFuncInterface
| Constructor and Description |
|---|
StoreFunc() |
| Modifier and Type | Method and Description |
|---|---|
void |
checkSchema(ResourceSchema s)
Set the schema for data to be stored.
|
void |
cleanupOnFailure(String location,
org.apache.hadoop.mapreduce.Job job)
This method will be called by Pig if the job which contains this store
fails.
|
static void |
cleanupOnFailureImpl(String location,
org.apache.hadoop.mapreduce.Job job)
Default implementation for
cleanupOnFailure(String, Job)
and cleanupOnSuccess(String, Job). |
void |
cleanupOnSuccess(String location,
org.apache.hadoop.mapreduce.Job job)
This method will be called by Pig if the job which contains this store
is successful, and some cleanup of intermediate resources is required.
|
abstract org.apache.hadoop.mapreduce.OutputFormat |
getOutputFormat()
Return the OutputFormat associated with StoreFunc.
|
abstract void |
prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
Initialize StoreFunc to write data.
|
abstract void |
putNext(Tuple t)
Write a tuple to the data store.
|
String |
relToAbsPathForStoreLocation(String location,
org.apache.hadoop.fs.Path curDir)
This method is called by the Pig runtime in the front end to convert the
output location to an absolute path if the location is relative.
|
void |
setStoreFuncUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
StoreFunc which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. |
abstract void |
setStoreLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the storer the location where the data needs to be stored.
|
Boolean |
supportsParallelWriteToStoreLocation()
DAG execution engines like Tez support optimizing union by writing to
output location in parallel from tasks of different vertices.
|
void |
warn(String msg,
Enum warningEnum)
Issue a warning.
|
public String relToAbsPathForStoreLocation(String location, org.apache.hadoop.fs.Path curDir) throws IOException
relToAbsPathForStoreLocation in interface StoreFuncInterfacelocation - location as provided in the "store" statement of the scriptcurDir - the current working direction based on any "cd" statements
in the script before the "store" statement. If there are no "cd" statements
in the script, this would be the home directory -
/user/
IOException - if the conversion is not possiblepublic abstract org.apache.hadoop.mapreduce.OutputFormat getOutputFormat()
throws IOException
getOutputFormat in interface StoreFuncInterfaceOutputFormat associated with StoreFuncIOException - if an exception occurs while constructing the
OutputFormatpublic abstract void setStoreLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFunc here is the
return value of relToAbsPathForStoreLocation(String, Path)
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.
checkSchema(ResourceSchema) will be called before any call to
setStoreLocation(String, Job).setStoreLocation in interface StoreFuncInterfacelocation - Location returned by
relToAbsPathForStoreLocation(String, Path)job - The Job objectIOException - if the location is not valid.public void checkSchema(ResourceSchema s) throws IOException
checkSchema in interface StoreFuncInterfaces - to be checkedIOException - if this schema is not acceptable. It should include
a detailed error message indicating what is wrong with the schema.public abstract void prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
throws IOException
prepareToWrite in interface StoreFuncInterfacewriter - RecordWriter to use.IOException - if an exception occurs during initializationpublic abstract void putNext(Tuple t) throws IOException
putNext in interface StoreFuncInterfacet - the tuple to store.IOException - if an exception occurs during the writepublic void setStoreFuncUDFContextSignature(String signature)
StoreFunc which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. This method
will be called before other methods in StoreFunc. This is necessary
because in a Pig Latin script with multiple stores, the different
instances of store functions need to be able to find their (and only their)
data in the UDFContext object. The default implementation is a no-op.setStoreFuncUDFContextSignature in interface StoreFuncInterfacesignature - a unique signature to identify this StoreFuncpublic void cleanupOnFailure(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
FileSystem location.cleanupOnFailure in interface StoreFuncInterfacelocation - Location returned by
relToAbsPathForStoreLocation(String, Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOExceptionpublic void cleanupOnSuccess(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
cleanupOnSuccess in interface StoreFuncInterfacelocation - Location returned by
relToAbsPathForStoreLocation(String, Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOExceptionpublic static void cleanupOnFailureImpl(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
cleanupOnFailure(String, Job)
and cleanupOnSuccess(String, Job). This removes a file
from HDFS.location - file name (or URI) of file to removejob - Hadoop job, used to access the appropriate file system.IOExceptionpublic Boolean supportsParallelWriteToStoreLocation()
PigConfiguration.PIG_TEZ_OPT_UNION_SUPPORTED_STOREFUNCS and
PigConfiguration.PIG_TEZ_OPT_UNION_UNSUPPORTED_STOREFUNCS
settings to determine if the StoreFunc supports it.Copyright © 2007-2017 The Apache Software Foundation