public class JsonStorage extends StoreFunc implements StoreMetadata, StoreResources
| Modifier and Type | Field and Description |
|---|---|
protected ResourceSchema |
schema |
protected org.apache.hadoop.mapreduce.RecordWriter |
writer |
| Constructor and Description |
|---|
JsonStorage() |
| Modifier and Type | Method and Description |
|---|---|
void |
checkSchema(ResourceSchema s)
Set the schema for data to be stored.
|
ResourceSchema |
fixSchema(ResourceSchema s) |
List<String> |
getCacheFiles()
Allow a StoreFunc to specify a list of files it would like placed in the distributed
cache.
|
org.apache.hadoop.mapreduce.OutputFormat |
getOutputFormat()
Return the OutputFormat associated with StoreFunc.
|
List<String> |
getShipFiles()
Allow a StoreFunc to specify a list of files located locally and would like to ship to backend
(through distributed cache).
|
void |
prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
Initialize StoreFunc to write data.
|
void |
putNext(Tuple t)
Write a tuple to the data store.
|
void |
setStoreFuncUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
StoreFunc which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. |
void |
setStoreLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the storer the location where the data needs to be stored.
|
void |
storeSchema(ResourceSchema schema,
String location,
org.apache.hadoop.mapreduce.Job job)
Store schema of the data being written
|
void |
storeStatistics(ResourceStatistics stats,
String location,
org.apache.hadoop.mapreduce.Job job)
Store statistics about the data being written.
|
cleanupOnFailure, cleanupOnFailureImpl, cleanupOnSuccess, relToAbsPathForStoreLocation, supportsParallelWriteToStoreLocation, warnprotected org.apache.hadoop.mapreduce.RecordWriter writer
protected ResourceSchema schema
public org.apache.hadoop.mapreduce.OutputFormat getOutputFormat()
throws IOException
StoreFuncgetOutputFormat in interface StoreFuncInterfacegetOutputFormat in class StoreFuncOutputFormat associated with StoreFuncIOException - if an exception occurs while constructing the
OutputFormatpublic void setStoreLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFuncStoreFunc here is the
return value of StoreFunc.relToAbsPathForStoreLocation(String, Path)
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.
StoreFunc.checkSchema(ResourceSchema) will be called before any call to
StoreFunc.setStoreLocation(String, Job).setStoreLocation in interface StoreFuncInterfacesetStoreLocation in class StoreFunclocation - Location returned by
StoreFunc.relToAbsPathForStoreLocation(String, Path)job - The Job objectIOException - if the location is not valid.public void setStoreFuncUDFContextSignature(String signature)
StoreFuncStoreFunc which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. This method
will be called before other methods in StoreFunc. This is necessary
because in a Pig Latin script with multiple stores, the different
instances of store functions need to be able to find their (and only their)
data in the UDFContext object. The default implementation is a no-op.setStoreFuncUDFContextSignature in interface StoreFuncInterfacesetStoreFuncUDFContextSignature in class StoreFuncsignature - a unique signature to identify this StoreFuncpublic void checkSchema(ResourceSchema s) throws IOException
StoreFunccheckSchema in interface StoreFuncInterfacecheckSchema in class StoreFuncs - to be checkedIOException - if this schema is not acceptable. It should include
a detailed error message indicating what is wrong with the schema.public void prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
throws IOException
StoreFuncprepareToWrite in interface StoreFuncInterfaceprepareToWrite in class StoreFuncwriter - RecordWriter to use.IOException - if an exception occurs during initializationpublic void putNext(Tuple t) throws IOException
StoreFuncputNext in interface StoreFuncInterfaceputNext in class StoreFunct - the tuple to store.IOException - if an exception occurs during the writepublic void storeStatistics(ResourceStatistics stats, String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreMetadatastoreStatistics in interface StoreMetadatastats - statistics to be recordedlocation - Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOExceptionpublic void storeSchema(ResourceSchema schema, String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreMetadatastoreSchema in interface StoreMetadataschema - Schema to be recordedlocation - Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)job - The Job object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration() and not to set/query
any runtime job information.IOExceptionpublic ResourceSchema fixSchema(ResourceSchema s)
public List<String> getShipFiles()
StoreResourcesFuncUtils for utility function to facilitate it
The default implementation returns null.getShipFiles in interface StoreResourcespublic List<String> getCacheFiles()
StoreResourcesgetCacheFiles in interface StoreResourcesCopyright © 2007-2017 The Apache Software Foundation