Package org.apache.nutch.crawl
Class CrawlDatum
- java.lang.Object
-
- org.apache.nutch.crawl.CrawlDatum
-
- All Implemented Interfaces:
Cloneable,Comparable<CrawlDatum>,Writable,WritableComparable<CrawlDatum>
public class CrawlDatum extends Object implements WritableComparable<CrawlDatum>, Cloneable
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classCrawlDatum.ComparatorA Comparator optimized for CrawlDatum.
-
Field Summary
Fields Modifier and Type Field Description static StringFETCH_DIR_NAMEstatic StringGENERATE_DIR_NAMEstatic StringPARSE_DIR_NAMEstatic HashMap<Byte,String>statNamesstatic byteSTATUS_DB_DUPLICATEPage was marked as being a duplicate of another pagestatic byteSTATUS_DB_FETCHEDPage was successfully fetched.static byteSTATUS_DB_GONEPage no longer exists.static byteSTATUS_DB_MAXMaximum value of DB-related status.static byteSTATUS_DB_NOTMODIFIEDPage was successfully fetched and found not modified.static byteSTATUS_DB_ORPHANPage was marked as orphan, e.g.static byteSTATUS_DB_REDIR_PERMPage permanently redirects to other page.static byteSTATUS_DB_REDIR_TEMPPage temporarily redirects to other page.static byteSTATUS_DB_UNFETCHEDPage was not fetched yet.static byteSTATUS_FETCH_GONEFetching unsuccessful - page is gone.static byteSTATUS_FETCH_MAXMaximum value of fetch-related status.static byteSTATUS_FETCH_NOTMODIFIEDFetching successful - page is not modified.static byteSTATUS_FETCH_REDIR_PERMFetching permanently redirected to other page.static byteSTATUS_FETCH_REDIR_TEMPFetching temporarily redirected to other page.static byteSTATUS_FETCH_RETRYFetching unsuccessful, needs to be retried (transient errors).static byteSTATUS_FETCH_SUCCESSFetching was successful.static byteSTATUS_INJECTEDPage was newly injected.static byteSTATUS_LINKEDPage discovered through a link.static byteSTATUS_PARSE_METAPage got metadata from a parserstatic byteSTATUS_SIGNATUREPage signature.
-
Constructor Summary
Constructors Constructor Description CrawlDatum()CrawlDatum(int status, int fetchInterval)CrawlDatum(int status, int fetchInterval, float score)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description Objectclone()intcompareTo(CrawlDatum that)Sort twoCrawlDatumobjects by decreasing score.booleanequals(Object o)booleanexecute(org.apache.commons.jexl3.JexlScript expr, String url)intgetFetchInterval()longgetFetchTime()Get the fetch time.MapWritablegetMetaData()Get CrawlDatum metadatalonggetModifiedTime()bytegetRetriesSinceFetch()floatgetScore()byte[]getSignature()bytegetStatus()static bytegetStatusByName(String name)static StringgetStatusName(byte value)static booleanhasDbStatus(CrawlDatum datum)static booleanhasFetchStatus(CrawlDatum datum)inthashCode()voidputAllMetaData(CrawlDatum other)Add all metadata from other CrawlDatum to this CrawlDatum.static CrawlDatumread(DataInput in)voidreadFields(DataInput in)voidset(CrawlDatum that)Copy the contents of another instance into this instance.voidsetFetchInterval(float fetchInterval)voidsetFetchInterval(int fetchInterval)voidsetFetchTime(long fetchTime)Sets either the time of the last fetch or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.voidsetMetaData(MapWritable mapWritable)voidsetModifiedTime(long modifiedTime)voidsetRetriesSinceFetch(int retries)voidsetScore(float score)voidsetSignature(byte[] signature)voidsetStatus(int status)StringtoString()voidwrite(DataOutput out)
-
-
-
Field Detail
-
GENERATE_DIR_NAME
public static final String GENERATE_DIR_NAME
- See Also:
- Constant Field Values
-
FETCH_DIR_NAME
public static final String FETCH_DIR_NAME
- See Also:
- Constant Field Values
-
PARSE_DIR_NAME
public static final String PARSE_DIR_NAME
- See Also:
- Constant Field Values
-
STATUS_DB_UNFETCHED
public static final byte STATUS_DB_UNFETCHED
Page was not fetched yet.- See Also:
- Constant Field Values
-
STATUS_DB_FETCHED
public static final byte STATUS_DB_FETCHED
Page was successfully fetched.- See Also:
- Constant Field Values
-
STATUS_DB_GONE
public static final byte STATUS_DB_GONE
Page no longer exists.- See Also:
- Constant Field Values
-
STATUS_DB_REDIR_TEMP
public static final byte STATUS_DB_REDIR_TEMP
Page temporarily redirects to other page.- See Also:
- Constant Field Values
-
STATUS_DB_REDIR_PERM
public static final byte STATUS_DB_REDIR_PERM
Page permanently redirects to other page.- See Also:
- Constant Field Values
-
STATUS_DB_NOTMODIFIED
public static final byte STATUS_DB_NOTMODIFIED
Page was successfully fetched and found not modified.- See Also:
- Constant Field Values
-
STATUS_DB_DUPLICATE
public static final byte STATUS_DB_DUPLICATE
Page was marked as being a duplicate of another page- See Also:
- Constant Field Values
-
STATUS_DB_ORPHAN
public static final byte STATUS_DB_ORPHAN
Page was marked as orphan, e.g. has no inlinks anymore- See Also:
- Constant Field Values
-
STATUS_DB_MAX
public static final byte STATUS_DB_MAX
Maximum value of DB-related status.- See Also:
- Constant Field Values
-
STATUS_FETCH_SUCCESS
public static final byte STATUS_FETCH_SUCCESS
Fetching was successful.- See Also:
- Constant Field Values
-
STATUS_FETCH_RETRY
public static final byte STATUS_FETCH_RETRY
Fetching unsuccessful, needs to be retried (transient errors).- See Also:
- Constant Field Values
-
STATUS_FETCH_REDIR_TEMP
public static final byte STATUS_FETCH_REDIR_TEMP
Fetching temporarily redirected to other page.- See Also:
- Constant Field Values
-
STATUS_FETCH_REDIR_PERM
public static final byte STATUS_FETCH_REDIR_PERM
Fetching permanently redirected to other page.- See Also:
- Constant Field Values
-
STATUS_FETCH_GONE
public static final byte STATUS_FETCH_GONE
Fetching unsuccessful - page is gone.- See Also:
- Constant Field Values
-
STATUS_FETCH_NOTMODIFIED
public static final byte STATUS_FETCH_NOTMODIFIED
Fetching successful - page is not modified.- See Also:
- Constant Field Values
-
STATUS_FETCH_MAX
public static final byte STATUS_FETCH_MAX
Maximum value of fetch-related status.- See Also:
- Constant Field Values
-
STATUS_SIGNATURE
public static final byte STATUS_SIGNATURE
Page signature.- See Also:
- Constant Field Values
-
STATUS_INJECTED
public static final byte STATUS_INJECTED
Page was newly injected.- See Also:
- Constant Field Values
-
STATUS_LINKED
public static final byte STATUS_LINKED
Page discovered through a link.- See Also:
- Constant Field Values
-
STATUS_PARSE_META
public static final byte STATUS_PARSE_META
Page got metadata from a parser- See Also:
- Constant Field Values
-
-
Method Detail
-
hasDbStatus
public static boolean hasDbStatus(CrawlDatum datum)
-
hasFetchStatus
public static boolean hasFetchStatus(CrawlDatum datum)
-
getStatus
public byte getStatus()
-
getStatusName
public static String getStatusName(byte value)
-
getStatusByName
public static byte getStatusByName(String name)
-
setStatus
public void setStatus(int status)
-
getFetchTime
public long getFetchTime()
Get the fetch time.- Returns:
- long value indicating either the time of the last fetch, or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.
-
setFetchTime
public void setFetchTime(long fetchTime)
Sets either the time of the last fetch or the next fetch time, depending on whether Fetcher or CrawlDbReducer set the time.- Parameters:
fetchTime- the fetch time to set.
-
getModifiedTime
public long getModifiedTime()
-
setModifiedTime
public void setModifiedTime(long modifiedTime)
-
getRetriesSinceFetch
public byte getRetriesSinceFetch()
-
setRetriesSinceFetch
public void setRetriesSinceFetch(int retries)
-
getFetchInterval
public int getFetchInterval()
-
setFetchInterval
public void setFetchInterval(int fetchInterval)
-
setFetchInterval
public void setFetchInterval(float fetchInterval)
-
getScore
public float getScore()
-
setScore
public void setScore(float score)
-
getSignature
public byte[] getSignature()
-
setSignature
public void setSignature(byte[] signature)
-
setMetaData
public void setMetaData(MapWritable mapWritable)
-
putAllMetaData
public void putAllMetaData(CrawlDatum other)
Add all metadata from other CrawlDatum to this CrawlDatum.- Parameters:
other- CrawlDatum
-
getMetaData
public MapWritable getMetaData()
Get CrawlDatum metadata- Returns:
- a MapWritable if it was set or read in #readFields(DataInput), returns empty map in case CrawlDatum was freshly created (lazily instantiated).
- See Also:
readFields(DataInput)
-
read
public static CrawlDatum read(DataInput in) throws IOException
- Throws:
IOException
-
readFields
public void readFields(DataInput in) throws IOException
- Specified by:
readFieldsin interfaceWritable- Throws:
IOException
-
write
public void write(DataOutput out) throws IOException
- Specified by:
writein interfaceWritable- Throws:
IOException
-
set
public void set(CrawlDatum that)
Copy the contents of another instance into this instance.- Parameters:
that- an existingCrawlDatum
-
compareTo
public int compareTo(CrawlDatum that)
Sort twoCrawlDatumobjects by decreasing score.- Specified by:
compareToin interfaceComparable<CrawlDatum>- Parameters:
that- an existingCrawlDatum- Returns:
- 1 if any one field (score, status, fetchTime, retries,
fetchInterval or modifiedTime) of the new
CrawlDatumminus the correspoinding field of the existingCrawlDatumis greater than 0, otherwise return -1.
-
execute
public boolean execute(org.apache.commons.jexl3.JexlScript expr, String url)
-
-