public class RobotsManager extends BaseTable
| Field | Type | Description |
|---|---|---|
| hostname | VARCHAR(255) | Primary Key |
| robotsdata | BIGINT | |
| expirationtime | BLOB |
| Modifier and Type | Class and Description |
|---|---|
protected static class |
RobotsManager.HostDescription
This is the object description for a robots host object.
|
protected static class |
RobotsManager.HostExecutor
This is the executor object for locating robots host objects.
|
protected static class |
RobotsManager.Record
This class represents a record in a robots.txt file.
|
protected static class |
RobotsManager.RobotsCacheClass
Cache class for robots.
|
protected static class |
RobotsManager.RobotsData
This is a cached data item.
|
| Modifier and Type | Field and Description |
|---|---|
static String |
_rcsid |
protected static String |
expirationField |
protected static String |
hostField |
protected static RobotsManager.RobotsCacheClass |
robotsCacheClass |
protected static String |
robotsField |
dbInterface, tableName| Constructor and Description |
|---|
RobotsManager(IThreadContext tc,
IDBInterface database)
Constructor.
|
| Modifier and Type | Method and Description |
|---|---|
Boolean |
checkFetchAllowed(String userAgent,
String hostName,
long currentTime,
String pathString,
IVersionActivity activities)
Read robots.txt data from the cache or from the database.
|
void |
deinstall()
Uninstall the manager.
|
protected static boolean |
doesPathMatch(String path,
int pathIndex,
String spec,
int specIndex)
Recursive method for matching specification to path.
|
protected static boolean |
doesPathMatch(String path,
String spec)
Check if path matches specification
|
protected static String |
getRobotsKey(String hostName)
Construct a key which represents an individual host name.
|
void |
install()
Install the manager.
|
protected static String |
makeReadable(String inputString)
Convert a string from the robots file into a readable form that does NOT contain NUL characters (since postgresql does not accept those).
|
protected RobotsManager.RobotsData |
readRobotsData(String hostName,
IVersionActivity activities)
Read robots data, if it exists.
|
void |
writeRobotsData(String hostName,
long expirationTime,
InputStream data)
Write robots.txt, replacing any existing row.
|
addTableIndex, analyzeTable, beginTransaction, buildConjunctionClause, constructCountClause, constructDistinctOnClause, constructDoubleCastClause, constructOffsetLimitClause, constructRegexpClause, constructSubstringClause, endTransaction, findConjunctionClauseMax, getDatabaseCacheKey, getDBInterface, getMaxInClause, getMaxOrClause, getSleepAmt, getTableIndexes, getTableName, getTableSchema, getTransactionID, getWindowedReportMaxRows, makeTableKey, noteModifications, performAddIndex, performAlter, performCommit, performCreate, performDelete, performDrop, performInsert, performModification, performQuery, performQuery, performRemoveIndex, performUpdate, prepareRowForSave, readRow, reindexTable, signalRollback, sleepForpublic static final String _rcsid
protected static RobotsManager.RobotsCacheClass robotsCacheClass
protected static final String hostField
protected static final String robotsField
protected static final String expirationField
public RobotsManager(IThreadContext tc, IDBInterface database) throws ManifoldCFException
tc - is the thread context.database - is the database handle.ManifoldCFExceptionpublic void install()
throws ManifoldCFException
ManifoldCFExceptionpublic void deinstall()
throws ManifoldCFException
ManifoldCFExceptionpublic Boolean checkFetchAllowed(String userAgent, String hostName, long currentTime, String pathString, IVersionActivity activities) throws ManifoldCFException
hostName - is the host for which the data is desired.currentTime - is the time of the check.ManifoldCFExceptionpublic void writeRobotsData(String hostName, long expirationTime, InputStream data) throws ManifoldCFException, IOException
hostName - is the host.expirationTime - is the time this data should expire.data - is the robots data stream. May be null.ManifoldCFExceptionIOExceptionprotected static String getRobotsKey(String hostName)
hostName - is the name of the connector.protected RobotsManager.RobotsData readRobotsData(String hostName, IVersionActivity activities) throws ManifoldCFException
ManifoldCFExceptionprotected static String makeReadable(String inputString)
protected static boolean doesPathMatch(String path, String spec)