Uses of Interface
org.apache.nutch.parse.Parse
-
Packages that use Parse Package Description org.apache.nutch.analysis.lang Text document language identifier.org.apache.nutch.crawl Crawl control code and tools to run the crawler.org.apache.nutch.indexer Index content, configure and run indexing and cleaning jobs to add, update, and delete documents from an index.org.apache.nutch.indexer.anchor An indexing plugin for inbound anchor text.org.apache.nutch.indexer.arbitrary Indexing filter to add document arbitrary data to the index from the output of a user-specified class.org.apache.nutch.indexer.basic A basic indexing plugin, adds basic fields: url, host, title, content, etc.org.apache.nutch.indexer.feed Indexing filter to index meta data from RSS feeds.org.apache.nutch.indexer.filter org.apache.nutch.indexer.geoip This plugin implements an indexing filter which takes advantage of the GeoIP2-java API.org.apache.nutch.indexer.jexl This plugin implements a dynamic indexing filter which uses JEXL expressions to allow filtering based on the page's metadataorg.apache.nutch.indexer.links org.apache.nutch.indexer.metadata Indexing filter to add document metadata to the index.org.apache.nutch.indexer.more A more indexing plugin, adds "more" index fields:last modified date, MIME type, content length.org.apache.nutch.indexer.replace Indexing filter to allow pattern replacements on metadata.org.apache.nutch.indexer.staticfield A simple plugin called at indexing that adds fields with static data.org.apache.nutch.indexer.subcollection Indexing filter to assign documents to subcollections.org.apache.nutch.indexer.tld Top Level Domain Indexing plugin.org.apache.nutch.indexer.urlmeta URL Meta Tag Indexing Pluginorg.apache.nutch.microformats.reltag A microformats Rel-Tag Parser/Indexer/Querier plugin.org.apache.nutch.parse TheParseinterface and related classes.org.apache.nutch.scoring TheScoringFilterinterface.org.apache.nutch.scoring.depth Scoring filter to stop crawling at a configurable depth (number of "hops" from seed URLs).org.apache.nutch.scoring.link Scoring filter used in conjunction withWebGraph.org.apache.nutch.scoring.metadata Metadata Scoring Pluginorg.apache.nutch.scoring.opic Scoring filter implementing a variant of the Online Page Importance Computation (OPIC) algorithm.org.apache.nutch.scoring.similarity org.apache.nutch.scoring.similarity.cosine Implements the cosine similarity metric for scoring relevant documentsorg.apache.nutch.scoring.tld Top Level Domain Scoring plugin.org.apache.nutch.scoring.urlmeta URL Meta Tag Scoring Pluginorg.creativecommons.nutch Sample plugins that parse and index Creative Commons metadata. -
-
Uses of Parse in org.apache.nutch.analysis.lang
Methods in org.apache.nutch.analysis.lang with parameters of type Parse Modifier and Type Method Description NutchDocumentLanguageIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.crawl
Methods in org.apache.nutch.crawl with parameters of type Parse Modifier and Type Method Description byte[]MD5Signature. calculate(Content content, Parse parse)abstract byte[]Signature. calculate(Content content, Parse parse)byte[]TextMD5Signature. calculate(Content content, Parse parse)byte[]TextProfileSignature. calculate(Content content, Parse parse) -
Uses of Parse in org.apache.nutch.indexer
Methods in org.apache.nutch.indexer with parameters of type Parse Modifier and Type Method Description NutchDocumentIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)Adds fields or otherwise modifies the document that will be indexed for a parse.NutchDocumentIndexingFilters. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)Run all defined filters. -
Uses of Parse in org.apache.nutch.indexer.anchor
Methods in org.apache.nutch.indexer.anchor with parameters of type Parse Modifier and Type Method Description NutchDocumentAnchorIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)TheAnchorIndexingFilterfilter object which supports boolean configuration settings for the deduplication of anchors. -
Uses of Parse in org.apache.nutch.indexer.arbitrary
Methods in org.apache.nutch.indexer.arbitrary with parameters of type Parse Modifier and Type Method Description NutchDocumentArbitraryIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)TheArbitraryIndexingFilterfilter object uses reflection to instantiate the configured class and invoke the configured method. -
Uses of Parse in org.apache.nutch.indexer.basic
Methods in org.apache.nutch.indexer.basic with parameters of type Parse Modifier and Type Method Description NutchDocumentBasicIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)TheBasicIndexingFilterfilter object which supports few configuration settings for adding basic searchable fields. -
Uses of Parse in org.apache.nutch.indexer.feed
Methods in org.apache.nutch.indexer.feed with parameters of type Parse Modifier and Type Method Description NutchDocumentFeedIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)Extracts out the relevant fields: FEED_AUTHOR FEED_TAGS FEED_PUBLISHED FEED_UPDATED FEED And sends them to theIndexerfor indexing within the Nutch index. -
Uses of Parse in org.apache.nutch.indexer.filter
Methods in org.apache.nutch.indexer.filter with parameters of type Parse Modifier and Type Method Description NutchDocumentMimeTypeIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.geoip
Methods in org.apache.nutch.indexer.geoip with parameters of type Parse Modifier and Type Method Description NutchDocumentGeoIPIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.jexl
Methods in org.apache.nutch.indexer.jexl with parameters of type Parse Modifier and Type Method Description NutchDocumentJexlIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.links
Methods in org.apache.nutch.indexer.links with parameters of type Parse Modifier and Type Method Description NutchDocumentLinksIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.metadata
Methods in org.apache.nutch.indexer.metadata with parameters of type Parse Modifier and Type Method Description NutchDocumentMetadataIndexer. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.more
Methods in org.apache.nutch.indexer.more with parameters of type Parse Modifier and Type Method Description NutchDocumentMoreIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.replace
Methods in org.apache.nutch.indexer.replace with parameters of type Parse Modifier and Type Method Description NutchDocumentReplaceIndexer. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.staticfield
Methods in org.apache.nutch.indexer.staticfield with parameters of type Parse Modifier and Type Method Description NutchDocumentStaticFieldIndexer. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)TheStaticFieldIndexerfilter object which adds fields as per configuration setting. -
Uses of Parse in org.apache.nutch.indexer.subcollection
Methods in org.apache.nutch.indexer.subcollection with parameters of type Parse Modifier and Type Method Description NutchDocumentSubcollectionIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.tld
Methods in org.apache.nutch.indexer.tld with parameters of type Parse Modifier and Type Method Description NutchDocumentTLDIndexingFilter. filter(NutchDocument doc, Parse parse, Text urlText, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.indexer.urlmeta
Methods in org.apache.nutch.indexer.urlmeta with parameters of type Parse Modifier and Type Method Description NutchDocumentURLMetaIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)This will take the metatags that you have listed in your "urlmeta.tags" property, and looks for them inside the CrawlDatum object. -
Uses of Parse in org.apache.nutch.microformats.reltag
Methods in org.apache.nutch.microformats.reltag with parameters of type Parse Modifier and Type Method Description NutchDocumentRelTagIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of Parse in org.apache.nutch.parse
Classes in org.apache.nutch.parse that implement Parse Modifier and Type Class Description classParseImplThe result of parsing a page's raw content.Methods in org.apache.nutch.parse that return Parse Modifier and Type Method Description ParseParseResult. get(String key)Retrieve a single parse output.ParseParseResult. get(Text key)Retrieve a single parse output.ParseParseStatus. getEmptyParse(Configuration conf)Creates an emptyParseinstance containing the statusMethods in org.apache.nutch.parse that return types with arguments of type Parse Modifier and Type Method Description RecordWriter<Text,Parse>ParseOutputFormat. getRecordWriter(TaskAttemptContext context)Iterator<Map.Entry<Text,Parse>>ParseResult. iterator()Iterate over all entries in the <url, Parse> map.Methods in org.apache.nutch.parse with parameters of type Parse Modifier and Type Method Description static ParseResultParseResult. createParseResult(String url, Parse parse)Convenience method for obtainingParseResultfrom a singleParseoutput.Constructors in org.apache.nutch.parse with parameters of type Parse Constructor Description ParseImpl(Parse parse) -
Uses of Parse in org.apache.nutch.scoring
Methods in org.apache.nutch.scoring with parameters of type Parse Modifier and Type Method Description floatAbstractScoringFilter. indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)floatScoringFilter. indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)This method calculates a indexed document score/boost.floatScoringFilters. indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)voidAbstractScoringFilter. passScoreAfterParsing(Text url, Content content, Parse parse)voidScoringFilter. passScoreAfterParsing(Text url, Content content, Parse parse)Currently a part of score distribution is performed using only data coming from the parsing process.voidScoringFilters. passScoreAfterParsing(Text url, Content content, Parse parse) -
Uses of Parse in org.apache.nutch.scoring.depth
Methods in org.apache.nutch.scoring.depth with parameters of type Parse Modifier and Type Method Description floatDepthScoringFilter. indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)voidDepthScoringFilter. passScoreAfterParsing(Text url, Content content, Parse parse) -
Uses of Parse in org.apache.nutch.scoring.link
Methods in org.apache.nutch.scoring.link with parameters of type Parse Modifier and Type Method Description floatLinkAnalysisScoringFilter. indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)voidLinkAnalysisScoringFilter. passScoreAfterParsing(Text url, Content content, Parse parse) -
Uses of Parse in org.apache.nutch.scoring.metadata
Methods in org.apache.nutch.scoring.metadata with parameters of type Parse Modifier and Type Method Description voidMetadataScoringFilter. passScoreAfterParsing(Text url, Content content, Parse parse)Takes the metadata, which was lumped inside the content, and replicates it within your parse data. -
Uses of Parse in org.apache.nutch.scoring.opic
Methods in org.apache.nutch.scoring.opic with parameters of type Parse Modifier and Type Method Description floatOPICScoringFilter. indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)Dampen the boost value by scorePower.voidOPICScoringFilter. passScoreAfterParsing(Text url, Content content, Parse parse)Copy the value from Content metadata under Fetcher.SCORE_KEY to parseData. -
Uses of Parse in org.apache.nutch.scoring.similarity
Methods in org.apache.nutch.scoring.similarity with parameters of type Parse Modifier and Type Method Description voidSimilarityScoringFilter. passScoreAfterParsing(Text url, Content content, Parse parse)floatSimilarityModel. setURLScoreAfterParsing(Text url, Content content, Parse parse) -
Uses of Parse in org.apache.nutch.scoring.similarity.cosine
Methods in org.apache.nutch.scoring.similarity.cosine with parameters of type Parse Modifier and Type Method Description floatCosineSimilarity. setURLScoreAfterParsing(Text url, Content content, Parse parse) -
Uses of Parse in org.apache.nutch.scoring.tld
Methods in org.apache.nutch.scoring.tld with parameters of type Parse Modifier and Type Method Description floatTLDScoringFilter. indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) -
Uses of Parse in org.apache.nutch.scoring.urlmeta
Methods in org.apache.nutch.scoring.urlmeta with parameters of type Parse Modifier and Type Method Description voidURLMetaScoringFilter. passScoreAfterParsing(Text url, Content content, Parse parse)Takes the metadata, which was lumped inside the content, and replicates it within your parse data. -
Uses of Parse in org.creativecommons.nutch
Methods in org.creativecommons.nutch with parameters of type Parse Modifier and Type Method Description NutchDocumentCCIndexingFilter. filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)
-