Package org.apache.lucene.analysis
Class TokenStreamToAutomaton
java.lang.Object
org.apache.lucene.analysis.TokenStreamToAutomaton
Consumes a TokenStream and creates an
Automaton where the transition labels are UTF8
bytes (or Unicode code points if unicodeArcs is true) from the TermToBytesRefAttribute.
Between tokens we insert POS_SEP and for holes we insert HOLE.- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
Field Summary
Fields -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprotected BytesRefchangeToken(BytesRef in) Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.voidsetFinalOffsetGapAsHole(boolean finalOffsetGapAsHole) If true, any final offset gaps will result in adding a position hole.voidsetPreservePositionIncrements(boolean enablePositionIncrements) Whether to generate holes in the automaton for missing positions,trueby default.voidsetUnicodeArcs(boolean unicodeArcs) Whether to make transition labels Unicode code points instead of UTF8 bytes,falseby defaultPulls the graph (includingPositionLengthAttribute) from the providedTokenStream, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.
-
Field Details
-
POS_SEP
public static final int POS_SEPWe create transition between two adjacent tokens.- See Also:
-
HOLE
public static final int HOLEWe add this arc to represent a hole.- See Also:
-
-
Constructor Details
-
TokenStreamToAutomaton
public TokenStreamToAutomaton()Sole constructor.
-
-
Method Details
-
setPreservePositionIncrements
public void setPreservePositionIncrements(boolean enablePositionIncrements) Whether to generate holes in the automaton for missing positions,trueby default. -
setFinalOffsetGapAsHole
public void setFinalOffsetGapAsHole(boolean finalOffsetGapAsHole) If true, any final offset gaps will result in adding a position hole. -
setUnicodeArcs
public void setUnicodeArcs(boolean unicodeArcs) Whether to make transition labels Unicode code points instead of UTF8 bytes,falseby default -
changeToken
Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph. -
toAutomaton
Pulls the graph (includingPositionLengthAttribute) from the providedTokenStream, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.- Throws:
IOException
-