org.apache.mahout.utils.nlp.collocations.llr
Class BloomTokenFilter

java.lang.Object
  extended by org.apache.lucene.util.AttributeSource
      extended by org.apache.lucene.analysis.TokenStream
          extended by org.apache.lucene.analysis.TokenFilter
              extended by org.apache.mahout.utils.nlp.collocations.llr.BloomTokenFilter
All Implemented Interfaces:
Closeable

public final class BloomTokenFilter
extends org.apache.lucene.analysis.TokenFilter

Emits tokens based on bloom filter membership.


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State
 
Field Summary
 
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
 
Constructor Summary
BloomTokenFilter(org.apache.hadoop.util.bloom.Filter filter, boolean keepMembers, org.apache.lucene.analysis.TokenStream in)
           
 
Method Summary
 boolean incrementToken()
           
 
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end, reset
 
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Constructor Detail

BloomTokenFilter

public BloomTokenFilter(org.apache.hadoop.util.bloom.Filter filter,
                        boolean keepMembers,
                        org.apache.lucene.analysis.TokenStream in)
Parameters:
filter - tokens will be checked for membership in this bloom filter
in - the tokenstream to read.
keepMembers - keep memoers of the bloom filter? If true works like a whitelist and members found in the list are kept and all others are dropped. If false works like a stoplist and members found in the filter are dropped all others are kept.
Method Detail

incrementToken

public boolean incrementToken()
                       throws IOException
Specified by:
incrementToken in class org.apache.lucene.analysis.TokenStream
Throws:
IOException


Copyright © 2008–2014 The Apache Software Foundation. All rights reserved.