一步一步跟我学习lucene（3）---lucene的analysis相关和自定义分词器

analysis说明

lucene ananlysis应用场景

lucene提供了analysis用来将文本转换到索引文件或提供给IndexSearcher查询索引；

对于lucene而言，不管是索引还是检索，都是针对于纯文本输入来讲的；

通过lucene的强大类库我们可以访问各种格式的文档，如HTML、XML、PDF、Word、TXT等，

我们需要传递给lucene的只是文件中的纯文本内容；

 lucene的词语切分

lucene的索引和检索前提是其对文本内容的分析和词组的切分；比如，文档中有一句话叫“Hello World,Welcome to China”

我们想找到包含这段话的文档，而用户输入的查询条件又不尽详细（可能只是hello）

这里我们就需要用到lucene索引该文档的时候预先对文档内容进行切分，将词源和文本对应起来。

有时候对词语进行简单切分还远远不够，我们还需要对字符串进行深度切分，lucene不仅能够对索引内容预处理还可以对请求参数进行切分；

使用analyzer

lucene的索引使用如下：

package com.lucene.analysis;

import java.io.IOException;
import java.io.StringReader;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
import org.junit.Test;

public class AnalysisTest {
    @Test
    public void tokenTest() {
        Analyzer analyzer = new StandardAnalyzer(); // or any other analyzer
        TokenStream ts = null;
        try {
            ts = analyzer.tokenStream("myfield", new StringReader(
                    "some text goes here"));
            OffsetAttribute offsetAtt = ts.addAttribute(OffsetAttribute.class);
            ts.reset(); // Resets this stream to the beginning. (Required)
            while (ts.incrementToken()) {
                // Use AttributeSource.reflectAsString(boolean)
                // for token stream debugging.
                System.out.println("token: " + ts.reflectAsString(true));

                System.out.println("token start offset: "
                        + offsetAtt.startOffset());
                System.out.println("token end offset: "
                        + offsetAtt.endOffset());
            }
            ts.end();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } finally {
            try {
                ts.close();
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    }

}

自定义Analyzer和实现自己的analysis模块

1.要实现自己的analyzer，我们需要继承Analyzer并重写其中的分词模块。

2.维护停止词词典

3.重写TokenStreamComponents方法，选择合适的分词方法，对词语进行过滤

示例代码如下

package com.lucene.analysis.self;

import java.io.IOException;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.core.LowerCaseTokenizer;
import org.apache.lucene.analysis.core.StopAnalyzer;
import org.apache.lucene.analysis.core.StopFilter;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.util.CharArraySet;

public class MyAnalyzer extends Analyzer {
    private CharArraySet stopWordSet;//停止词词典

    public CharArraySet getStopWordSet() {
        return stopWordSet;
    }

    public void setStopWordSet(CharArraySet stopWordSet) {
        this.stopWordSet = stopWordSet;
    }


    public MyAnalyzer() {
        super();
        this.stopWordSet = StopAnalyzer.ENGLISH_STOP_WORDS_SET;//可在此基础上拓展停止词
    }

    /**扩展停止词
     * @param stops
     */
    public MyAnalyzer(String[] stops) {
        this();
        stopWordSet.addAll(StopFilter.makeStopSet(stops));
    }

    @Override
    protected TokenStreamComponents createComponents(String fieldName) {
        //正则匹配分词
        Tokenizer source = new LowerCaseTokenizer();
        return new TokenStreamComponents(source, new StopFilter(source, stopWordSet));
    }
    public static void main(String[] args) {
        Analyzer analyzer = new MyAnalyzer();
        String words = "A AN yuyu";
        TokenStream stream = null;

        try {
            stream = analyzer.tokenStream("myfield", words);
            stream.reset();
            CharTermAttribute  offsetAtt = stream.addAttribute(CharTermAttribute.class);
            while (stream.incrementToken()) {
                System.out.println(offsetAtt.toString());
            }
            stream.end();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }finally{
            try {
                stream.close();
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    }
}

运行结果如下：

yuyu

说明该分词器对a an 进行了过滤，这些过滤的词在stopWordSet中

 添加字长过滤器

有时候我们需要对字符串中的短字符进行过滤，比如welcome to BeiJIng中过滤掉长度小于2的字符串，我们期望的结果就变成了Welcome BeiJing,我们仅需要重新实现createComponents方法，相关代码如下:

package com.lucene.analysis.self;

import java.io.IOException;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.core.LowerCaseTokenizer;
import org.apache.lucene.analysis.core.StopAnalyzer;
import org.apache.lucene.analysis.core.StopFilter;
import org.apache.lucene.analysis.core.WhitespaceTokenizer;
import org.apache.lucene.analysis.miscellaneous.LengthFilter;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.util.CharArraySet;

public class LengFilterAanlyzer extends Analyzer {
    private int len;

    public int getLen() {
        return len;
    }


    public void setLen(int len) {
        this.len = len;
    }


    public LengFilterAanlyzer() {
        super();
    }


    public LengFilterAanlyzer(int len) {
        super();
        this.len = len;
    }


    @Override
    protected TokenStreamComponents createComponents(String fieldName) {
        final Tokenizer source = new WhitespaceTokenizer();
        TokenStream result = new LengthFilter(source, len, Integer.MAX_VALUE);
        return new TokenStreamComponents(source,result);

    }
    public static void main(String[] args) {
        Analyzer analyzer = new LengFilterAanlyzer(2);
        String words = "I am a java coder";
        TokenStream stream = null;

        try {
            stream = analyzer.tokenStream("myfield", words);
            stream.reset();
            CharTermAttribute  offsetAtt = stream.addAttribute(CharTermAttribute.class);
            while (stream.incrementToken()) {
                System.out.println(offsetAtt.toString());
            }
            stream.end();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }finally{
            try {
                stream.close();
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    }
}

程序的执行结果如下：

am
java
coder

说明小于2个字符的文本被过滤了。