有关Lucene的问题(2):stemming和lemmatization

2017-09-14 11:37:50来源:CSDN作者:artaganan8人点击

分享

问题:

我试验了一下文章中提到的 stemming 和 lemmatization

  • 将单词缩减为词根形式,如“cars”到“car”等。这种操作称为:stemming。
  • 将单词转变为词根形式,如“drove”到“drive”等。这种操作称为:lemmatization。

试验没有成功

代码如下:

public class TestNorms {    
    public void createIndex() throws IOException {    
        Directory d = new SimpleFSDirectory(new File("d:/falconTest/lucene3/norms"));    
        IndexWriter writer = new IndexWriter(d, new StandardAnalyzer(Version.LUCENE_30),  
                                                                                      true, IndexWriter.MaxFieldLength.UNLIMITED);    
        Field field = new Field("desc", "", Field.Store.YES, Field.Index.ANALYZED);    
        Document doc = new Document();    
        field.setValue("Hello students was drive");    
        doc.add(field);    
        writer.addDocument(doc);    
        writer.optimize();    
        writer.close();    
    }    
    public void search() throws IOException {    
        Directory d = new SimpleFSDirectory(new File("d:/falconTest/lucene3/norms"));    
        IndexReader reader = IndexReader.open(d);    
        IndexSearcher searcher = new IndexSearcher(reader);    
        TopDocs docs = searcher.search(new TermQuery(new Term("desc","drove")), 10);    
        System.out.println(docs.totalHits);    
    }    
    public static void main(String[] args) throws IOException {    
        TestNorms test= new TestNorms();    
        test.createIndex();    
        test.search();    
    }    

不管是单复数,还是单词的变化,都是没有体现的

不知道是不是分词器的原因?

回答:

的确是分词器的问题,StandardAnalyzer并不能进行stemming和lemmatization,因而不能够区分单复数和词型。

文章中讲述的是全文检索的基本原理,理解了他,有利于更好的理解Lucene,但不代表Lucene是完全按照此基本流程进行的。

(1) 有关stemming

作为stemming,一个著名的算法是The Porter Stemming Algorithm,其主页为http://tartarus.org/~martin/PorterStemmer/,也可查看其论文http://tartarus.org/~martin/PorterStemmer/def.txt

通过以下网页可以进行简单的测试:Porter's Stemming Algorithm Online[http://facweb.cs.depaul.edu/mobasher/classes/csc575/porter.html]

cars –> car

driving –> drive

tokenization –> token

然而

drove –> drove

可见stemming是通过规则缩减为词根的,而不能识别词型的变化。

在最新的Lucene 3.0中,已经有了PorterStemFilter这个类来实现上述算法,只可惜没有Analyzer向匹配,不过不要紧,我们可以简单实现:

public class PorterStemAnalyzer extends Analyzer 

    @Override 
    public TokenStream tokenStream(String fieldName, Reader reader) { 
      return new PorterStemFilter(new LowerCaseTokenizer(reader)); 
    } 
}

把此分词器用在你的程序中,就能够识别单复数和规则的词型变化了。

public void createIndex() throws IOException { 
  Directory d = new SimpleFSDirectory(new File("d:/falconTest/lucene3/norms")); 
  IndexWriter writer = new IndexWriter(d, new PorterStemAnalyzer(), true, IndexWriter.MaxFieldLength.UNLIMITED);

  Field field = new Field("desc", "", Field.Store.YES, Field.Index.ANALYZED); 
  Document doc = new Document(); 
  field.setValue("Hello students was driving cars professionally"); 
  doc.add(field);

  writer.addDocument(doc); 
  writer.optimize(); 
  writer.close(); 
}

public void search() throws IOException { 
  Directory d = new SimpleFSDirectory(new File("d:/falconTest/lucene3/norms")); 
  IndexReader reader = IndexReader.open(d); 
  IndexSearcher searcher = new IndexSearcher(reader); 
  TopDocs docs = searcher.search(new TermQuery(new Term("desc", "car")), 10); 
  System.out.println(docs.totalHits); 
  docs = searcher.search(new TermQuery(new Term("desc", "drive")), 10); 
  System.out.println(docs.totalHits); 
  docs = searcher.search(new TermQuery(new Term("desc", "profession")), 10); 
  System.out.println(docs.totalHits); 
}

(2) 有关lemmatization

至于lemmatization,一般是有字典的,方能够由"drove"对应到"drive".

在网上搜了一下,找到European languages lemmatizer[http://lemmatizer.org/],只不过是在linux下面C++开发的,有兴趣可以试验一下。

首先按照网站的说明下载,编译,安装:

libMAFSA is the core of the lemmatizer. All other libraries depend on it. Download the last version from the following page, unpack it and compile:

# tar xzf libMAFSA-0.2.tar.gz# cd libMAFSA-0.2/# cmake .# make# sudo make install
After this you should install libturglem. You can download it at the same place.
# tar xzf libturglem-0.2.tar.gz# cd libturglem-0.2# cmake .# make# sudo make install
Next you should install english dictionaries with some additional features to work with.
# tar xzf turglem-english-0.2.tar.gz# cd turglem-english-0.2# cmake .# make# sudo make install

安装完毕后:

  • /usr/local/include/turglem是头文件,用于编译自己编写的代码
  • /usr/local/share/turglem/english是字典文件,其中lemmas.xml中我们可以看到"drove"和"drive"的对应,"was"和"be"的对应。
  • /usr/local/lib中的libMAFSA.a  libturglem.a  libturglem-english.a  libtxml.a是用于生成应用程序的静态库

<l id="DRIVE" p="6" />

<l id="DROVE" p="6" />

<l id="DRIVING" p="6" />

在turglem-english-0.2目录下有例子测试程序test_utf8.cpp

#include <stdio.h> 
#include <stdlib.h> 
#include <string.h> 
#include <unistd.h> 
#include <turglem/lemmatizer.h> 
#include <turglem/lemmatizer.hpp> 
#include <turglem/english/charset_adapters.hpp>

int main(int argc, char **argv) 

        char in_s_buf[1024]; 
        char *nl_ptr;

        tl::lemmatizer lem;

        if(argc != 4) 
        { 
                printf("Usage: %s words.dic predict.dic flexias.bin/n", argv[0]); 
                return -1; 
        }

        lem.load_lemmatizer(argv[1], argv[3], argv[2]);

        while (!feof(stdin)) 
        { 
                fgets(in_s_buf, 1024, stdin); 
                nl_ptr = strchr(in_s_buf, '/n'); 
                if (nl_ptr) *nl_ptr = 0; 
                nl_ptr = strchr(in_s_buf, '/r'); 
                if (nl_ptr) *nl_ptr = 0;

                if (in_s_buf[0]) 
                { 
                        printf("processing %s/n", in_s_buf); 
                        tl::lem_result pars; 
                        size_t pcnt = lem.lemmatize<english_utf8_adapter>(in_s_buf, pars); 
                        printf("%d/n", pcnt); 
                        for (size_t i = 0; i < pcnt; i++) 
                        { 
                                std::string s; 
                                u_int32_t src_form = lem.get_src_form(pars, i); 
                                s = lem.get_text<english_utf8_adapter>(pars, i, 0); 
                                printf("PARADIGM %d: normal form '%s'/n", (unsigned int)i, s.c_str()); 
                                printf("/tpart of speech:%d/n", lem.get_part_of_speech(pars, (unsigned int)i, src_form)); 
                        } 
                } 
        }

        return 0; 
}

编译此文件,并且链接静态库:注意链接顺序,否则可能出错。

g++ -g -o output test_utf8.cpp -L/usr/local/lib/ -lturglem-english -lturglem -lMAFSA –ltxml

运行编译好的程序:

./output /usr/local/share/turglem/english/dict_english.auto

/usr/local/share/turglem/english/prediction_english.auto

/usr/local/share/turglem/english/paradigms_english.bin

做测试,虽然对其机制尚不甚了解,但是可以看到lemmatization的作用:

drove 
processing drove 

PARADIGM 0: normal form 'DROVE' 
        part of speech:0 
PARADIGM 1: normal form 'DROVE' 
        part of speech:2 
PARADIGM 2: normal form 'DRIVE' 
        part of speech:2

was 
processing was 

PARADIGM 0: normal form 'BE' 
        part of speech:3 
PARADIGM 1: normal form 'BE' 
        part of speech:3 
PARADIGM 2: normal form 'BE' 
        part of speech:3

最新文章

123

最新摄影

微信扫一扫

第七城市微信公众平台