参考

http://rockybean.info/2015/02/09/elasticsearch-immense-term-exception

ElasticSearchimmenseterm错误

作者:rockybean时间:February9,2015分类:技术

在使用ElasticSearch的过程中遇到了一个immenseterm的异常报错,调研了一下出现的原因,又学习到些新东西,见到记录在这里。

这个错误大致内容如下:

java.lang.IllegalArgumentException:Documentcontainsatleastoneimmenseterminfield="reqParams.data"(whoseUTF8encodingislongerthanthemaxlength32766),allofwhichwereskipped.Pleasecorrecttheanalyzertonotproducesuchterms.Theprefixofthefirstimmensetermis:'[123,34,98,114,111,97,100,99,97,115,116,73,100,34,58,49,52,48,56,49,57,57,57,56,56,44,34,116,121,112]...',originalmessage:bytescanbeatmost32766inlength;got40283

atorg.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:685)

atorg.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:359)

atorg.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:318)

atorg.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)

atorg.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:454)

atorg.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1511)

atorg.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1246)

atorg.elasticsearch.index.engine.internal.InternalEngine.innerCreateNoLock(InternalEngine.java:482)

atorg.elasticsearch.index.engine.internal.InternalEngine.innerCreate(InternalEngine.java:435)

atorg.elasticsearch.index.engine.internal.InternalEngine.create(InternalEngine.java:404)

atorg.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:403)

atorg.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:449)

atorg.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:541)

atorg.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:240)

atorg.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:511)

atorg.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)

atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

atjava.lang.Thread.run(Thread.java:745)

Causedby:org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:bytescanbeatmost32766inlength;got40283

atorg.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)

atorg.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:151)

atorg.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:659)

...18more

大意就是说,文档里面有一个巨大的term,超过了lucene处理的最大值(32766),不予处理并抛出异常。错误描述很明确,term太大了,超过了32766字节。网上简单搜索了下,相关文章很多,这里不啰嗦了,讲下查到的一个解决方案。

首先,term是用于搜索的最小单位,一般来讲一个过长的term意义不会太大,有谁会去完整匹配一个100字的关键词呢?!一般都是输入一段关键语句,搜索引擎先将这关键语句分词,获取一系列的term,然后拿这些term去匹配已有文档的倒排索引,打分后返回结果。所以term一般不会很长,像32766这种长度的term即便存下来对于搜索来讲也是毫无意义的,所以当遇到这种超长的term时,如果可以只存储其部分信息,那么就可以解决我们遇到的immenseterm的问题了。好在ElasticSearch已经提供了解决方案,就是ignore_above,这个配置详情可以查看链接,示例配置如下:

curl-XPUT'http://localhost:9200/twitter'-d'

{

"mappings":{

"tweet":{

"properties":{

"message":{"type":"string","index":"not_analyzed","ignore_above":256}

}

}

}

}

'

上面建立了twitter的索引,其中tweet下的message字段不做分词等处理,直接将原始内容来做索引,当内容长度大于256字节时,只索引前面256个字符,后面的内容被丢弃。这样就不会出现前文所提的immenseterm的错误了。

一般ignore_above设置就是为not_analyzed字段存在的,不可滥用。

参考资料

UTF8encodingislongerthanthemaxlength32766

elasticsearch的mapping疑问