参考
http://rockybean.info/2015/02/09/elasticsearch-immense-term-exception
ElasticSearchimmenseterm错误
作者:rockybean时间:February9,2015分类:技术
在使用ElasticSearch的过程中遇到了一个immenseterm的异常报错,调研了一下出现的原因,又学习到些新东西,见到记录在这里。
这个错误大致内容如下:
java.lang.IllegalArgumentException:Documentcontainsatleastoneimmenseterminfield="reqParams.data"(whoseUTF8encodingislongerthanthemaxlength32766),allofwhichwereskipped.Pleasecorrecttheanalyzertonotproducesuchterms.Theprefixofthefirstimmensetermis:'[123,34,98,114,111,97,100,99,97,115,116,73,100,34,58,49,52,48,56,49,57,57,57,56,56,44,34,116,121,112]...',originalmessage:bytescanbeatmost32766inlength;got40283
atorg.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:685)
atorg.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:359)
atorg.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:318)
atorg.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
atorg.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:454)
atorg.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1511)
atorg.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1246)
atorg.elasticsearch.index.engine.internal.InternalEngine.innerCreateNoLock(InternalEngine.java:482)
atorg.elasticsearch.index.engine.internal.InternalEngine.innerCreate(InternalEngine.java:435)
atorg.elasticsearch.index.engine.internal.InternalEngine.create(InternalEngine.java:404)
atorg.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:403)
atorg.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:449)
atorg.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:541)
atorg.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:240)
atorg.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:511)
atorg.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
atjava.lang.Thread.run(Thread.java:745)
Causedby:org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException:bytescanbeatmost32766inlength;got40283
atorg.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
atorg.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:151)
atorg.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:659)
...18more
大意就是说,文档里面有一个巨大的term,超过了lucene处理的最大值(32766),不予处理并抛出异常。错误描述很明确,term太大了,超过了32766字节。网上简单搜索了下,相关文章很多,这里不啰嗦了,讲下查到的一个解决方案。
首先,term是用于搜索的最小单位,一般来讲一个过长的term意义不会太大,有谁会去完整匹配一个100字的关键词呢?!一般都是输入一段关键语句,搜索引擎先将这关键语句分词,获取一系列的term,然后拿这些term去匹配已有文档的倒排索引,打分后返回结果。所以term一般不会很长,像32766这种长度的term即便存下来对于搜索来讲也是毫无意义的,所以当遇到这种超长的term时,如果可以只存储其部分信息,那么就可以解决我们遇到的immenseterm的问题了。好在ElasticSearch已经提供了解决方案,就是ignore_above,这个配置详情可以查看链接,示例配置如下:
curl-XPUT'http://localhost:9200/twitter'-d'
{
"mappings":{
"tweet":{
"properties":{
"message":{"type":"string","index":"not_analyzed","ignore_above":256}
}
}
}
}
'
上面建立了twitter的索引,其中tweet下的message字段不做分词等处理,直接将原始内容来做索引,当内容长度大于256字节时,只索引前面256个字符,后面的内容被丢弃。这样就不会出现前文所提的immenseterm的错误了。
一般ignore_above设置就是为not_analyzed字段存在的,不可滥用。
参考资料
UTF8encodingislongerthanthemaxlength32766
elasticsearch的mapping疑问