ES медленно заполняет кучу с течением времени и имеет размер 14 ГБ, а самый большой индекс составляет 164 МБ?

У меня есть проблема с Elasticsearch, иногда он пытается запустить GC непрерывно, так как этот не может освободить, потому что размер кучи, установленный в 14GB (min и max), считается полностью выделенным:

(...)
[2014-09-18 13:43:45,984][INFO ][monitor.jvm              ] [staging02.onldev] [gc][old][1128185][65590] duration [7.1s], collections
 [1]/[7.2s], total [7.1s]/[9.3h], memory [13.9gb]->[13.9gb]/[13.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [
49.9mb]->[49.6mb]/[66.5mb]}{[old] [13.3gb]->[13.3gb]/[13.3gb]}
[2014-09-18 13:43:53,307][INFO ][monitor.jvm              ] [staging02.onldev] [gc][old][1128186][65591] duration [7.2s], collections
 [1]/[7.3s], total [7.2s]/[9.3h], memory [13.9gb]->[13.9gb]/[13.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [
49.6mb]->[49.7mb]/[66.5mb]}{[old] [13.3gb]->[13.3gb]/[13.3gb]}
[2014-09-18 13:43:58,647][INFO ][monitor.jvm              ] [staging02.onldev] [gc][old][1128187][65592] duration [5.2s], collections
 [1]/[5.3s], total [5.2s]/[9.3h], memory [13.9gb]->[13.9gb]/[13.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [
49.7mb]->[49.8mb]/[66.5mb]}{[old] [13.3gb]->[13.3gb]/[13.3gb]}

На этом этапе ES не отвечает, и мы перезапускаем его

Когда я наблюдаю за кучей ES, и наши работники приложений используют ES, память кучи увеличивается, каждые несколько минут GC запускается и куча почти снова очищается, но не полностью. И медленно, в течение многих дней, кажется, что в куче памяти нет памяти. Похоже, что произошла утечка памяти, кроме как это может быть в нашем коде Ruby с использованием Gem Tire, так как мы говорим о куче ES? Могут ли некоторые модели использования ES вызвать утечку памяти из ES?

По сути, ES - это выделенный сервер с 16 ГБ ОЗУ, без реплик, 5 индексами и 1 осколок на индекс. Он работает с java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64, использует mlockall, а min и max heap установлены в 14 ГБ. На сервере больше ничего не работает. Мы используем Elasticsearch 0.90.x, потому что команда разработчиков не может позволить себе заменить драгоценный камень Tire, который они используют для подключения в рабочих Ruby.

products
size: 164Mi (164Mi)
docs: 98,760 (157,138)

product_brands
size: 4.52Mi (4.52Mi)
docs: 5,123 (5,123)

product_categories
size: 358ki (358ki)
docs: 538 (538)

store_company_categories
size: 389ki (389ki)
docs: 4,028 (4,028)

stores
size: 1.44Mi (1.44Mi)
docs: 1,090 (1,090)

Самый большой индекс productsи отображается как 164MB в Bigdesk. Как ES может использовать до 14GB со временем?

Что-то не так с метаданными индекса?

{
state: open
settings: {
index.analysis.filter.french_stop.stopwords.0: alors
index.analysis.filter.french_stop.stopwords.1: au
index.analysis.filter.french_stop.stopwords.4: autre
index.analysis.filter.french_stop.stopwords.5: avant
index.analysis.filter.french_stop.stopwords.2: aucuns
index.analysis.filter.french_stop.stopwords.3: aussi
index.analysis.filter.french_stop.stopwords.22: dehors
index.analysis.filter.french_stop.stopwords.8: bon
index.analysis.filter.french_stop.stopwords.23: depuis
index.analysis.filter.french_stop.stopwords.9: car
index.analysis.filter.french_stop.stopwords.20: du
index.analysis.filter.french_stop.stopwords.6: avec
index.analysis.filter.french_stop.stopwords.21: dedans
index.analysis.filter.french_stop.stopwords.7: avoir
index.analysis.filter.french_stop.stopwords.29: droite
index.analysis.filter.french_stop.stopwords.28: dos
index.analysis.filter.french_stop.stopwords.27: donc
index.analysis.filter.french_stop.stopwords.26: doit
index.analysis.filter.french_stop.stopwords.25: devrait
index.analysis.filter.french_stop.stopwords.24: deux
index.analysis.analyzer.nGram_analyzer.type: custom
index.analysis.filter.nGram_filter.token_chars.0: letter
index.analysis.analyzer.product_analyzer.type: custom
index.analysis.filter.nGram_filter.token_chars.1: digit
index.analysis.filter.nGram_filter.token_chars.2: punctuation
index.analysis.filter.french_stemmer.type: stemmer
index.analysis.filter.nGram_filter.type: nGram
index.analysis.filter.french_stop.stopwords.10: ce
index.analysis.filter.french_stop.stopwords.11: cela
index.analysis.filter.french_stop.stopwords.12: ces
index.analysis.analyzer.product_analyzer.filter.0: lowercase
index.analysis.filter.french_stop.stopwords.91: sans
index.analysis.filter.french_stop.stopwords.18: dans
index.analysis.analyzer.product_analyzer.filter.1: french_stemmer
index.analysis.filter.french_stop.stopwords.92: ses
index.analysis.filter.french_stop.stopwords.17: comment
index.analysis.analyzer.product_analyzer.filter.2: asciifolding
index.analysis.analyzer.product_analyzer.filter.3: unique
index.analysis.filter.french_stop.stopwords.90: sa
index.analysis.filter.french_stop.stopwords.19: des
index.analysis.filter.french_stop.stopwords.14: chaque
index.analysis.analyzer.product_analyzer.filter.4: french_stop
index.analysis.filter.french_stop.stopwords.13: ceux
index.analysis.filter.nGram_filter.min_gram: 2
index.analysis.filter.french_stop.stopwords.16: comme
index.analysis.analyzer.category_analyzer.type: custom
index.analysis.filter.french_stop.stopwords.15: ci
index.analysis.filter.french_stop.stopwords.99: soyez
index.analysis.filter.french_stop.stopwords.97: sont
index.analysis.filter.french_stop.stopwords.98: sous
index.analysis.filter.french_stop.stopwords.95: sien
index.analysis.filter.french_stop.stopwords.96: son
index.analysis.filter.french_stop.stopwords.93: seulement
index.analysis.filter.french_stop.stopwords.94: si
index.analysis.analyzer.nGram_analyzer.tokenizer: whitespace
index.analysis.filter.french_stop.stopwords.80: plupart
index.analysis.filter.french_stop.stopwords.81: pour
index.number_of_replicas: 0
index.analysis.filter.french_stop.stopwords.82: pourquoi
index.analysis.filter.french_stop.stopwords.83: quand
index.analysis.filter.french_stop.stopwords.84: que
index.analysis.filter.french_stop.stopwords.85: quel
index.analysis.filter.french_stop.stopwords.86: quelle
index.analysis.filter.french_stop.stopwords.87: quelles
index.analysis.filter.french_stop.stopwords.88: quels
index.analysis.filter.french_stop.stopwords.89: qui
index.analysis.analyzer.product_analyzer.tokenizer: standard
index.analysis.filter.french_stop.stopwords.79: pièce
index.analysis.filter.french_stop.stopwords.70: ou
index.analysis.filter.french_stop.stopwords.73: parce
index.analysis.filter.french_stop.stopwords.74: parole
index.uuid: B_JF7UG5R6S_ZC0L0IMFYw
index.analysis.filter.french_stop.stopwords.71: où
index.analysis.filter.french_stop.stopwords.72: par
index.analysis.filter.french_stop.stopwords.77: peut
index.analysis.filter.french_stop.stopwords.78: peu
index.analysis.filter.french_stop.stopwords.75: pas
index.analysis.filter.french_stop.stopwords.76: personnes
index.analysis.filter.french_stop.stopwords.68: nous
index.analysis.filter.french_stop.stopwords.69: nouveaux
index.analysis.filter.french_stop.stopwords.65: ni
index.analysis.analyzer.category_analyzer.filter.0: lowercase
index.analysis.filter.french_stop.stopwords.64: même
index.analysis.filter.french_stop.stopwords.67: notre
index.analysis.filter.french_stop.stopwords.66: nommés
index.analysis.filter.french_stop.stopwords.61: moins
index.analysis.filter.french_stop.stopwords.60: mine
index.analysis.analyzer.category_analyzer.filter.1: french_stemmer
index.analysis.filter.french_stop.stopwords.63: mot
index.analysis.analyzer.category_analyzer.filter.2: french_stop
index.analysis.filter.french_stop.stopwords.62: mon
index.analysis.filter.french_stop.stopwords.120: ça
index.analysis.filter.french_stop.stopwords.121: étaient
index.analysis.filter.french_stop.stopwords.122: état
index.analysis.filter.french_stop.stopwords.123: étions
index.analysis.filter.french_stop.stopwords.124: été
index.analysis.filter.french_stop.stopwords.125: être
index.analysis.filter.nGram_filter.max_gram: 20
index.analysis.filter.french_stop.stopwords.126: rayon
index.analysis.filter.french_stop.stopwords.127: rayons
index.analysis.filter.french_stop.stopwords.128: root
index.number_of_shards: 1
index.analysis.filter.french_stop.stopwords.129: roots
index.analysis.filter.french_stop.stopwords.59: mes
index.analysis.filter.french_stop.stopwords.57: maintenant
index.analysis.filter.french_stop.stopwords.58: mais
index.analysis.filter.french_stop.stopwords.56: ma
index.analysis.filter.french_stop.stopwords.55: là
index.analysis.analyzer.whitespace_analyzer.tokenizer: whitespace
index.analysis.filter.french_stop.stopwords.54: leur
index.analysis.filter.french_stop.stopwords.53: les
index.analysis.filter.french_stop.stopwords.52: le
index.analysis.filter.french_stop.stopwords.51: la
index.analysis.analyzer.whitespace_analyzer.type: custom
index.analysis.filter.french_stop.stopwords.50: juste
index.analysis.analyzer.whitespace_analyzer.filter.1: french_stemmer
index.analysis.analyzer.whitespace_analyzer.filter.0: lowercase
index.analysis.filter.french_stop.type: stop
index.analysis.analyzer.whitespace_analyzer.filter.2: asciifolding
index.analysis.filter.french_stop.stopwords.114: voie
index.analysis.filter.french_stop.stopwords.115: voient
index.analysis.filter.french_stop.stopwords.112: tu
index.analysis.filter.french_stop.stopwords.113: valeur
index.analysis.filter.french_stop.stopwords.110: trop
index.analysis.filter.french_stop.stopwords.111: très
index.version.created: 901399
index.analysis.filter.french_stop.stopwords.46: ici
index.analysis.filter.french_stop.stopwords.47: il
index.analysis.filter.french_stop.stopwords.48: ils
index.analysis.filter.french_stop.stopwords.49: je
index.analysis.filter.french_stop.stopwords.118: vous
index.analysis.filter.french_stop.stopwords.119: vu
index.analysis.filter.french_stop.stopwords.116: vont
index.analysis.filter.french_stop.stopwords.117: votre
index.analysis.filter.french_stop.stopwords.41: fois
index.analysis.filter.nGram_filter.token_chars.3: symbol
index.analysis.filter.french_stop.stopwords.40: faites
index.analysis.analyzer.category_analyzer.tokenizer: standard
index.analysis.filter.french_stop.stopwords.43: force
index.analysis.filter.french_stop.stopwords.42: font
index.analysis.filter.french_stop.stopwords.45: hors
index.analysis.filter.french_stop.stopwords.44: haut
index.analysis.filter.french_stop.stopwords.101: sur
index.analysis.filter.french_stop.stopwords.102: ta
index.analysis.analyzer.nGram_analyzer.filter.3: nGram_filter
index.analysis.filter.french_stop.stopwords.103: tandis
index.analysis.analyzer.nGram_analyzer.filter.2: french_stemmer
index.analysis.filter.french_stop.stopwords.104: tellement
index.analysis.filter.french_stemmer.name: minimal_french
index.analysis.filter.french_stop.stopwords.100: sujet
index.analysis.filter.french_stop.stopwords.37: et
index.analysis.filter.french_stop.stopwords.109: tout
index.analysis.filter.french_stop.stopwords.38: eu
index.analysis.filter.french_stop.stopwords.35: essai
index.analysis.filter.french_stop.stopwords.36: est
index.analysis.analyzer.nGram_analyzer.filter.1: asciifolding
index.analysis.filter.french_stop.stopwords.105: tels
index.analysis.analyzer.nGram_analyzer.filter.0: lowercase
index.analysis.filter.french_stop.stopwords.106: tes
index.analysis.filter.french_stop.stopwords.39: fait
index.analysis.filter.french_stop.stopwords.107: ton
index.analysis.filter.french_stop.stopwords.108: tous
index.analysis.filter.french_stop.stopwords.30: début
index.analysis.filter.french_stop.stopwords.34: encore
index.analysis.filter.french_stop.stopwords.33: en
index.analysis.filter.french_stop.stopwords.32: elles
index.analysis.filter.french_stop.stopwords.31: elle
}

mappings: {
product_category: {
properties: {
tags: {
analyzer: category_analyzer
type: string
}
ancestry_path: {
type: string
}
name: {
analyzer: product_analyzer
type: string
}
leaf?: {
type: boolean
}
category_depth_0: {
properties: {
tags: {
type: string
}
name: {
analyzer: product_analyzer
type: string
}
}
}
name_suggest: {
index_analyzer: nGram_analyzer
search_analyzer: whitespace_analyzer
type: string
}
category_depth_3: {
properties: {
name: {
type: string
}
}
}
self_and_ancestors_ids: {
type: string
}
depth: {
type: integer
}
category_depth_1: {
properties: {
tags: {
type: string
}
name: {
analyzer: product_analyzer
type: string
}
}
}
category_depth_2: {
properties: {
tags: {
type: string
}
name: {
analyzer: product_analyzer
type: string
}
}
}
}
}
}

aliases: [ ]
}

Я пытался использовать минимальный / максимальный размер кучи 6 ГБ, но он демонстрирует то же самое поведение, только перестает отвечать на запросы раньше.