Сбор лямбда-логов из cloudwatch с помощью fluentd

Может кто-нибудь, пожалуйста, помогите мне в сборе журналов из cloudwatch и отправьте их в asticsearch. Моя конфигурация

<source>
  @type cloudwatch_logs
  region us-east-1      # You must supply a region
  aws_use_sts true
  aws_sts_role_arn arn:aws:iam::xxx:role/fluentd
  log_group_name /aws/lambda/event_processor
  log_stream_name 2019/04/04
  use_log_stream_name_prefix true
  state_file /var/log/td-agent/group_stream.in.state
  tag cloudwatch.logs
</source>

<filter cloudwatch.log>
type record_transformer
</filter>
<match cloudwatch.log>
@type stdout
</match>

Я получаю сообщение об ошибке в журналах

2019-04-10 10:06:54 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::CloudwatchLogsInput title=:in_cloudwatch_logs_runner error_class=Yajl::ParseError **error="lexical error: invalid char in json text.\n                                       START RequestId: 4cf17868-77f0-\n                     (right here) ------^\n"**
2019-04-10 10:06:54 +0000 [error]: #0 unexpected error error_class=Yajl::ParseError error="lexical error: invalid char in json text.\n                                       START RequestId: 4cf17868-77f0-\n                     (right here) ------^\n"
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/yajl-ruby-1.4.0/lib/yajl.rb:44:in `parse'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/yajl-ruby-1.4.0/lib/yajl.rb:44:in `parse'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/yajl-ruby-1.4.0/lib/yajl.rb:18:in `load'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.7.3/lib/fluent/plugin/in_cloudwatch_logs.rb:144:in `emit'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.7.3/lib/fluent/plugin/in_cloudwatch_logs.rb:119:in `block (2 levels) in run'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.7.3/lib/fluent/plugin/in_cloudwatch_logs.rb:118:in `each'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.7.3/lib/fluent/plugin/in_cloudwatch_logs.rb:118:in `block in run'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.7.3/lib/fluent/plugin/in_cloudwatch_logs.rb:115:in `each'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.7.3/lib/fluent/plugin/in_cloudwatch_logs.rb:115:in `run'
  2019-04-10 10:06:54 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-04-10 10:06:54 +0000 [error]: #0 unexpected error error_class=Yajl::ParseError error="lexical error: invalid char in json text.\n                                       START RequestId: 4cf17868-77f0-\n                     (right here) ------^\n"
  2019-04-10 10:06:54 +0000 [error]: #0 suppressed same stacktrace
2019-04-10 10:06:54 +0000 [info]: Worker 0 finished unexpectedly with status 1
2019-04-10 10:06:55 +0000 [warn]: #0 'include' is deprecated. Use '@include' instead
2019-04-10 10:06:55 +0000 [info]: gem 'fluent-plugin-cloudwatch' version '2.1.1'

и вы можете помочь определить динамическое имя потока, если мое имя Stram, как

2019/04/04/[$LATEST]7ed82a142a934cea9209d6315d42b95f
2019/03/29/[$LATEST]1e5b72624b784ec3b1022147e3c736f6

0 ответов

Другие вопросы по тегам