Error alarm using elastalert

  monitor, python

About elastalert

Elastalert is an elasticsearch-based alert service produced by yelp and written in python. The overall idea is still based on polling method. If rules are applied, frequency, spike, flatline, blacklist/whitelist, any and change are built in. In case of alarm, Email, HipChat, Slack, Telegram, etc. are provided.

dockerfile

# Elastalert Docker image running on ubuntu
# Based off of ivankrizsan/elastalert:latest .
FROM ubuntu:14.04

MAINTAINER Tom Ganem
ENV SET_CONTAINER_TIMEZONE false
ENV ELASTALERT_VERSION 0.0.95
ENV CONTAINER_TIMEZONE Asia/Shanghai
ENV ELASTALERT_URL https://github.com/Yelp/elastalert/archive/v${ELASTALERT_VERSION}.tar.gz
ENV ELASTALERT_DIRECTORY_NAME elastalert
ENV ELASTALERT_HOME /opt/${ELASTALERT_DIRECTORY_NAME}
ENV RULES_DIRECTORY /opt/${ELASTALERT_DIRECTORY_NAME}/rules


WORKDIR /opt

RUN apt-get update && \
    apt-get install tar curl python-dev tzdata -y

RUN curl -Lo get-pip.py https://bootstrap.pypa.io/get-pip.py && \
    python get-pip.py && \
    rm get-pip.py

RUN mkdir -p ${ELASTALERT_HOME}

RUN curl -Lo elastalert.tar.gz ${ELASTALERT_URL} && \
    tar xvf *.tar.gz -C ${ELASTALERT_HOME} --strip-components 1 && \
    rm *.tar.gz

WORKDIR ${ELASTALERT_HOME}

RUN mkdir -p ${RULES_DIRECTORY}
RUN sed -i -e "s|'elasticsearch'|'${ELASTALERT_VERSION_CONSTRAINT}'|g" setup.py
RUN python setup.py install && \
    pip install -e .
RUN pip install elasticsearch

COPY ./docker-entrypoint.sh ${ELASTALERT_HOME}/docker-entrypoint.sh
ENTRYPOINT ["/opt/elastalert/docker-entrypoint.sh"]
CMD ["python", "elastalert/elastalert.py", "--verbose"]

On docker-entrypoint.sh

#!/bin/sh

rules_directory=${RULES_FOLDER:-/opt/elastalert/rules}
es_port=${ELASTICSEARCH_PORT:-9200}

# Render rules files
for file in $(find . -name '*.yaml' -or -name '*.yml');
do
    cat $file | sed "s|es_host: [[:print:]]*|es_host: ${ELASTICSEARCH_HOST}|g" | sed "s|es_port: [[:print:]]*|es_port: $es_port|g" | sed "s|rules_folder: [[:print:]]*|rules_folder: $rules_directory|g" > config
    cat config > $file
    rm config
done

echo "Creating Elastalert index in Elasticsearch..."
elastalert-create-index --index elastalert_status --old-index "" --no-auth;

exec "$@"

It is mainly to replace the relevant variables in config file from the environment variables.

Configuration file

rules_folder: /opt/elastalert/rules
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The elasticsearch hostname for metadata writeback
# Note that every rule can have it's own elasticsearch host
es_host: 192.168.99.101
es_port: 9200

smtp_host: smtp.126.com
smtp_port: 25
smtp_auth_file: /opt/elastalert/smtp_cfg.yaml
from_addr: XXXX@126.com

use_ssl: False

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

rules

# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
# es_host: elasticsearch.example.com

# (Optional)
# Elasticsearch port
# es_port: 14900

# (OptionaL) Connect with SSL to elasticsearch
#use_ssl: True

# (Optional) basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# (Required)
# Rule name, must be unique
name: Example rule

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency

# (Required)
# Index to search, wildcard supported
index: logstash-*

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 50

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
  hours: 4

# (Required)
# A list of elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- query:
    query_string:
      query: "field: value"

# (Required)
# The alert is use when a match is found
alert:
- "email"

# (required, email specific)
# a list of email addresses to send alerts to
email:
- "elastalert@example.com"

Start up

docker run -e "ELASTICSEARCH_HOST=192.168.99.101" -e "ELASTICSEARCH_PORT=9200" -e "RULES_FOLDER=/opt/elastalert/rules" -v $PWD/rules:/opt/elastalert/rules -v $PWD/smtp_cfg.yaml:/opt/elastalert/smtp_cfg.yaml -v $PWD/config.yaml:/opt/elastalert/config.yaml -it esalert /bin/bash

550 error about smtp

Is the user is locked, need to set in Netease mailbox to open smtp, at the same time set the authorization code, and then use the authorization code to replace the password to send mail

docs