Es consistency problem

  elastic-search

summarize

Internal factors

The consistency of es mainly has two aspects:

  • Refresh Problem Caused by lucene Index Mechanism

  • Replica consistency problems caused by fragmentation and replication (consistency:one、all、quorum)

external factor

External factors, that is, if the synchronization mechanism of db and es is used, there will be a certain delay in synchronization here, and there may also be inconsistent situations due to abnormal situations, such as transaction rollback.

Refresh after update operation

org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository

@Override
    public <S extends T> S save(S entity) {
        Assert.notNull(entity, "Cannot save 'null' entity.");
        elasticsearchOperations.index(createIndexQuery(entity));
        elasticsearchOperations.refresh(entityInformation.getIndexName());
        return entity;
    }

    public <S extends T> List<S> save(List<S> entities) {
        Assert.notNull(entities, "Cannot insert 'null' as a List.");
        Assert.notEmpty(entities, "Cannot insert empty List.");
        List<IndexQuery> queries = new ArrayList<IndexQuery>();
        for (S s : entities) {
            queries.add(createIndexQuery(s));
        }
        elasticsearchOperations.bulkIndex(queries);
        elasticsearchOperations.refresh(entityInformation.getIndexName());
        return entities;
    }

Refresh to filesystem cache once there are changes, so that it can be searched.

clipboard.png

Copy consistency problem

However, there is another problem. Once there are multiple replication, consistency is involved.

  • If consistency is one, then the writing speed is fast and there is no guarantee to read the latest changes.

  • If it is quorum, it is a relatively compromise version. when writing, W>N/2, that is, the number of nodes w participating in the write operation, must exceed half of the number of replica nodes n. If it is a quorum policy, if reading is to ensure consistency, read quorum must be used to read W of N copies and then arbitrate to obtain the latest data. Or specify to read from primary.
    Related classes

org/elasticsearch/action/WriteConsistencyLevel.java
org/elasticsearch/action/RealtimeRequest.java
  • realtime request
    Es provides a realtime request, which is read from translog and can be guaranteed to be up-to-date.

public class GetRequest extends SingleShardRequest<GetRequest> implements RealtimeRequest {
  //......
}

However, note that get is up-to-date, but other methods such as retrieval are not (If the search is also up-to-date and refresh is required, this will refresh the shard but not the entire index. therefore, if the read request is distributed to the repliac shard, it may not read the latest data, and the preference=_primary needs to be specified at this time.)。

clipboard.png

  • All strategy is a strongly consistent strategy.

Summary

If you want to ensure strong consistency in reading:

  • When write consistency is not all, you need to specify to read from primary shard.

  • When write consistency is all and replication is sync mode (default), no additional specification is required. If replication is async mode, it needs to be read from primary shard.

curl -XGET 192.168.99.100:9200/myindex/_settings

curl -XPUT '192.168.99.100:9200/myindex/_settings' -d '
{
    "index" : {
        "action.write_consistency" : "all"
    }
}'

However, it is necessary to manually refresh during update.

If it is an application that reads more than writes less (especially when there are not many replica), write consistency can be designated as all, which can make good use of replica shard’s reading to improve es’ reading performance.

References