Reenable multiple elasticsearch indices

Registered by Steve McLellan

In https://review.openstack.org/#/c/293685/ (for https://review.openstack.org/#/c/293685/) we were forced to remove the ability to configure separate elasticsearch indices (or aliases) for plugins.

To reiterate the reason, a number of plugins use has_parent queries in RBAC filters. The searchlight query is run across all appropriate indices with a series of `indices` filters, one per resource type, that contain the filters for that type. When a type does not exist at all in an index, has_parent (or has_child) queries will fail, and unfortunately the query parser doesn't take into account that a particular filter should not be run on a particular index.

I'm hoping for a response to https://discuss.elastic.co/t/has-parent-queries-across-multiple-indices-where-types-may-not-exist/44582, but the two approaches i can think of are:
 * Use a multi-search (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html). This is problematic because it means we need to reorder and re-page things
 * Create (empty?) type mappings across all indices (though possibly include type mappings). This actually seems like an OK solution. If there's a way to ensure no documents are inappropriately indexed, that would be excellent.

This may not need a full spec document since it's a straightforward idea.

After some consideration, we will elect to create empty mappings across all indices. This will be done once at startup. After all of the "real" mappings are created we will cycle through all indices. For each index we will cycle through each doc_type. If a mapping does not exist for that doc_type, we will create it. We should add a single bogus field that would never be used and mark the mapping as "static". This should prevent it from every being used.

Here is a quick bash script using curl to show the concept:

# We will place the following doc_types in Index 1:
# OS::Swift::Account, OS::Swift::Container [Child of Account],
# OS::Swift::Object [Child of Container], OS::Nova::Server
# We will place the following doc_types in Index 2:
# OS::Designate::Zone, OS::Designate::RecordSet [Child of Zone]
# OS::Cinder::Volume, OS::Cinder::Snapshot [Child of Volume]

TYPES="OS::Swift::Account OS::Swift::Container OS::Swift::Object OS::Nova::Server OS::Designate::Zone OS::Designate::RecordSet OS::Cinder::Volume OS::Cinder::Snapshot"

INDICES="index1 index2"

HOST=localhost
PORT=9200
IP="http://${HOST}:${PORT}"

# Clean out Elasticsearch
curl -s -XDELETE "${IP}/_all" >/dev/null

# Set up Index1
curl -s -XPUT "${IP}/index1" -d'{"mappings": {"OS::Swift::Container": {"properties": {"name": {"type":"string"}}, "_parent": {"type": "OS::Swift::Account"}}}}' >/dev/null
curl -s -XPUT "${IP}/index1/_mapping/OS::Swift::Object" -d'{ "OS::Swift::Object" : { "properties" : { "name": { "type": "string"}}}, "_parent": {"type": "OS::Swift::Account"}}}}' >/dev/null
curl -s -XPUT "${IP}/index1/_mapping/OS::Swift::Account" -d'{ "OS::Swift::Account" : { "properties" : {"name": {"type":"string"}}}}' >/dev/null
curl -s -XPUT "${IP}/index1/_mapping/OS::Nova::Server" -d'{ "OS::Nova::Server" : { "properties" : {"name": { "type": "string"}}}}' >/dev/null

# Do a query on Index1 based on an Index1 doc_type: Pass
echo "Query Passes"
curl -s -XGET "${IP}/index1/_search" -d'{"query": {"has_parent": {"type": "OS::Swift::Account", "query": {"match_all": {}}}}}'
echo ""

# Set up Index2
curl -s -XPUT "${IP}/index2" -d'{"mappings": {"OS::Designate::RecordSet": {"properties": {"name": {"type":"string"}}, "_parent": {"type": "OS::Designate::Zone"}}}}' >/dev/null
curl -s -XPUT "${IP}/index2/_mapping/OS::Designate::Zone" -d'{"OS::Designate::Zone": {"properties": {"name": { "type": "string"}}}}' >/dev/null
curl -s -XPUT "${IP}/index2/_mapping/OS::Cinder::Snapshot" -d'{"OS::Cinder::Snapshot": {"properties": {"name": { "type": "string"}}}, "_parent": {"type": "OS::Cinder::Volume"}}' >/dev/null
curl -s -XPUT "${IP}/index2/_mapping/OS::Cinder::Volume" -d'{"OS::Cinder::Volume": {"properties": {"name": { "type": "string"}}}}' >/dev/null

# Do a query on Index1 based on an Index1 doc_type: Pass
echo "Query Passes"
curl -s -XGET "${IP}/index1/_search" -d'{"query": {"has_parent": {"type": "OS::Swift::Account", "query": {"match_all": {}}}}}'
echo ""
# Do a query on Index2 based on an Index2 doc_type: Pass
echo "Query Passes"
curl -s -XGET "${IP}/index2/_search" -d'{"query": {"has_parent": {"type": "OS::Cinder::Volume", "query": {"match_all": {}}}}}'
echo ""

# Do a query across both indices based on an index1 doc_type: Fail
echo "Query Fails"
curl -s -XGET "${IP}/index1,index2/_search" -d'{"query": {"has_parent": {"type": "OS::Swift::Account", "query": {"match_all": {}}}}}'
echo ""
# Do a query across both indices based on an index2 doc_type: Fail
echo "Query Fails"
curl -s -XGET "${IP}/index1,index2/_search" -d'{"query": {"has_parent": {"type": "OS::Cinder::Volume", "query": {"match_all": {}}}}}'
echo ""

# Proof Of Concept: Fill out indcies with "missing" doc-types.
for ndx in ${INDICES}
do
    for type in ${TYPES}
    do
        mapping=`curl -s -XGET "${IP}/${ndx}/${type}/_mapping"`
        if [ "${mapping}" = "{}" ]
        then
            curl -s -XPUT "${IP}/${ndx}/_mapping/${type}" -d"{\"${type}\": {\"properties\": {\"name\": { \"type\": \"string\"}}}}" >/dev/null
            echo ""
        fi
    done
done

# Do a query across both indices based on an index1 doc_type: Pass!
echo "Query Passes"
curl -s -XGET "${IP}/index1,index2/_search" -d'{"query": {"has_parent": {"type": "OS::Swift::Account", "query": {"match_all": {}}}}}'
echo ""
# Do a query across both indices based on an index2 doc_type: Pass!
echo "Query Passes"
curl -s -XGET "${IP}/index1,index2/_search" -d'{"query": {"has_parent": {"type": "OS::Cinder::Volume", "query": {"match_all": {}}}}}'
echo ""

Blueprint information

Status:
Complete
Approver:
Travis Tripp
Priority:
High
Drafter:
Steve McLellan
Direction:
Approved
Assignee:
Rick Aulino
Definition:
Approved
Series goal:
Accepted for newton
Implementation:
Implemented
Milestone target:
milestone icon newton-1
Started by
Travis Tripp
Completed by
Travis Tripp

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/s,n,z

Addressed by: https://review.openstack.org/304878
    Reenable multiple elasticsearch indices

Gerrit topic: https://review.openstack.org/#q,topic:bug/1547118,n,z

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.