How to connect to AWS OpenSearch or Elasticsearch clusters using python

Connecting to an OpenSearch (ES) service running in AWS using Python is painful. Most examples I find online either don’t work or are outdated, leaving me constantly fixing the same issues. To save time and frustration, here’s a collection of working code snippets, up-to-date as of December 2024.

Connect using the opensearch-py library (OpenSearch + ElasticSearch)
Connect using the elasticsearch library (ElasticSearch only)
- elasticsearch >= 8
- elasticsearch < 8

Connect using the opensearch-py library (OpenSearch + ElasticSearch)

This is my preferred way of connecting to an ES instance managed by AWS. It works for both ElasticSearch and OpenSearch clusters, and the authentication can take advantage of AWS profiles.

Install opensearch-py and boto3 (for authentication):


pip <span>install </span>opensearch-py boto3
pip <span>install </span>opensearch-py boto3
pip install opensearch-py boto3

Enter fullscreen mode Exit fullscreen mode

At the time of writing, this installs opensearch-py==2.8.0 and boto3==1.35.81.

Now, you can create a client using the following:


<span>import</span> <span>boto3</span>
<span>from</span> <span>opensearchpy</span> <span>import</span> <span>(</span>
    <span>AWSV4SignerAuth</span><span>,</span>
    <span>OpenSearch</span><span>,</span>
    <span>RequestsHttpConnection</span><span>,</span>
<span>)</span>
<span>es_host</span> <span>=</span> <span>"</span><span>search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com</span><span>"</span>
<span>aws_access_key</span> <span>=</span> <span>"</span><span>AKIAXCUEGTAF3CV7GYKA</span><span>"</span>
<span>aws_secret_key</span> <span>=</span> <span>"</span><span>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</span><span>"</span>
<span>region</span> <span>=</span> <span>"</span><span>eu-central-1</span><span>"</span>
<span># Note: you can also use boto3.Session(profile_name="my-profile") or other ways </span><span>session</span> <span>=</span> <span>boto3</span><span>.</span><span>Session</span><span>(</span>
    <span>aws_access_key_id</span><span>=</span><span>aws_access_key</span><span>,</span>
    <span>aws_secret_access_key</span><span>=</span><span>aws_secret_key</span><span>,</span>
    <span>region_name</span><span>=</span><span>region</span><span>,</span>
<span>)</span>
<span>client</span> <span>=</span> <span>OpenSearch</span><span>(</span>
    <span>hosts</span><span>=</span><span>[{</span><span>"</span><span>host</span><span>"</span><span>:</span> <span>es_host</span><span>,</span> <span>"</span><span>port</span><span>"</span><span>:</span> <span>443</span><span>}],</span>
    <span>http_auth</span><span>=</span><span>AWSV4SignerAuth</span><span>(</span><span>session</span><span>.</span><span>get_credentials</span><span>(),</span> <span>region</span><span>,</span> <span>"</span><span>es</span><span>"</span><span>),</span>
    <span>connection_class</span><span>=</span><span>RequestsHttpConnection</span><span>,</span>
    <span>use_ssl</span><span>=</span><span>True</span><span>,</span>
<span>)</span>
<span>import</span> <span>boto3</span>

<span>from</span> <span>opensearchpy</span> <span>import</span> <span>(</span>
    <span>AWSV4SignerAuth</span><span>,</span>
    <span>OpenSearch</span><span>,</span>
    <span>RequestsHttpConnection</span><span>,</span>
<span>)</span>

<span>es_host</span> <span>=</span> <span>"</span><span>search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com</span><span>"</span>
<span>aws_access_key</span> <span>=</span> <span>"</span><span>AKIAXCUEGTAF3CV7GYKA</span><span>"</span>
<span>aws_secret_key</span> <span>=</span> <span>"</span><span>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</span><span>"</span>
<span>region</span> <span>=</span> <span>"</span><span>eu-central-1</span><span>"</span>

<span># Note: you can also use boto3.Session(profile_name="my-profile") or other ways </span><span>session</span> <span>=</span> <span>boto3</span><span>.</span><span>Session</span><span>(</span>
    <span>aws_access_key_id</span><span>=</span><span>aws_access_key</span><span>,</span>
    <span>aws_secret_access_key</span><span>=</span><span>aws_secret_key</span><span>,</span>
    <span>region_name</span><span>=</span><span>region</span><span>,</span>
<span>)</span>

<span>client</span> <span>=</span> <span>OpenSearch</span><span>(</span>
    <span>hosts</span><span>=</span><span>[{</span><span>"</span><span>host</span><span>"</span><span>:</span> <span>es_host</span><span>,</span> <span>"</span><span>port</span><span>"</span><span>:</span> <span>443</span><span>}],</span>
    <span>http_auth</span><span>=</span><span>AWSV4SignerAuth</span><span>(</span><span>session</span><span>.</span><span>get_credentials</span><span>(),</span> <span>region</span><span>,</span> <span>"</span><span>es</span><span>"</span><span>),</span>
    <span>connection_class</span><span>=</span><span>RequestsHttpConnection</span><span>,</span>
    <span>use_ssl</span><span>=</span><span>True</span><span>,</span>
<span>)</span>
import boto3

from opensearchpy import (
    AWSV4SignerAuth,
    OpenSearch,
    RequestsHttpConnection,
)

es_host = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

# Note: you can also use boto3.Session(profile_name="my-profile") or other ways session = boto3.Session(
    aws_access_key_id=aws_access_key,
    aws_secret_access_key=aws_secret_key,
    region_name=region,
)

client = OpenSearch(
    hosts=[{"host": es_host, "port": 443}],
    http_auth=AWSV4SignerAuth(session.get_credentials(), region, "es"),
    connection_class=RequestsHttpConnection,
    use_ssl=True,
)

Enter fullscreen mode Exit fullscreen mode

Note that boto3.Session supports various ways of creating a session: using a profile, environment variables, and more. I will let you check it out!

Once you have it, check the connection using:


<span>client</span><span>.</span><span>ping</span><span>()</span> <span># should return True </span><span>client</span><span>.</span><span>info</span><span>()</span> <span># use this to get a proper error message if ping fails </span>
<span>client</span><span>.</span><span>ping</span><span>()</span> <span># should return True </span><span>client</span><span>.</span><span>info</span><span>()</span> <span># use this to get a proper error message if ping fails </span>
client.ping() # should return True client.info() # use this to get a proper error message if ping fails

Enter fullscreen mode Exit fullscreen mode

To check indices:


<span># List all indices </span><span>client</span><span>.</span><span>cat</span><span>.</span><span>indices</span><span>()</span>
<span>client</span><span>.</span><span>indices</span><span>.</span><span>get</span><span>(</span><span>"</span><span>*</span><span>"</span><span>)</span>
<span># Check the existence of an indice </span><span>client</span><span>.</span><span>indices</span><span>.</span><span>exists</span><span>(</span><span>"</span><span>my-index</span><span>"</span><span>)</span>
<span># List all indices </span><span>client</span><span>.</span><span>cat</span><span>.</span><span>indices</span><span>()</span>
<span>client</span><span>.</span><span>indices</span><span>.</span><span>get</span><span>(</span><span>"</span><span>*</span><span>"</span><span>)</span>

<span># Check the existence of an indice </span><span>client</span><span>.</span><span>indices</span><span>.</span><span>exists</span><span>(</span><span>"</span><span>my-index</span><span>"</span><span>)</span>
# List all indices client.cat.indices()
client.indices.get("*")

# Check the existence of an indice client.indices.exists("my-index")

Enter fullscreen mode Exit fullscreen mode

Connect using the elasticsearch library (ElasticSearch only)

This only works for ElasticSearch clusters! Connecting to an OpenSearch cluster raises

UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product

elasticsearch >= 8

Most snippets are still referencing RequestsHttpConnection, a class that was removed in elasticsearch 8.X. If you were googling for the error cannot import name 'RequestsHttpConnection' from 'elasticsearch’, you are at the right place!

Install elasticsearch (this should install elastic-transport as well), and requests_aws4auth . The latter, based on requests, is required to handle authentication to AWS:


pip <span>install </span>elasticsearch requests-aws4auth
pip <span>install </span>elasticsearch requests-aws4auth
pip install elasticsearch requests-aws4auth

Enter fullscreen mode Exit fullscreen mode

At the time of writing, this installs elastic-transport==8.15.1, elasticsearch==8.17.0 and requests-aws4auth==1.3.1.

Now, you can create a client using the following:


<span>from</span> <span>elastic_transport</span> <span>import</span> <span>RequestsHttpNode</span>
<span>from</span> <span>elasticsearch</span> <span>import</span> <span>Elasticsearch</span>
<span>from</span> <span>requests_aws4auth</span> <span>import</span> <span>AWS4Auth</span>
<span>es_endpoint</span> <span>=</span> <span>"</span><span>search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com</span><span>"</span>
<span>aws_access_key</span> <span>=</span> <span>"</span><span>AKIAXCUEGTAF3CV7GYKA</span><span>"</span>
<span>aws_secret_key</span> <span>=</span> <span>"</span><span>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</span><span>"</span>
<span>region</span> <span>=</span> <span>"</span><span>eu-central-1</span><span>"</span>
<span>es</span> <span>=</span> <span>Elasticsearch</span><span>(</span>
    <span>f</span><span>"</span><span>https://</span><span>{</span><span>es_host</span><span>}</span><span>"</span><span>,</span>
    <span>http_auth</span><span>=</span><span>AWS4Auth</span><span>(</span>
        <span>aws_access_key</span><span>,</span> 
        <span>aws_secret_key</span><span>,</span> 
        <span>region</span><span>,</span>
        <span>"</span><span>es</span><span>"</span><span>,</span>
    <span>),</span>
    <span>verify_certs</span><span>=</span><span>True</span><span>,</span>
    <span>node_class</span><span>=</span><span>RequestsHttpNode</span><span>,</span>
<span>)</span>
<span>from</span> <span>elastic_transport</span> <span>import</span> <span>RequestsHttpNode</span>
<span>from</span> <span>elasticsearch</span> <span>import</span> <span>Elasticsearch</span>
<span>from</span> <span>requests_aws4auth</span> <span>import</span> <span>AWS4Auth</span>

<span>es_endpoint</span> <span>=</span> <span>"</span><span>search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com</span><span>"</span>
<span>aws_access_key</span> <span>=</span> <span>"</span><span>AKIAXCUEGTAF3CV7GYKA</span><span>"</span>
<span>aws_secret_key</span> <span>=</span> <span>"</span><span>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</span><span>"</span>
<span>region</span> <span>=</span> <span>"</span><span>eu-central-1</span><span>"</span>

<span>es</span> <span>=</span> <span>Elasticsearch</span><span>(</span>
    <span>f</span><span>"</span><span>https://</span><span>{</span><span>es_host</span><span>}</span><span>"</span><span>,</span>
    <span>http_auth</span><span>=</span><span>AWS4Auth</span><span>(</span>
        <span>aws_access_key</span><span>,</span> 
        <span>aws_secret_key</span><span>,</span> 
        <span>region</span><span>,</span>
        <span>"</span><span>es</span><span>"</span><span>,</span>
    <span>),</span>
    <span>verify_certs</span><span>=</span><span>True</span><span>,</span>
    <span>node_class</span><span>=</span><span>RequestsHttpNode</span><span>,</span>
<span>)</span>
from elastic_transport import RequestsHttpNode
from elasticsearch import Elasticsearch
from requests_aws4auth import AWS4Auth

es_endpoint = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

es = Elasticsearch(
    f"https://{es_host}",
    http_auth=AWS4Auth(
        aws_access_key, 
        aws_secret_key, 
        region,
        "es",
    ),
    verify_certs=True,
    node_class=RequestsHttpNode,
)

Enter fullscreen mode Exit fullscreen mode

Once you have it, check the connection using:


<span>es</span><span>.</span><span>ping</span><span>()</span> <span># should return True </span><span>es</span><span>.</span><span>info</span><span>()</span> <span># use this to get a proper error message if ping fails </span>
<span>es</span><span>.</span><span>ping</span><span>()</span> <span># should return True </span><span>es</span><span>.</span><span>info</span><span>()</span> <span># use this to get a proper error message if ping fails </span>
es.ping() # should return True es.info() # use this to get a proper error message if ping fails

Enter fullscreen mode Exit fullscreen mode

elasticsearch < 8

If you are still on an old version of elasticsearch:


<span>pip</span> <span>install</span> <span>"</span><span>elasticsearch<8</span><span>"</span> <span>requests</span><span>-</span><span>aws4auth</span>
<span>pip</span> <span>install</span> <span>"</span><span>elasticsearch<8</span><span>"</span> <span>requests</span><span>-</span><span>aws4auth</span>
pip install "elasticsearch<8" requests-aws4auth

Enter fullscreen mode Exit fullscreen mode

Currently elasticsearch==7.17.12, requests-aws4auth==1.3.1.

Now, you can create a client using the following:


<span>from</span> <span>elasticsearch</span> <span>import</span> <span>Elasticsearch</span><span>,</span> <span>RequestsHttpConnection</span>
<span>from</span> <span>requests_aws4auth</span> <span>import</span> <span>AWS4Auth</span>
<span>es_endpoint</span> <span>=</span> <span>"</span><span>search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com</span><span>"</span>
<span>aws_access_key</span> <span>=</span> <span>"</span><span>AKIAXCUEGTAF3CV7GYKA</span><span>"</span>
<span>aws_secret_key</span> <span>=</span> <span>"</span><span>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</span><span>"</span>
<span>region</span> <span>=</span> <span>"</span><span>eu-central-1</span><span>"</span>
<span>es</span> <span>=</span> <span>Elasticsearch</span><span>(</span>
    <span>host</span><span>=</span><span>es_endpoint</span><span>,</span>
    <span>http_auth</span><span>=</span><span>AWS4Auth</span><span>(</span>
        <span>aws_access_key</span><span>,</span> <span>aws_secret_key</span><span>,</span> <span>region</span><span>,</span> <span>"</span><span>es</span><span>"</span>
    <span>),</span>
    <span>use_ssl</span><span>=</span><span>True</span><span>,</span>
    <span>port</span><span>=</span><span>443</span><span>,</span>
    <span>verify_certs</span><span>=</span><span>True</span><span>,</span>
    <span>connection_class</span><span>=</span><span>RequestsHttpConnection</span><span>,</span>
<span>)</span>
<span>from</span> <span>elasticsearch</span> <span>import</span> <span>Elasticsearch</span><span>,</span> <span>RequestsHttpConnection</span>
<span>from</span> <span>requests_aws4auth</span> <span>import</span> <span>AWS4Auth</span>

<span>es_endpoint</span> <span>=</span> <span>"</span><span>search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com</span><span>"</span>
<span>aws_access_key</span> <span>=</span> <span>"</span><span>AKIAXCUEGTAF3CV7GYKA</span><span>"</span>
<span>aws_secret_key</span> <span>=</span> <span>"</span><span>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</span><span>"</span>
<span>region</span> <span>=</span> <span>"</span><span>eu-central-1</span><span>"</span>

<span>es</span> <span>=</span> <span>Elasticsearch</span><span>(</span>
    <span>host</span><span>=</span><span>es_endpoint</span><span>,</span>
    <span>http_auth</span><span>=</span><span>AWS4Auth</span><span>(</span>
        <span>aws_access_key</span><span>,</span> <span>aws_secret_key</span><span>,</span> <span>region</span><span>,</span> <span>"</span><span>es</span><span>"</span>
    <span>),</span>
    <span>use_ssl</span><span>=</span><span>True</span><span>,</span>
    <span>port</span><span>=</span><span>443</span><span>,</span>
    <span>verify_certs</span><span>=</span><span>True</span><span>,</span>
    <span>connection_class</span><span>=</span><span>RequestsHttpConnection</span><span>,</span>
<span>)</span>
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

es_endpoint = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

es = Elasticsearch(
    host=es_endpoint,
    http_auth=AWS4Auth(
        aws_access_key, aws_secret_key, region, "es"
    ),
    use_ssl=True,
    port=443,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
)

Enter fullscreen mode Exit fullscreen mode

Check the connection:


<span>es</span><span>.</span><span>ping</span><span>()</span> <span># should return True </span><span>es</span><span>.</span><span>info</span><span>()</span> <span># use this to get a proper error message if ping fails </span>
<span>es</span><span>.</span><span>ping</span><span>()</span> <span># should return True </span><span>es</span><span>.</span><span>info</span><span>()</span> <span># use this to get a proper error message if ping fails </span>
es.ping() # should return True es.info() # use this to get a proper error message if ping fails

Enter fullscreen mode Exit fullscreen mode

原文链接：How to connect to AWS OpenSearch or Elasticsearch clusters using python

展开阅读全文

文章版权声明 1、本网站名称：拾光赋
2、本站永久网址：https://www.blogs.ink
3、本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ：805375623进行删除处理。
4、本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5、本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6、本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。

THE END