Add support to test against various versions of OpenSearch (#146)

* Adding support to run integration tests across various versions of OpenSearch

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Adding a compatibility matrix and getting started guide with formatting

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Splitting jobs for opendistro and opensearch

Signed-off-by: Vacha Shah <vachshah@amazon.com>
This commit is contained in:
Vacha Shah
2022-04-05 14:06:54 -07:00
committed by GitHub
parent fe1730a88b
commit 172a3179ab
12 changed files with 257 additions and 202 deletions
+1 -5
View File
@@ -1,4 +1,4 @@
ARG OPENSEARCH_VERSION=1.0.0
ARG OPENSEARCH_VERSION
FROM opensearchproject/opensearch:$OPENSEARCH_VERSION
ARG opensearch_path=/usr/share/opensearch
@@ -6,7 +6,3 @@ ARG opensearch_yml=$opensearch_path/config/opensearch.yml
ARG SECURE_INTEGRATION
RUN if [ "$SECURE_INTEGRATION" != "true" ] ; then echo "plugins.security.disabled: true" >> $opensearch_yml; fi
COPY --chown=opensearch:opensearch .ci/opensearch/opensearch-run.sh $opensearch_path/
CMD /usr/share/opensearch/opensearch-run.sh
-4
View File
@@ -1,4 +0,0 @@
#!/bin/bash
set -ex
/usr/share/opensearch/bin/opensearch -Ediscovery.type=single-node
+10 -14
View File
@@ -3,21 +3,8 @@
# Launch one or more OpenSearch nodes via the Docker image,
# to form a cluster suitable for running the REST API tests.
#
# Export the STACK_VERSION variable, eg. '8.0.0-SNAPSHOT'.
# Export the TEST_SUITE variable, i.e. 'oss'
# Export the NUMBER_OF_NODES variable to start more than 1 node
# Version 1.4.0
# - Initial version of the run-opensearch.sh script
# - Deleting the volume should not dependent on the container still running
# - Fixed `ES_JAVA_OPTS` config
# - Moved to STACK_VERSION and TEST_VERSION
# - Refactored into functions and imports
# - Support NUMBER_OF_NODES
# - Added 5 retries on docker pull for fixing transient network errors
# - Added flags to make local CCR configurations work
# - Added action.destructive_requires_name=false as the default will be true in v8
script_path=$(dirname $(realpath -s $0))
source $script_path/functions/imports.sh
set -euo pipefail
@@ -32,7 +19,7 @@ declare -a volumes
environment=($(cat <<-END
--env node.name=$opensearch_node_name
--env cluster.name=$cluster_name
--env cluster.initial_master_nodes=$master_node_name
--env discovery.type=single-node
--env discovery.seed_hosts=$master_node_name
--env cluster.routing.allocation.disk.threshold_enabled=false
--env bootstrap.memory_lock=true
@@ -43,6 +30,13 @@ environment=($(cat <<-END
END
))
if [[ "$SECURE_INTEGRATION" == "false" ]] && [[ "$CLUSTER" == "opensearch" ]]; then
security=($(cat <<-END
--env plugins.security.disabled=true
END
))
fi
NUMBER_OF_NODES=${NUMBER_OF_NODES-1}
http_port=9200
for (( i=0; i<$NUMBER_OF_NODES; i++, http_port++ )); do
@@ -68,6 +62,7 @@ END
docker build \
--file=.ci/$CLUSTER/Dockerfile \
--build-arg SECURE_INTEGRATION=$SECURE_INTEGRATION \
--build-arg OPENSEARCH_VERSION=$OPENSEARCH_VERSION \
--tag=$CLUSTER-secure-$SECURE_INTEGRATION \
.
@@ -84,6 +79,7 @@ END
--env "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
"${environment[@]}" \
"${volumes[@]}" \
"${security[@]}" \
--publish "$http_port":9200 \
--ulimit nofile=65536:65536 \
--ulimit memlock=-1:-1 \
+2 -3
View File
@@ -1,8 +1,6 @@
#!/usr/bin/env bash
#
# Version 1.1
# - Moved to .ci folder and seperated out `run-repository.sh`
# - Add `$RUNSCRIPTS` env var for running opensearch dependent products
# Entrypoint to run integration tests
# Default environment variables
export TEST_SUITE="${TEST_SUITE:=oss}"
@@ -10,6 +8,7 @@ export PYTHON_VERSION="${PYTHON_VERSION:=3.9}"
export PYTHON_CONNECTION_CLASS="${PYTHON_CONNECTION_CLASS:=Urllib3HttpConnection}"
export CLUSTER="${1:-opensearch}"
export SECURE_INTEGRATION="${2:-false}"
export OPENSEARCH_VERSION="${3:-latest}"
if [[ "$SECURE_INTEGRATION" == "true" ]]; then
export OPENSEARCH_URL_EXTENSION="https"
else
@@ -0,0 +1,18 @@
name: Integration tests with OpenDistro
on: [push, pull_request]
jobs:
integration:
name: Integ
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
secured: ["true", "false"]
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Integ opendistro secured=${{ matrix.secured }}
run: "./.ci/run-tests opendistro ${{ matrix.secured }}"
+4 -3
View File
@@ -9,11 +9,12 @@ jobs:
strategy:
fail-fast: false
matrix:
cluster: ["opendistro", "opensearch"]
cluster: ["opensearch"]
secured: ["true", "false"]
opensearch_version: [1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.3.0, 1.3.1]
steps:
- name: Checkout
uses: actions/checkout@v2
- run: "./.ci/run-tests ${{ matrix.cluster }} ${{ matrix.secured }}"
name: Integ ${{ matrix.stack_version }} ${{ matrix.cluster }} secured=${{ matrix.secured }}
- name: Integ ${{ matrix.cluster }} secured=${{ matrix.secured }} version=${{matrix.opensearch_version}}
run: "./.ci/run-tests ${{ matrix.cluster }} ${{ matrix.secured }} ${{ matrix.opensearch_version }}"
+7
View File
@@ -1,3 +1,10 @@
- [Overview](#overview)
- [Current Admins](#current-admins)
- [Admin Responsibilities](#admin-responsibilities)
- [Prioritize Security](#prioritize-security)
- [Enforce Code of Conduct](#enforce-code-of-conduct)
- [Adopt Organizational Best Practices](#adopt-organizational-best-practices)
## Overview
This document explains who the admins are (see below), what they do in this repo, and how they should be doing it. If you're interested in becoming a maintainer, see [MAINTAINERS](MAINTAINERS.md). If you're interested in contributing, see [CONTRIBUTING](CONTRIBUTING.md).
+18
View File
@@ -0,0 +1,18 @@
- [Compatibility with OpenSearch](#compatibility-with-opensearch)
## Compatibility with OpenSearch
The below matrix shows the compatibility of the [`opensearch-py`](https://pypi.org/project/opensearch-py/) with versions of [`OpenSearch`](https://opensearch.org/downloads.html#opensearch).
| OpenSearch Version | Client Version |
| --- | --- |
| 1.0.0 | 1.0.0 |
| 1.0.1 | 1.0.0 |
| 1.1.0 | 1.1.0 |
| 1.2.0 | 1.1.0 |
| 1.2.1 | 1.0.0 |
| 1.2.2 | 1.0.0 |
| 1.2.3 | 1.0.0 |
| 1.2.4 | 1.0.0 |
| 1.3.0 | 1.1.0 |
| 1.3.1 | 1.1.0 |
+9 -2
View File
@@ -1,3 +1,8 @@
- [Developer Guide](#developer-guide)
- [Linter and Test Suite](#linter-and-test-suite)
- [Integration Tests](#integration-tests)
- [Build the Documentation with Sphinx](#build-the-documentation-with-sphinx)
# Developer Guide
## Linter and Test Suite
@@ -18,10 +23,12 @@ $ nox -rs test
To run the integration tests locally, run:
```
./.ci/run-tests opensearch true
./.ci/run-tests opensearch true 1.3.0
```
The first argument, `opensearch` tells the server type to run integration test against. Valid values are `opensearch` and `opendistro`. The second argument tells whether to run server with security plugin enabled or not.
The first argument, `opensearch` tells the server type to run integration test against. Valid values are `opensearch` and `opendistro`.
The second argument tells whether to run server with security plugin enabled or not.
The third argument specifies the version of OpenSearch the tests should run against, if not specified, the tests run against the latest version. This argument is ignored if the server type is `opendistro`.
Note that integration tests require docker to be installed and running, and downloads quite a bit of data from over the internet and hence take few minutes to complete.
+173
View File
@@ -0,0 +1,173 @@
- [Getting Started with the OpenSearch Python Client](#getting-started-with-the-opensearch-python-client)
- [Setup](#setup)
- [Sample code](#sample-code)
- [Using IAM credentials for authentication](#using-iam-credentials-for-authentication)
- [Pre-requisites to use `AWSV4SignerAuth`](#pre-requisites-to-use-awsv4signerauth)
# Getting Started with the OpenSearch Python Client
## Setup
To add the client to your project, install it using [pip](https://pip.pypa.io/):
```bash
pip install opensearch-py
```
Then import it like any other module:
```python
from opensearchpy import OpenSearch
```
If you prefer to add the client manually or just want to examine the source code, see [opensearch-py on GitHub](https://github.com/opensearch-project/opensearch-py).
## Sample code
```python
from opensearchpy import OpenSearch
host = 'localhost'
port = 9200
auth = ('admin', 'admin') # For testing only. Don't store credentials in code.
ca_certs_path = '/full/path/to/root-ca.pem' # Provide a CA bundle if you use intermediate CAs with your root CA.
# Optional client certificates if you don't want to use HTTP basic authentication.
# client_cert_path = '/full/path/to/client.pem'
# client_key_path = '/full/path/to/client-key.pem'
# Create the client with SSL/TLS enabled, but hostname verification disabled.
client = OpenSearch(
hosts = [{'host': host, 'port': port}],
http_compress = True, # enables gzip compression for request bodies
http_auth = auth,
# client_cert = client_cert_path,
# client_key = client_key_path,
use_ssl = True,
verify_certs = True,
ssl_assert_hostname = False,
ssl_show_warn = False,
ca_certs = ca_certs_path
)
# Create an index with non-default settings.
index_name = 'python-test-index3'
index_body = {
'settings': {
'index': {
'number_of_shards': 4
}
}
}
response = client.indices.create(index_name, body=index_body)
print('\nCreating index:')
print(response)
# Add a document to the index.
document = {
'title': 'Moneyball',
'director': 'Bennett Miller',
'year': '2011'
}
id = '1'
response = client.index(
index = index_name,
body = document,
id = id,
refresh = True
)
print('\nAdding document:')
print(response)
# Search for the document.
q = 'miller'
query = {
'size': 5,
'query': {
'multi_match': {
'query': q,
'fields': ['title^2', 'director']
}
}
}
response = client.search(
body = query,
index = index_name
)
print('\nSearch results:')
print(response)
# Delete the document.
response = client.delete(
index = index_name,
id = id
)
print('\nDeleting document:')
print(response)
# Delete the index.
response = client.indices.delete(
index = index_name
)
print('\nDeleting index:')
print(response)
```
## Using IAM credentials for authentication
Refer the AWS documentation regarding usage of IAM credentials to sign requests to OpenSearch APIs - [Signing HTTP requests to Amazon OpenSearch Service.](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-python)
Opensearch-py client library also provides an in-house IAM based authentication feature, `AWSV4SignerAuth` that will help users to connect to their opensearch clusters by making use of IAM roles.
#### Pre-requisites to use `AWSV4SignerAuth`
- Python version 3.6 or above,
- Install [botocore](https://pypi.org/project/botocore/) using pip
`pip install botocore`
Here is the sample code that uses `AWSV4SignerAuth` -
```python
from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
import boto3
host = '' # cluster endpoint, for example: my-test-domain.us-east-1.es.amazonaws.com
region = 'us-west-2'
credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region)
index_name = 'python-test-index3'
client = OpenSearch(
hosts = [{'host': host, 'port': 443}],
http_auth = auth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
q = 'miller'
query = {
'size': 5,
'query': {
'multi_match': {
'query': q,
'fields': ['title^2', 'director']
}
}
}
response = client.search(
body = query,
index = index_name
)
print('\nSearch results:')
print(response)
```
+9 -9
View File
@@ -1,15 +1,15 @@
- [Overview](#overview)
- [Current Maintainers](#current-maintainers)
- [Maintainer Responsibilities](#maintainer-responsibilities)
- [Uphold Code of Conduct](#uphold-code-of-conduct)
- [Prioritize Security](#prioritize-security)
- [Review Pull Requests](#review-pull-requests)
- [Triage Open Issues](#triage-open-issues)
- [Be Responsive](#be-responsive)
- [Maintain Overall Health of the Repo](#maintain-overall-health-of-the-repo)
- [Use Semver](#use-semver)
- [Release Frequently](#release-frequently)
- [Promote Other Maintainers](#promote-other-maintainers)
- [Uphold Code of Conduct](#uphold-code-of-conduct)
- [Prioritize Security](#prioritize-security)
- [Review Pull Requests](#review-pull-requests)
- [Triage Open Issues](#triage-open-issues)
- [Be Responsive](#be-responsive)
- [Maintain Overall Health of the Repo](#maintain-overall-health-of-the-repo)
- [Use Semver](#use-semver)
- [Release Frequently](#release-frequently)
- [Promote Other Maintainers](#promote-other-maintainers)
## Overview
+6 -162
View File
@@ -10,6 +10,8 @@
OpenSearch Python Client
- [Welcome!](#welcome)
- [Getting Started](#getting-started)
- [Compatibility with OpenSearch](#compatibility-with-opensearch)
- [Project Resources](#project-resources)
- [Code of Conduct](#code-of-conduct)
- [License](#license)
@@ -23,171 +25,13 @@ For more information, see [opensearch.org](https://opensearch.org/).
This is the low-level client. A high-level Python client is in the works, and will be available soon.
## Setup
## Getting Started
To add the client to your project, install it using [pip](https://pip.pypa.io/):
To get started with the OpenSearch Python Client, see [Getting Started](GETTING_STARTED.md).
```bash
pip install opensearch-py
```
## Compatibility with OpenSearch
Then import it like any other module:
```python
from opensearchpy import OpenSearch
```
If you prefer to add the client manually or just want to examine the source code, see [opensearch-py on GitHub](https://github.com/opensearch-project/opensearch-py).
## Sample code
```python
from opensearchpy import OpenSearch
host = 'localhost'
port = 9200
auth = ('admin', 'admin') # For testing only. Don't store credentials in code.
ca_certs_path = '/full/path/to/root-ca.pem' # Provide a CA bundle if you use intermediate CAs with your root CA.
# Optional client certificates if you don't want to use HTTP basic authentication.
# client_cert_path = '/full/path/to/client.pem'
# client_key_path = '/full/path/to/client-key.pem'
# Create the client with SSL/TLS enabled, but hostname verification disabled.
client = OpenSearch(
hosts = [{'host': host, 'port': port}],
http_compress = True, # enables gzip compression for request bodies
http_auth = auth,
# client_cert = client_cert_path,
# client_key = client_key_path,
use_ssl = True,
verify_certs = True,
ssl_assert_hostname = False,
ssl_show_warn = False,
ca_certs = ca_certs_path
)
# Create an index with non-default settings.
index_name = 'python-test-index3'
index_body = {
'settings': {
'index': {
'number_of_shards': 4
}
}
}
response = client.indices.create(index_name, body=index_body)
print('\nCreating index:')
print(response)
# Add a document to the index.
document = {
'title': 'Moneyball',
'director': 'Bennett Miller',
'year': '2011'
}
id = '1'
response = client.index(
index = index_name,
body = document,
id = id,
refresh = True
)
print('\nAdding document:')
print(response)
# Search for the document.
q = 'miller'
query = {
'size': 5,
'query': {
'multi_match': {
'query': q,
'fields': ['title^2', 'director']
}
}
}
response = client.search(
body = query,
index = index_name
)
print('\nSearch results:')
print(response)
# Delete the document.
response = client.delete(
index = index_name,
id = id
)
print('\nDeleting document:')
print(response)
# Delete the index.
response = client.indices.delete(
index = index_name
)
print('\nDeleting index:')
print(response)
```
## Using IAM credentials for authentication
Refer the AWS documentation regarding usage of IAM credentials to sign requests to OpenSearch APIs - [Signing HTTP requests to Amazon OpenSearch Service.](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-python)
Opensearch-py client library also provides an in-house IAM based authentication feature, `AWSV4SignerAuth` that will help users to connect to their opensearch clusters by making use of IAM roles.
#### Pre-requisites to use `AWSV4SignerAuth`
- Python version 3.6 or above,
- Install [botocore](https://pypi.org/project/botocore/) using pip
`pip install botocore`
Here is the sample code that uses `AWSV4SignerAuth` -
```python
from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
import boto3
host = '' # cluster endpoint, for example: my-test-domain.us-east-1.es.amazonaws.com
region = 'us-west-2'
credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region)
index_name = 'python-test-index3'
client = OpenSearch(
hosts = [{'host': host, 'port': 443}],
http_auth = auth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
q = 'miller'
query = {
'size': 5,
'query': {
'multi_match': {
'query': q,
'fields': ['title^2', 'director']
}
}
}
response = client.search(
body = query,
index = index_name
)
print('\nSearch results:')
print(response)
```
See [Compatibility](COMPATIBILITY.md).
## Project Resources