actions/opensearch-pyd
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Update API and license tools

Browse Source
This commit is contained in:
Seth Michael Larson
2020-04-23 14:10:49 -05:00
committed by Seth Michael Larson
parent f0280d9302
commit 24c57840cb
12 changed files with 448 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.ci/test-matrix.yml
+1 -1
View File
@@ -1,5 +1,5 @@
ELASTICSEARCH_VERSION:
- 7.7-SNAPSHOT
- 7.x-SNAPSHOT
TEST_SUITE:
- oss
docs/xpack.rst
+31 -19
View File
@@ -12,7 +12,7 @@ easily enable or disable the features you want to use.
Info
----
`X-Pack info <https://www.elastic.co/guide/en/elasticsearch/reference/current/info-api.html>`_
`X-Pack info <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/info-api.html>`_
provides general info about the installed X-Pack.
.. py:module:: elasticsearch.client.xpack
@@ -24,7 +24,7 @@ provides general info about the installed X-Pack.
Async Search APIs
-----------------
`Async Search API <https://www.elastic.co/guide/en/elasticsearch/reference/current/async-search.html>`_
`Async Search API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/async-search.html>`_
lets you asynchronously execute a search request, monitor its progress,
and retrieve partial results as they become available.
@@ -37,7 +37,7 @@ and retrieve partial results as they become available.
Autoscaling APIs
-----------------
`Autoscaling API <https://www.elastic.co/guide/en/elasticsearch/reference/master/autoscaling-apis.html>`_
`Autoscaling API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/autoscaling-apis.html>`_
gets the current autoscaling decision based on the configured autoscaling policy.
.. py:module:: elasticsearch.client.autoscaling
@@ -49,7 +49,7 @@ gets the current autoscaling decision based on the configured autoscaling policy
EQL APIs
--------
`EQL API <https://www.elastic.co/guide/en/elasticsearch/reference/current/eql.html>`_
`EQL API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/eql.html>`_
allows querying with Event Query Language (EQL) to search logs and events and match
them with shared properties.
@@ -62,7 +62,7 @@ them with shared properties.
Graph Explore APIs
------------------
`Graph Explore API <https://www.elastic.co/guide/en/elasticsearch/reference/current/graph-explore-api.html>`_
`Graph Explore API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/graph-explore-api.html>`_
enables you to extract and summarize information about the documents and terms in your Elasticsearch index.
.. py:module:: elasticsearch.client.graph
@@ -74,7 +74,7 @@ enables you to extract and summarize information about the documents and terms i
Licensing APIs
--------------
`Licensing API <https://www.elastic.co/guide/en/elasticsearch/reference/current/licensing-apis.html>`_
`Licensing API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/licensing-apis.html>`_
can be used to manage your licences.
@@ -87,7 +87,7 @@ can be used to manage your licences.
Machine Learning APIs
---------------------
`Machine Learning <https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-apis.html>`_
`Machine Learning <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/ml-apis.html>`_
can be useful for discovering new patterns about your data. For a more detailed explanation
about X-Pack's machine learning please refer to the official documentation.
@@ -101,7 +101,7 @@ about X-Pack's machine learning please refer to the official documentation.
Security APIs
-------------
`Security API <https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api.html>`_
`Security API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/security-api.html>`_
can be used to help secure your Elasticsearch cluster. Integrating with LDAP and Active Directory.
.. py:module:: elasticsearch.client.security
@@ -113,7 +113,7 @@ can be used to help secure your Elasticsearch cluster. Integrating with LDAP and
Watcher APIs
-------------
`Watcher API <https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api.html>`_
`Watcher API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/watcher-api.html>`_
can be used to notify you when certain pre-defined thresholds have happened.
.. py:module:: elasticsearch.client.watcher
@@ -125,7 +125,7 @@ can be used to notify you when certain pre-defined thresholds have happened.
Migration APIs
---------------
`Migration API <https://www.elastic.co/guide/en/elasticsearch/reference/current/migration-api.html>`_
`Migration API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/migration-api.html>`_
helps simplify upgrading X-Pack indices from one version to another.
.. py:module:: elasticsearch.client.migration
@@ -137,7 +137,7 @@ helps simplify upgrading X-Pack indices from one version to another.
Enrich APIs
------------
`Enrich API <https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-apis.html>`_
`Enrich API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/enrich-apis.html>`_
can be used to add data from your existing indices to incoming documents during ingest.
.. py:module:: elasticsearch.client.enrich
@@ -148,7 +148,7 @@ can be used to add data from your existing indices to incoming documents during
SQL APIs
--------
The `SQL REST API <https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-rest.html>`_
The `SQL REST API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/sql-rest.html>`_
accepts SQL in a JSON document, executes it, and returns the results.
.. py:module:: elasticsearch.client.sql
@@ -159,7 +159,7 @@ accepts SQL in a JSON document, executes it, and returns the results.
Cross-Cluster Replication APIs
-------------------------------
`Cross-Cluster Replication API <https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-apis.html>`_
`Cross-Cluster Replication API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/ccr-apis.html>`_
used to perform cross-cluster replication operations.
.. py:module:: elasticsearch.client.ccr
@@ -171,7 +171,7 @@ used to perform cross-cluster replication operations.
Monitoring APIs
----------------
`Monitoring API <https://www.elastic.co/guide/en/elasticsearch/reference/master/es-monitoring.html>`_
`Monitoring API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/es-monitoring.html>`_
used to collect data from the Elasticsearch nodes, Logstash nodes, Kibana instances, and Beats in your cluster.
.. py:module:: elasticsearch.client.monitoring
@@ -183,7 +183,7 @@ used to collect data from the Elasticsearch nodes, Logstash nodes, Kibana instan
Rollup APIs
------------
`Rollup API <https://www.elastic.co/guide/en/elasticsearch/reference/current/rollup-apis.html>`_
`Rollup API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/rollup-apis.html>`_
enables searching through rolled-up data using the standard query DSL.
.. py:module:: elasticsearch.client.rollup
@@ -195,7 +195,7 @@ enables searching through rolled-up data using the standard query DSL.
Snapshot Lifecycle Management APIs
-----------------------------------
`Snapshot Lifecycle Management API <https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-lifecycle-management-api.html>`_
`Snapshot Lifecycle Management API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/snapshot-lifecycle-management-api.html>`_
can be used to set up policies to automatically take snapshots and control how long they are retained.
.. py:module:: elasticsearch.client.slm
@@ -204,10 +204,22 @@ can be used to set up policies to automatically take snapshots and control how l
:members:
Searchable Snapshots APIs
-------------------------
`Searchable Snapshots API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/searchable-snapshots-apis.html>`_
used to perform searchable snapshots operations.
.. py:module:: elasticsearch.client.searchable_snapshots
.. autoclass:: SearchableSnapshotsClient
:members:
Index Lifecycle Management APIs
--------------------------------
`Index Lifecycle Management API <https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management-api.html>`_
`Index Lifecycle Management API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/index-lifecycle-management-api.html>`_
used to set up policies to automatically manage the index lifecycle.
.. py:module:: elasticsearch.client.ilm
@@ -219,7 +231,7 @@ used to set up policies to automatically manage the index lifecycle.
Transform APIs
---------------
`Transform API <https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-apis.html>`_
`Transform API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/transform-apis.html>`_
manages transformation operations from grabbing data from source indices, transforms it, and
saves it to a destination index.
@@ -232,7 +244,7 @@ saves it to a destination index.
Deprecation APIs
-----------------
`Deprecation API <https://www.elastic.co/guide/en/elasticsearch/reference/master/migration-api-deprecation.html>`_
`Deprecation API <https://www.elastic.co/guide/en/elasticsearch/reference/7.x/migration-api-deprecation.html>`_
used to retrieve information about different cluster, node, and index level settings that use deprecated features
that will be removed or changed in the next major version.
elasticsearch/client/__init__.py
+14
View File
@@ -35,6 +35,7 @@ from .sql import SqlClient
from .ssl import SslClient
from .watcher import WatcherClient
from .enrich import EnrichClient
from .searchable_snapshots import SearchableSnapshotsClient
from .slm import SlmClient
from .transform import TransformClient
@@ -259,6 +260,7 @@ class Elasticsearch(object):
self.ssl = SslClient(self)
self.watcher = WatcherClient(self)
self.enrich = EnrichClient(self)
self.searchable_snapshots = SearchableSnapshotsClient(self)
self.slm = SlmClient(self)
self.transform = TransformClient(self)
@@ -300,6 +302,7 @@ class Elasticsearch(object):
@query_params(
"pipeline",
"prefer_v2_templates",
"refresh",
"routing",
"timeout",
@@ -319,6 +322,8 @@ class Elasticsearch(object):
:arg doc_type: The type of the document
:arg pipeline: The pipeline id to preprocess incoming documents
with
:arg prefer_v2_templates: favor V2 templates instead of V1
templates during automatic index creation
:arg refresh: If `true` then refresh the affected shards to make
this operation visible to search, if `wait_for` then wait for a refresh
to make this operation visible to search, if `false` (the default) then
@@ -354,6 +359,7 @@ class Elasticsearch(object):
"if_seq_no",
"op_type",
"pipeline",
"prefer_v2_templates",
"refresh",
"routing",
"timeout",
@@ -381,6 +387,8 @@ class Elasticsearch(object):
without an explicit document ID Valid choices: index, create
:arg pipeline: The pipeline id to preprocess incoming documents
with
:arg prefer_v2_templates: favor V2 templates instead of V1
templates during automatic index creation
:arg refresh: If `true` then refresh the affected shards to make
this operation visible to search, if `wait_for` then wait for a refresh
to make this operation visible to search, if `false` (the default) then
@@ -416,6 +424,7 @@ class Elasticsearch(object):
"_source_excludes",
"_source_includes",
"pipeline",
"prefer_v2_templates",
"refresh",
"routing",
"timeout",
@@ -440,6 +449,8 @@ class Elasticsearch(object):
return from the _source field, can be overridden on each sub-request
:arg pipeline: The pipeline id to preprocess incoming documents
with
:arg prefer_v2_templates: favor V2 templates instead of V1
templates during automatic index creation
:arg refresh: If `true` then refresh the affected shards to make
this operation visible to search, if `wait_for` then wait for a refresh
to make this operation visible to search, if `false` (the default) then
@@ -1797,6 +1808,7 @@ class Elasticsearch(object):
"if_primary_term",
"if_seq_no",
"lang",
"prefer_v2_templates",
"refresh",
"retry_on_conflict",
"routing",
@@ -1826,6 +1838,8 @@ class Elasticsearch(object):
operation that has changed the document has the specified sequence
number
:arg lang: The script language (default: painless)
:arg prefer_v2_templates: favor V2 templates instead of V1
templates during automatic index creation
:arg refresh: If `true` then refresh the affected shards to make
this operation visible to search, if `wait_for` then wait for a refresh
to make this operation visible to search, if `false` (the default) then
elasticsearch/client/autoscaling.py
+55 -1
View File
@@ -1,4 +1,4 @@
from .utils import NamespacedClient, query_params
from .utils import NamespacedClient, query_params, SKIP_IN_PATH, _make_path
class AutoscalingClient(NamespacedClient):
@@ -12,3 +12,57 @@ class AutoscalingClient(NamespacedClient):
return self.transport.perform_request(
"GET", "/_autoscaling/decision", params=params, headers=headers
)
@query_params()
def delete_autoscaling_policy(self, name, params=None, headers=None):
"""
`<https://www.elastic.co/guide/en/elasticsearch/reference/current/autoscaling-delete-autoscaling-policy.html>`_
:arg name: the name of the autoscaling policy
"""
if name in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument 'name'.")
return self.transport.perform_request(
"DELETE",
_make_path("_autoscaling", "policy", name),
params=params,
headers=headers,
)
@query_params()
def get_autoscaling_policy(self, name, params=None, headers=None):
"""
`<https://www.elastic.co/guide/en/elasticsearch/reference/current/autoscaling-get-autoscaling-policy.html>`_
:arg name: the name of the autoscaling policy
"""
if name in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument 'name'.")
return self.transport.perform_request(
"GET",
_make_path("_autoscaling", "policy", name),
params=params,
headers=headers,
)
@query_params()
def put_autoscaling_policy(self, name, body, params=None, headers=None):
"""
`<https://www.elastic.co/guide/en/elasticsearch/reference/current/autoscaling-put-autoscaling-policy.html>`_
:arg name: the name of the autoscaling policy
:arg body: the specification of the autoscaling policy
"""
for param in (name, body):
if param in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument.")
return self.transport.perform_request(
"PUT",
_make_path("_autoscaling", "policy", name),
params=params,
headers=headers,
body=body,
)
elasticsearch/client/cluster.py
+22
View File
@@ -298,3 +298,25 @@ class ClusterClient(NamespacedClient):
headers=headers,
body=body,
)
@query_params("local", "master_timeout")
def exists_component_template(self, name, params=None, headers=None):
"""
Returns information about whether a particular component template exist
`<https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-component-templates.html>`_
:arg name: The name of the template
:arg local: Return local information, do not retrieve the state
from master node (default: false)
:arg master_timeout: Explicit operation timeout for connection
to master node
"""
if name in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument 'name'.")
return self.transport.perform_request(
"HEAD",
_make_path("_component_template", name),
params=params,
headers=headers,
)
elasticsearch/client/indices.py
+98 -2
View File
@@ -79,7 +79,11 @@ class IndicesClient(NamespacedClient):
)
@query_params(
"include_type_name", "master_timeout", "timeout", "wait_for_active_shards"
"include_type_name",
"master_timeout",
"prefer_v2_templates",
"timeout",
"wait_for_active_shards",
)
def create(self, index, body=None, params=None, headers=None):
"""
@@ -92,6 +96,8 @@ class IndicesClient(NamespacedClient):
:arg include_type_name: Whether a type should be expected in the
body of the mappings.
:arg master_timeout: Specify timeout for connection to master
:arg prefer_v2_templates: favor V2 templates instead of V1
templates during index creation
:arg timeout: Explicit operation timeout
:arg wait_for_active_shards: Set the number of active shards to
wait for before the operation returns.
@@ -1135,6 +1141,7 @@ class IndicesClient(NamespacedClient):
"dry_run",
"include_type_name",
"master_timeout",
"prefer_v2_templates",
"timeout",
"wait_for_active_shards",
)
@@ -1154,6 +1161,8 @@ class IndicesClient(NamespacedClient):
:arg include_type_name: Whether a type should be included in the
body of the mappings.
:arg master_timeout: Specify timeout for connection to master
:arg prefer_v2_templates: favor V2 templates instead of V1
templates during automatic index creation
:arg timeout: Explicit operation timeout
:arg wait_for_active_shards: Set the number of active shards to
wait for on the newly created rollover index before the operation
@@ -1309,8 +1318,95 @@ class IndicesClient(NamespacedClient):
Returns data streams.
`<https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html>`_
:arg name: The comma separated names of data streams
:arg name: The name or wildcard expression of the requested data
streams
"""
return self.transport.perform_request(
"GET", _make_path("_data_streams", name), params=params, headers=headers
)
@query_params("master_timeout", "timeout")
def delete_index_template(self, name, params=None, headers=None):
"""
Deletes an index template.
`<https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html>`_
:arg name: The name of the template
:arg master_timeout: Specify timeout for connection to master
:arg timeout: Explicit operation timeout
"""
if name in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument 'name'.")
return self.transport.perform_request(
"DELETE",
_make_path("_index_template", name),
params=params,
headers=headers,
)
@query_params("flat_settings", "local", "master_timeout")
def exists_index_template(self, name, params=None, headers=None):
"""
Returns information about whether a particular index template exists.
`<https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html>`_
:arg name: The name of the template
:arg flat_settings: Return settings in flat format (default:
false)
:arg local: Return local information, do not retrieve the state
from master node (default: false)
:arg master_timeout: Explicit operation timeout for connection
to master node
"""
if name in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument 'name'.")
return self.transport.perform_request(
"HEAD", _make_path("_index_template", name), params=params, headers=headers
)
@query_params("flat_settings", "local", "master_timeout")
def get_index_template(self, name=None, params=None, headers=None):
"""
Returns an index template.
`<https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html>`_
:arg name: The comma separated names of the index templates
:arg flat_settings: Return settings in flat format (default:
false)
:arg local: Return local information, do not retrieve the state
from master node (default: false)
:arg master_timeout: Explicit operation timeout for connection
to master node
"""
return self.transport.perform_request(
"GET", _make_path("_index_template", name), params=params, headers=headers
)
@query_params("create", "master_timeout", "order")
def put_index_template(self, name, body, params=None, headers=None):
"""
Creates or updates an index template.
`<https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html>`_
:arg name: The name of the template
:arg body: The template definition
:arg create: Whether the index template should only be added if
new or can also replace an existing one
:arg master_timeout: Specify timeout for connection to master
:arg order: The order for this template when merging multiple
matching ones (higher numbers are merged later, overriding the lower
numbers)
"""
for param in (name, body):
if param in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument.")
return self.transport.perform_request(
"PUT",
_make_path("_index_template", name),
params=params,
headers=headers,
body=body,
)
elasticsearch/client/nodes.py
+6 -1
View File
@@ -3,11 +3,15 @@ from .utils import NamespacedClient, query_params, _make_path
class NodesClient(NamespacedClient):
@query_params("timeout")
def reload_secure_settings(self, node_id=None, params=None, headers=None):
def reload_secure_settings(
self, body=None, node_id=None, params=None, headers=None
):
"""
Reloads secure settings.
`<https://www.elastic.co/guide/en/elasticsearch/reference/master/secure-settings.html#reloadable-secure-settings>`_
:arg body: An object containing the password for the
elasticsearch keystore
:arg node_id: A comma-separated list of node IDs to span the
reload/reinit call. Should stay empty because reloading usually involves
all cluster nodes.
@@ -18,6 +22,7 @@ class NodesClient(NamespacedClient):
_make_path("_nodes", node_id, "reload_secure_settings"),
params=params,
headers=headers,
body=body,
)
@query_params("flat_settings", "timeout")
elasticsearch/client/searchable_snapshots.py
+84
View File
@@ -0,0 +1,84 @@
from .utils import NamespacedClient, query_params, _make_path, SKIP_IN_PATH
class SearchableSnapshotsClient(NamespacedClient):
@query_params("allow_no_indices", "expand_wildcards", "ignore_unavailable")
def clear_cache(self, index=None, params=None, headers=None):
"""
`<https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots-api-clear-cache.html>`_
:arg index: A comma-separated list of index name to limit the
operation
:arg allow_no_indices: Whether to ignore if a wildcard indices
expression resolves into no concrete indices. (This includes `_all`
string or when no indices have been specified)
:arg expand_wildcards: Whether to expand wildcard expression to
concrete indices that are open, closed or both. Valid choices: open,
closed, none, all Default: open
:arg ignore_unavailable: Whether specified concrete indices
should be ignored when unavailable (missing or closed)
"""
return self.transport.perform_request(
"POST",
_make_path(index, "_searchable_snapshots", "cache", "clear"),
params=params,
headers=headers,
)
@query_params("master_timeout", "wait_for_completion")
def mount(self, repository, snapshot, body, params=None, headers=None):
"""
`<https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots-api-mount-snapshot>`_
:arg repository: The name of the repository containing the
snapshot of the index to mount
:arg snapshot: The name of the snapshot of the index to mount
:arg body: The restore configuration for mounting the snapshot
as searchable
:arg master_timeout: Explicit operation timeout for connection
to master node
:arg wait_for_completion: Should this request wait until the
operation has completed before returning
"""
for param in (repository, snapshot, body):
if param in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument.")
return self.transport.perform_request(
"POST",
_make_path("_snapshot", repository, snapshot, "_mount"),
params=params,
headers=headers,
body=body,
)
@query_params()
def repository_stats(self, repository, params=None, headers=None):
"""
`<https://www.elastic.co/guide/en/elasticsearch/reference/master/searchable-snapshots-repository-stats.html>`_
:arg repository: The repository for which to get the stats for
"""
if repository in SKIP_IN_PATH:
raise ValueError("Empty value passed for a required argument 'repository'.")
return self.transport.perform_request(
"GET",
_make_path("_snapshot", repository, "_stats"),
params=params,
headers=headers,
)
@query_params()
def stats(self, index=None, params=None, headers=None):
"""
`<https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots-api-stats.html>`_
:arg index: A comma-separated list of index names
"""
return self.transport.perform_request(
"GET",
_make_path(index, "_searchable_snapshots", "stats"),
params=params,
headers=headers,
)
elasticsearch/client/tasks.py
+4 -1
View File
@@ -35,7 +35,7 @@ class TasksClient(NamespacedClient):
"GET", "/_tasks", params=params, headers=headers
)
@query_params("actions", "nodes", "parent_task_id")
@query_params("actions", "nodes", "parent_task_id", "wait_for_completion")
def cancel(self, task_id=None, params=None, headers=None):
"""
Cancels a task, if it can be cancelled through an API.
@@ -50,6 +50,9 @@ class TasksClient(NamespacedClient):
node you're connecting to, leave empty to get information from all nodes
:arg parent_task_id: Cancel tasks with specified parent task id
(node_id:task_number). Set to -1 to cancel all.
:arg wait_for_completion: Should the request block until the
cancellation of the task and its descendant tasks is completed. Defaults
to false
"""
return self.transport.perform_request(
"POST",
tox.ini
+16 -1
View File
@@ -15,6 +15,12 @@ commands =
elasticsearch/ \
test_elasticsearch/ \
setup.py
python utils/license_headers.py fix \
elasticsearch/ \
test_elasticsearch/ \
utils \
docs \
setup.py
[testenv:lint]
deps =
@@ -27,7 +33,16 @@ commands =
setup.py
flake8 \
elasticsearch/ \
test_elasticsearch/
test_elasticsearch/ \
utils \
docs \
setup.py
python utils/license_headers.py check \
elasticsearch/ \
test_elasticsearch/ \
utils \
docs \
setup.py
[testenv:docs]
deps =
utils/generate_api.py
+22
View File
@@ -1,9 +1,15 @@
#!/usr/bin/env python
# Licensed to Elasticsearch B.V under one or more agreements.
# Elasticsearch B.V licenses this file to you under the Apache 2.0 License.
# See the LICENSE file in the project root for more information
import os
import json
import re
import urllib3
from itertools import chain
from functools import lru_cache
import black
from click.testing import CliRunner
@@ -12,11 +18,14 @@ from pathlib import Path
from jinja2 import Environment, FileSystemLoader, TemplateNotFound
http = urllib3.PoolManager()
# line to look for in the original source file
SEPARATOR = " # AUTO-GENERATED-API-DEFINITIONS #"
# global substitutions for python keywords
SUBSTITUTIONS = {"type": "doc_type", "from": "from_"}
# api path(s)
BRANCH_NAME = "7.x"
CODE_ROOT = Path(__file__).absolute().parent.parent
BASE_PATH = (
CODE_ROOT.parent
@@ -53,6 +62,11 @@ def blacken(filename):
assert result.exit_code == 0, result.output
@lru_cache()
def is_valid_url(url):
return http.request("HEAD", url).status == 200
class Module:
def __init__(self, namespace):
self.namespace = namespace
@@ -135,6 +149,14 @@ class API:
if self.doc_url.startswith("http://"):
self.doc_url = self.doc_url.replace("http://", "https://")
# Try setting doc refs like 'current' and 'master' to our branches ref.
if BRANCH_NAME is not None:
revised_url = re.sub("/elasticsearch/reference/[^/]+/", f"/elasticsearch/reference/{BRANCH_NAME}/", self.doc_url)
if is_valid_url(revised_url):
self.doc_url = revised_url
else:
print(f"URL {revised_url!r}, falling back on {self.doc_url!r}")
@property
def all_parts(self):
parts = {}
utils/license_headers.py
+95
View File
@@ -0,0 +1,95 @@
# Licensed to Elasticsearch B.V under one or more agreements.
# Elasticsearch B.V licenses this file to you under the Apache 2.0 License.
# See the LICENSE file in the project root for more information
"""Script which verifies that all source files have a license header.
Has two modes: 'fix' and 'check'. 'fix' fixes problems, 'check' will
error out if 'fix' would have changed the file.
"""
import os
import sys
from typing import List, Iterator
from itertools import chain
lines_to_keep = [
"# -*- coding: utf-8 -*-\n",
"#!/usr/bin/env python\n"
]
license_header_lines = [
"# Licensed to Elasticsearch B.V under one or more agreements.\n",
"# Elasticsearch B.V licenses this file to you under the Apache 2.0 License.\n",
"# See the LICENSE file in the project root for more information\n",
"\n"
]
def find_files_to_fix(sources: List[str]) -> Iterator[str]:
"""Iterates over all files and dirs in 'sources' and returns
only the filepaths that need fixing.
"""
for source in sources:
if os.path.isfile(source) and does_file_need_fix(source):
yield source
elif os.path.isdir(source):
for root, _, filenames in os.walk(source):
for filename in filenames:
filepath = os.path.join(root, filename)
if does_file_need_fix(filepath):
yield filepath
def does_file_need_fix(filepath: str) -> bool:
if not filepath.endswith(".py"):
return False
with open(filepath, mode="r") as f:
first_license_line = None
for line in f:
if line == license_header_lines[0]:
first_license_line = line
break
elif line not in lines_to_keep:
return True
for header_line, line in zip(license_header_lines, chain((first_license_line,), f)):
if line != header_line:
return True
return False
def add_header_to_file(filepath: str) -> None:
with open(filepath, mode="r") as f:
lines = list(f)
i = 0
for i, line in enumerate(lines):
if line not in lines_to_keep:
break
lines = lines[:i] + license_header_lines + lines[i:]
with open(filepath, mode="w") as f:
f.truncate()
f.write("".join(lines))
print(f"Fixed {os.path.relpath(filepath, os.getcwd())}")
def main():
mode = sys.argv[1]
assert mode in ("fix", "check")
sources = [os.path.abspath(x) for x in sys.argv[2:]]
files_to_fix = find_files_to_fix(sources)
if mode == "fix":
for filepath in files_to_fix:
add_header_to_file(filepath)
else:
no_license_headers = list(files_to_fix)
if no_license_headers:
print("No license header found in:")
cwd = os.getcwd()
[print(f" - {os.path.relpath(filepath, cwd)}") for filepath in no_license_headers]
sys.exit(1)
else:
print("All files had license header")
if __name__ == "__main__":
main()