Move bulk helpers' docs outside of streaming_bulk's docstring

This commit is contained in:
Honza Král
2015-10-11 23:45:09 +02:00
parent 50a9902733
commit 22943e7e47
2 changed files with 84 additions and 42 deletions
+81
View File
@@ -7,12 +7,93 @@ Collection of simple helper functions that abstract some specifics or the raw
API.
Bulk helpers
------------
There are several helpers for the ``bulk`` API since it's requirement for
specific formatting and other considerations can make it cumbersome if used directly.
All bulk helpers accept an instance of ``Elasticsearch`` class and an iterable
``actions`` (any iterable, can also be a generator, which is ideal in most
cases since it will allow you to index large datasets without the need of
loading them into memory).
The items in the ``action`` iterable should be the documents we wish to index
in several formats. The most common one is the same as returned by
:meth:`~elasticsearch.Elasticsearch.search`, for example:
.. code:: python
{
'_index': 'index-name',
'_type': 'document',
'_id': 42,
'_parent': 5,
'_ttl': '1d',
'_source': {
"title": "Hello World!",
"body": "..."
}
}
Alternatively, if `_source` is not present, it will pop all metadata fields
from the doc and use the rest as the document data:
.. code:: python
{
"_id": 42,
"_parent": 5,
"title": "Hello World!",
"body": "..."
}
The :meth:`~elasticsearch.Elasticsearch.bulk` api accepts ``index``, ``create``,
``delete``, and ``update`` actions. Use the ``_op_type`` field to specify an
action (``_op_type`` defaults to ``index``):
.. code:: python
{
'_op_type': 'delete',
'_index': 'index-name',
'_type': 'document',
'_id': 42,
}
{
'_op_type': 'update',
'_index': 'index-name',
'_type': 'document',
'_id': 42,
'doc': {'question': 'The life, universe and everything.'}
}
.. note::
When reading raw json strings from a file, you can also pass them in
directly (without decoding to dicts first). In that case, however, you lose
the ability to specify anything (index, type, even id) on a per-record
basis, all documents will just be sent to elasticsearch to be indexed
as-is.
.. py:module:: elasticsearch.helpers
.. autofunction:: streaming_bulk
.. autofunction:: parallel_bulk
.. autofunction:: bulk
Scan
----
.. autofunction:: scan
Reindex
-------
.. autofunction:: reindex
+3 -42
View File
@@ -142,45 +142,6 @@ def streaming_bulk(client, actions, chunk_size=500, max_chunk_bytes=100 * 1014 *
bulk that returns summary information about the bulk operation once the
entire input is consumed and sent.
This function expects the action to be in the format as returned by
:meth:`~elasticsearch.Elasticsearch.search`, for example::
{
'_index': 'index-name',
'_type': 'document',
'_id': 42,
'_parent': 5,
'_ttl': '1d',
'_source': {
...
}
}
Alternatively, if `_source` is not present, it will pop all metadata fields
from the doc and use the rest as the document data.
When reading raw json strings from a file, you can also pass them in. In
that case, however, you lose the ability to specify anything (index, type,
even id) on a per-record basis, all documents will just be sent to
elasticsearch to be indexed as-is.
The :meth:`~elasticsearch.Elasticsearch.bulk` api accepts `index`, `create`,
`delete`, and `update` actions. Use the `_op_type` field to specify an
action (`_op_type` defaults to `index`)::
{
'_op_type': 'delete',
'_index': 'index-name',
'_type': 'document',
'_id': 42,
}
{
'_op_type': 'update',
'_index': 'index-name',
'_type': 'document',
'_id': 42,
'doc': {'question': 'The life, universe and everything.'}
}
:arg client: instance of :class:`~elasticsearch.Elasticsearch` to use
:arg actions: iterable containing the actions to be executed
@@ -208,8 +169,8 @@ def bulk(client, actions, stats_only=False, **kwargs):
information - number of successfully executed actions and either list of
errors or number of errors if `stats_only` is set to `True`.
See :func:`~elasticsearch.helpers.streaming_bulk` for more information
and accepted formats.
See :func:`~elasticsearch.helpers.streaming_bulk` for more accepted
parameters
:arg client: instance of :class:`~elasticsearch.Elasticsearch` to use
:arg actions: iterator containing the actions
@@ -240,7 +201,7 @@ def parallel_bulk(client, actions, thread_count=4, chunk_size=500,
max_chunk_bytes=100 * 1014 * 1024,
expand_action_callback=expand_action, **kwargs):
"""
Parallel version of the bulk helper.
Parallel version of the bulk helper run in multiple threads at once.
:arg client: instance of :class:`~elasticsearch.Elasticsearch` to use
:arg actions: iterator containing the actions