Move bulk helpers' docs outside of streaming_bulk's docstring
This commit is contained in:
@@ -7,12 +7,93 @@ Collection of simple helper functions that abstract some specifics or the raw
|
||||
API.
|
||||
|
||||
|
||||
Bulk helpers
|
||||
------------
|
||||
|
||||
There are several helpers for the ``bulk`` API since it's requirement for
|
||||
specific formatting and other considerations can make it cumbersome if used directly.
|
||||
|
||||
All bulk helpers accept an instance of ``Elasticsearch`` class and an iterable
|
||||
``actions`` (any iterable, can also be a generator, which is ideal in most
|
||||
cases since it will allow you to index large datasets without the need of
|
||||
loading them into memory).
|
||||
|
||||
The items in the ``action`` iterable should be the documents we wish to index
|
||||
in several formats. The most common one is the same as returned by
|
||||
:meth:`~elasticsearch.Elasticsearch.search`, for example:
|
||||
|
||||
.. code:: python
|
||||
|
||||
{
|
||||
'_index': 'index-name',
|
||||
'_type': 'document',
|
||||
'_id': 42,
|
||||
'_parent': 5,
|
||||
'_ttl': '1d',
|
||||
'_source': {
|
||||
"title": "Hello World!",
|
||||
"body": "..."
|
||||
}
|
||||
}
|
||||
|
||||
Alternatively, if `_source` is not present, it will pop all metadata fields
|
||||
from the doc and use the rest as the document data:
|
||||
|
||||
.. code:: python
|
||||
|
||||
{
|
||||
"_id": 42,
|
||||
"_parent": 5,
|
||||
"title": "Hello World!",
|
||||
"body": "..."
|
||||
}
|
||||
|
||||
The :meth:`~elasticsearch.Elasticsearch.bulk` api accepts ``index``, ``create``,
|
||||
``delete``, and ``update`` actions. Use the ``_op_type`` field to specify an
|
||||
action (``_op_type`` defaults to ``index``):
|
||||
|
||||
.. code:: python
|
||||
|
||||
{
|
||||
'_op_type': 'delete',
|
||||
'_index': 'index-name',
|
||||
'_type': 'document',
|
||||
'_id': 42,
|
||||
}
|
||||
{
|
||||
'_op_type': 'update',
|
||||
'_index': 'index-name',
|
||||
'_type': 'document',
|
||||
'_id': 42,
|
||||
'doc': {'question': 'The life, universe and everything.'}
|
||||
}
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
When reading raw json strings from a file, you can also pass them in
|
||||
directly (without decoding to dicts first). In that case, however, you lose
|
||||
the ability to specify anything (index, type, even id) on a per-record
|
||||
basis, all documents will just be sent to elasticsearch to be indexed
|
||||
as-is.
|
||||
|
||||
|
||||
.. py:module:: elasticsearch.helpers
|
||||
|
||||
.. autofunction:: streaming_bulk
|
||||
|
||||
.. autofunction:: parallel_bulk
|
||||
|
||||
.. autofunction:: bulk
|
||||
|
||||
|
||||
Scan
|
||||
----
|
||||
|
||||
.. autofunction:: scan
|
||||
|
||||
|
||||
Reindex
|
||||
-------
|
||||
|
||||
.. autofunction:: reindex
|
||||
|
||||
@@ -142,45 +142,6 @@ def streaming_bulk(client, actions, chunk_size=500, max_chunk_bytes=100 * 1014 *
|
||||
bulk that returns summary information about the bulk operation once the
|
||||
entire input is consumed and sent.
|
||||
|
||||
This function expects the action to be in the format as returned by
|
||||
:meth:`~elasticsearch.Elasticsearch.search`, for example::
|
||||
|
||||
{
|
||||
'_index': 'index-name',
|
||||
'_type': 'document',
|
||||
'_id': 42,
|
||||
'_parent': 5,
|
||||
'_ttl': '1d',
|
||||
'_source': {
|
||||
...
|
||||
}
|
||||
}
|
||||
|
||||
Alternatively, if `_source` is not present, it will pop all metadata fields
|
||||
from the doc and use the rest as the document data.
|
||||
|
||||
When reading raw json strings from a file, you can also pass them in. In
|
||||
that case, however, you lose the ability to specify anything (index, type,
|
||||
even id) on a per-record basis, all documents will just be sent to
|
||||
elasticsearch to be indexed as-is.
|
||||
|
||||
The :meth:`~elasticsearch.Elasticsearch.bulk` api accepts `index`, `create`,
|
||||
`delete`, and `update` actions. Use the `_op_type` field to specify an
|
||||
action (`_op_type` defaults to `index`)::
|
||||
|
||||
{
|
||||
'_op_type': 'delete',
|
||||
'_index': 'index-name',
|
||||
'_type': 'document',
|
||||
'_id': 42,
|
||||
}
|
||||
{
|
||||
'_op_type': 'update',
|
||||
'_index': 'index-name',
|
||||
'_type': 'document',
|
||||
'_id': 42,
|
||||
'doc': {'question': 'The life, universe and everything.'}
|
||||
}
|
||||
|
||||
:arg client: instance of :class:`~elasticsearch.Elasticsearch` to use
|
||||
:arg actions: iterable containing the actions to be executed
|
||||
@@ -208,8 +169,8 @@ def bulk(client, actions, stats_only=False, **kwargs):
|
||||
information - number of successfully executed actions and either list of
|
||||
errors or number of errors if `stats_only` is set to `True`.
|
||||
|
||||
See :func:`~elasticsearch.helpers.streaming_bulk` for more information
|
||||
and accepted formats.
|
||||
See :func:`~elasticsearch.helpers.streaming_bulk` for more accepted
|
||||
parameters
|
||||
|
||||
:arg client: instance of :class:`~elasticsearch.Elasticsearch` to use
|
||||
:arg actions: iterator containing the actions
|
||||
@@ -240,7 +201,7 @@ def parallel_bulk(client, actions, thread_count=4, chunk_size=500,
|
||||
max_chunk_bytes=100 * 1014 * 1024,
|
||||
expand_action_callback=expand_action, **kwargs):
|
||||
"""
|
||||
Parallel version of the bulk helper.
|
||||
Parallel version of the bulk helper run in multiple threads at once.
|
||||
|
||||
:arg client: instance of :class:`~elasticsearch.Elasticsearch` to use
|
||||
:arg actions: iterator containing the actions
|
||||
|
||||
Reference in New Issue
Block a user