_Y< @sBddlmZddlZddlmZddlmZmZddlm Z m Z ej dZ Gdd d eZ Gd d d eZd d ZddZddddZdd%dedddZdddZddd'eddZdddddddd d!Zddddiid"d#ZdS)()unicode_literalsN) methodcaller)ElasticsearchExceptionTransportError)map string_typeszelasticsearch.helpersc@s"eZdZeddZdS)BulkIndexErrorcCs |jdS)z2 List of errors from execution of the last chunk. )args)selfr G/tmp/pip-build-66pc0izb/elasticsearch/elasticsearch/helpers/__init__.pyerrors szBulkIndexError.errorsN)__name__ __module__ __qualname__propertyrr r r rr s r cs"eZdZfddZS) ScanErrorcs&tt|j||||_dS)N)superr__init__ scroll_id)r rr kwargs) __class__r rrszScanError.__init__)rrrrr r )rrrs rc Cst|trd|fS|j}|jdd}|ii}x1dD])}||krJ|j||||szparallel_bulk..N) Zmultiprocessing.dummyr]rimapr:rEr7closerC) rKr4Z thread_countr5r6rVrr]poolrWrRr )rKrr parallel_bulks  "  rdZ5mic ks|s(|r|jni}d|d<|jd|d|d|d||} | jd} | dkrndSzd } x| rd } n|j| d|d|} x| d d D] } | VqW| d d r"tjd| d d | d d|r"t| d| d d | d df| jd} | dksL| d d  rzPqzWWd| r|r|jdd| giddXdS)a Simple abstraction on top of the :meth:`~elasticsearch.Elasticsearch.scroll` api - a simple iterator that yields all hits as returned by underlining scroll requests. By default scan does not return results in any pre-determined order. To have a standard order in the returned documents (either by score or explicit sort definition) when scrolling, use ``preserve_order=True``. This may be an expensive operation and will negate the performance benefits of using ``scan``. :arg client: instance of :class:`~elasticsearch.Elasticsearch` to use :arg query: body for the :meth:`~elasticsearch.Elasticsearch.search` api :arg scroll: Specify how long a consistent view of the index should be maintained for scrolled search :arg raise_on_error: raises an exception (``ScanError``) if an error is encountered (some shards fail to execute). By default we raise. :arg preserve_order: don't set the ``search_type`` to ``scan`` - this will cause the scroll to paginate with preserving the order. Note that this can be an extremely expensive operation and can easily lead to unpredictable results, use with caution. :arg size: size (per shard) of the batch send at each iteration. :arg request_timeout: explicit timeout for each call to ``scan`` :arg clear_scroll: explicitly calls delete on the scroll id via the clear scroll API at the end of the method on completion or error, defaults to true. Any additional keyword arguments will be passed to the initial :meth:`~elasticsearch.Elasticsearch.search` call:: scan(es, query={"query": {"match": {"title": "python"}}}, index="orders-*", doc_type="books" ) Z_docsortbodyscrollr9request_timeoutZ _scroll_idNTFhitsZ_shardsr[z1Scroll request has failed on %d shards out of %d.totalrignore)rl)r)searchr+rgloggerwarningr clear_scroll) rKqueryrgrMZpreserve_orderr9rhrprrNrZ first_runhitr r rscans<'        rsc Cs}|dkr|n|}t|d|d|d||} dd} ddi} | j|t|| | |d || S) a Reindex all documents from one index that satisfy a given query to another, potentially (if `target_client` is specified) on a different cluster. If you don't specify the query you will reindex all the documents. Since ``2.3`` a :meth:`~elasticsearch.Elasticsearch.reindex` api is available as part of elasticsearch itself. It is recommended to use the api instead of this helper wherever possible. The helper is here mostly for backwards compatibility and for situations where more flexibility is needed. .. note:: This helper doesn't transfer mappings, just the data. :arg client: instance of :class:`~elasticsearch.Elasticsearch` to use (for read if `target_client` is specified as well) :arg source_index: index (or list of indices) to read documents from :arg target_index: name of the index in the target cluster to populate :arg query: body for the :meth:`~elasticsearch.Elasticsearch.search` api :arg target_client: optional, is specified will be used for writing (thus enabling reindex between clusters) :arg chunk_size: number of docs in one chunk sent to es (default: 500) :arg scroll: Specify how long a consistent view of the index should be maintained for scrolled search :arg scan_kwargs: additional kwargs to be passed to :func:`~elasticsearch.helpers.scan` :arg bulk_kwargs: additional kwargs to be passed to :func:`~elasticsearch.helpers.bulk` NrqrrgcssFx?|D]7}||d._change_doc_indexrYTr5)rsrIrB) rKZ source_indexZ target_indexrqZ target_clientr5rgZ scan_kwargsZ bulk_kwargsZdocsrvrr r rreindexFs!     rwii@ii@) __future__rloggingoperatorr exceptionsrrcompatrr getLoggerrnr rr0r:rTrXrBrdrsrwr r r rs*   :($ R