B [`?@sddlZddlZyddlmZWn ek r@ddlmZYnXddlmZddlZddl Z ddl Z ddl m Z ddZ defdd ZeZded d fd d Zd dZdddZdddZejejfddZedkredS)N)Iterable) check_if_numbers_are_consecutive)isnancCs|rd|||S|SdS)a Returns the new_key if no previous key exists, otherwise concatenates previous key, separator, and new_key :param previous_key: :param separator: :param new_key: :return: a string if previous_key exists and simply passes through the new_key otherwise z{}{}{}N)format)Z previous_key separatorZnew_keyr4/tmp/pip-build-hsww7auh/flatten-json/flatten_json.py_construct_keys r _csLt|tstdttjs&tdtfdd|dS)a Flattens a dictionary with nested structure to a dictionary with no hierarchy Consider ignoring keys that you are not interested in to prevent unnecessary processing This is specially true for very deep objects :param nested_dict: dictionary we want to flatten :param separator: string to separate dictionary keys by :param root_keys_to_ignore: set of root keys to ignore from flattening :return: flattened dictionary z#flatten requires a dictionary inputzseparator must be stringcs|s||<n~t|trJxr|D]&}|s.|ks||t||qWnBt|tttfrx0t|D]\}}|t||qdWn||<dS)a= For dict, list and set objects_ calls itself on the elements and for other types assigns the object_ to the corresponding key in the global flattened_dict :param object_: object to flatten :param key: carries the concatenated key for the object_ :return: None N) isinstancedictr listsettuple enumerate)object_key object_keyindexitem)_flattenflattened_dictroot_keys_to_ignorerrrr8s      zflatten.._flattenN)r r AssertionErrorsix string_types) nested_dictrrr)rrrrrflatten$s  rc st|tstdttjs&tdtfddfdd|dt}ttfdd |D}d |d  d d 7ttfd d |D}dd|D}d|gi|dd|ddS)a Flattens a dictionary with nested structure to a dictionary with no hierarchy Consider ignoring keys that you are not interested in to prevent unnecessary processing This is specially true for very deep objects This preserves list structure, and you can specify max_list_index and max_depth to limit processing Child elements with only one value inside will be unwrapped and become parent's value. :param nested_dict: dictionary we want to flatten :param separator: string to separate dictionary keys by :param root_keys_to_ignore: set of root keys to ignore from flattening :param max_list_index: maximum list index to process :param max_depth: maximum nesting depth to process :return: flattened dictionary z#flatten requires a dictionary inputzseparator must be a stringcs|s||<nt|trt|d}t|dkr^t||ts^t||ts^|||<qxv|D]&}|st|ksd||t||qdWnFt|tst|trx0t|D]\}}|t||qWn||<dS)a= For dict, list and set objects_ calls itself on the elements and for other types assigns the object_ to the corresponding key in the global flattened_dict :param object_: object to flatten :param key: carries the concatenated key for the object_ :return: None rN)r r r keyslenr rr)rr first_keyrrr)rrrrrrrxs      z(flatten_preserve_lists.._flattencs|d}d}||krJttt}xt|D] }|||<q6Wn|sttt}xt|D] }|||<qpWnxt|trZt|d}t|dkrt||tst||tsttt}xrt|D]}||||<qWnPxNt| ddddD]4\}} |s6|ks||t ||||qWnt|tsrt|t r|rt d|dtt|ttt} t t| } xt|D]\} } |rt d |d t| d | d }t| trt| d}t|tr|}t|s| kr| dkr\ttt}t | t|d<| |||nqWd dt Dd<x*ttD]}|dkr|=qW|rt dn4ttt}xt|D]}|||<qW|d8}dS)a> For dict, list and set objects_ calls itself on the elements and for other types assigns the object_ to the corresponding key in the global flattened_dict :param object_: object to flatten :param key: carries the concatenated key for the object_ :return: None rrcSs tt|dtt|dfS)Nr)strtyper!)xrrrszFflatten_preserve_lists.._flatten_low_entropy..F)rreversez parent key of list:z | length: z list key:z index: zvals: cSs&g|]\}}t|D] \}}|qqSr)r).0kvidxZsubelrrr szHflatten_preserve_lists.._flatten_low_entropy..0zcollapsed global listN)intmaxr r r#r r r!sorteditemsr rprintcopydeepcopyrvaluesfloatr)rr cur_depthmax_depth_innerdebugZglobal_max_recorddr"rvalZglobal_max_record_startentryrrsubZ first_value)_flatten_low_entropylist_prebuilt_flattened_dictmax_list_indexrrrrr?s         z4flatten_preserve_lists.._flatten_low_entropyNcs&g|]}|D]}|r |q qSr)isdigit)r)rchar)rrrr-!sz*flatten_preserve_lists..|.cs"g|]}tddd|qS)()rF)rer>)r)r)regexrrr-%scSsi|] }d|qS)Nr)r)columnrrr )sz*flatten_preserve_lists..r.r)r8r9) r r rrrr r rjoinreplace) rrrrAZ max_depthZreskeysZunique_integersZunique_columnsZprebuilt_flattened_dictr)rr?rr@rArJrrrflatten_preserve_listsYs&'}      rOcCsHt|tstdt|tjs&tdtdd|DsDtddS)Nz$un_flatten requires dictionary inputzseparator must be stringcss,|]$}| p"t|t p"t|tjVqdS)N)r rrr)r)valuerrr 7sz%_unflatten_asserts..zprovided dict is not flat)r r rrrallr6) flat_dictrrrr_unflatten_asserts4srTcCst||t}dd}t|}xjt|D]^\}}|t|dkrv||d||s||||||qq.||||||q.W|S)z Creates a hierarchical dictionary from a flattened dictionary Assumes no lists are present :param flat_dict: a dictionary with no hierarchy :param separator: a string that separates keys :return: a dictionary with hierarchy cSs2x |ddD]}||i}qW|||d<dS)Nr() setdefault)Zdicr rPrrrr _unflattenIszunflatten.._unflattenr)rTr r1r rr! startswithsplit)rSrunflattened_dictrVZ list_keysirrrr unflatten<s     r[cs0t||t||}fdd|dd|S)a^ Unflattens a dictionary, first assuming no lists exist and then tries to identify lists and replaces them This is probably not very efficient and has not been tested extensively Feel free to add test cases or rewrite the logic Issues that stand out to me: - Sorting all the keys in the dictionary, which specially for the root dictionary can be a lot of keys - Checking that numbers are consecutive is O(N) in number of keys :param flat_dict: dictionary with no hierarchy :param separator: a string that separates keys :return: a dictionary with hierarchy c st|trx*|D]"}t||tr||||qWydd|D}|Wnttfk rng}YnXt|}|dkrt|t|d|dkr|ddkr|d|dkrt|rg||<xBt |D]6\}}|| |t |||d|||qWdS)NcSsg|] }t|qSr)r/)r)rrrrr-xszAunflatten_list.._convert_dict_to_list..rrr() r r sort ValueError TypeErrorr!sumr/rrappendr#)rZ parent_objectZparent_object_keyrr Zkeys_lenZ key_index)_convert_dict_to_listrrrbrs(    &  z-unflatten_list.._convert_dict_to_listN)rTr[)rSrrYr)rbrunflatten_list^s     rccCs<|}t|}tt|}|d||dS)Nz{} )readjsonloadsdumpsrwriterflush)Z input_streamZ output_streamrawZ input_jsonoutputrrrclis  rl__main__)r )r )sysrecollections.abcr ImportError collectionsutilrrr4rImathrr rrZ flatten_jsonrOrTr[rcstdinstdoutrl__name__rrrrs.  2Z " 3