ó <¿CVc@sÂdZddlmZddlZddlZddlZddljZdZd„Z d„Z d„Z d„Z d „Z d d ed „Zed „Zd d ed„Zd„Zd„ZdS(s} Utility functions for the :module:`twitterclient` module which do not require the `twython` library to have been installed. iÿÿÿÿ(tprint_functionNt.cCsRg}xE|D]=}yt|||ƒWq tk rItd|ƒ‚q Xq W|S(sÓ Extract field values from a full tweet and return them as a list :param json tweet: The tweet in JSON format :param list fields: The fields to be extracted from the tweet :rtype: list(str) s6Fatal error when extracting fields. Cannot find field (t_add_field_to_outt TypeErrort RuntimeError(ttweettfieldstouttfield((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pytextract_fieldss  cCsJt|ƒr5t|ƒ\}}t||||ƒn|||g7}dS(N(t_is_composed_keyt_get_key_value_composedR(tjsonRRtkeytvalue((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pyR)s cCst|krtStS(N(tHIER_SEPARATORtTruetFalse(R((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pyR 0s cCs6|jtƒ}|d}tj|dƒ}||fS(Nii(tsplitRtjoin(RRR R((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pyR 5s cCsÏ|s dSt|tƒr„xd|jƒD]V\}}||krB|S|dksZ|dkr&t||ƒ}|dk r||Sq&q&WdSt|tƒrÇx-|D]%}t||ƒ}|dk rš|SqšWdSdSdS(Ntentitiestextended_entities(tNonet isinstancetdicttitemst_get_entity_recursivetlist(R tentityR Rt candidatetitem((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pyR<s$     tutf8treplacec Csrt||||ƒ\}}|j|ƒx9|D]1}tj|ƒ} t| |ƒ} |j| ƒq/W|jƒdS(s: Extract selected fields from a file of line-separated JSON tweets and write to a file in CSV format. This utility function allows a file of full tweets to be easily converted to a CSV file for easier processing. For example, just TweetIDs or just the text content of the Tweets can be extracted. Additionally, the function allows combinations of fields of other Twitter objects (mainly the users, see below). For Twitter entities (e.g. hashtags of a Tweet), and for geolocation, see `json2csv_entities` :param str infile: The name of the file containing full tweets :param str outfile: The name of the text file where results should be written :param list fields: The list of fields to be extracted. Useful examples are 'id_str' for the tweetID and 'text' for the text of the tweet. See for a full list of fields. e. g.: ['id_str'], ['id', 'text', 'favorite_count', 'retweet_count'] Additonally, it allows IDs from other Twitter objects, e. g., ['id', 'text', 'user.id', 'user.followers_count', 'user.friends_count'] :param error: Behaviour for encoding errors, see https://docs.python.org/3/library/codecs.html#codec-base-classes :param gzip_compress: if `True`, output files are compressed with gzip N(toutf_writer_compattwriterowR tloadsR tclose( tfptoutfileRtencodingterrorst gzip_compresstwritertoutftlineRtrow((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pytjson2csvUs!  cCs¬tjr]|r0tj|dd|d|ƒ}nt|dd|d|ƒ}tj|ƒ}nE|rxtj|dƒ}nt|dƒ}tj|d|d|ƒ}||fS(sB Identify appropriate CSV writer given the Python version twtR'R(twtwb(tcompattPY3tgziptopentcsvR*t UnicodeWriter(R&R'R(R)R+R*((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pyR!s !cCst||||ƒ\}} t|||ƒ} |j| ƒx»|D]³} tj| ƒ} t|ƒrÃt|ƒ\} }t| | ƒ}|sqAnt||ƒ}t||ƒ}t ||||ƒqAt| |ƒ}t| |ƒ}t ||||ƒqAW| j ƒdS(sl Extract selected fields from a file of line-separated JSON tweets and write to a file in CSV format. This utility function allows a file of full Tweets to be easily converted to a CSV file for easier processing of Twitter entities. For example, the hashtags or media elements of a tweet can be extracted. It returns one line per entity of a Tweet, e.g. if a tweet has two hashtags there will be two lines in the output file, one per hashtag :param tweets_file: the file-like object containing full Tweets :param str outfile: The path of the text file where results should be written :param list main_fields: The list of fields to be extracted from the main object, usually the tweet. Useful examples: 'id_str' for the tweetID. See for a full list of fields. e. g.: ['id_str'], ['id', 'text', 'favorite_count', 'retweet_count'] If `entity_type` is expressed with hierarchy, then it is the list of fields of the object that corresponds to the key of the entity_type, (e.g., for entity_type='user.urls', the fields in the main_fields list belong to the user object; for entity_type='place.bounding_box', the files in the main_field list belong to the place object of the tweet). :param list entity_type: The name of the entity: 'hashtags', 'media', 'urls' and 'user_mentions' for the tweet object. For a user object, this needs to be expressed with a hierarchy: `'user.urls'`. For the bounding box of the Tweet location, use `'place.bounding_box'`. :param list entity_fields: The list of fields to be extracted from the entity. E.g. `['text']` (of the Tweet) :param error: Behaviour for encoding errors, see https://docs.python.org/3/library/codecs.html#codec-base-classes :param gzip_compress: if `True`, ouput files are compressed with gzip N( R!tget_header_field_listR"R R#R R RR t_write_to_fileR$(t tweets_fileR&t main_fieldst entity_typet entity_fieldsR'R(R)R*R+theaderR,RR Rt object_jsont object_fieldsRt tweet_fields((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pytjson2csv_entities”s"*   c Cs t|ƒr-t|ƒ\}}|}|}n d}|}|rjg|D]}tj||gƒ^qF}n|}g|D]}tj||gƒ^qw} || S(N(R R RRR( R;R<R=R Rt main_entityt sub_entitytxtoutput1toutput2((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pyR8Òs  +(cCsX|s dSt|tƒr#|}g|D]}t|ƒs&|^q&}g|D]}t|ƒrK|^qK}xA|D]9}||} t| tƒrœ|| 7}qp|| g7}qpWxb|D]Z} t| ƒ\} } || } t| tƒsýtdj| ƒƒ‚n|| | g7}q´W|j|ƒdSx.|D]&}|t||ƒ}|j|ƒq*WdS(NsFKey {0} does not contain a dictionary in the json file( RRR RR RtformatR"R (R@RR=R*R-REtentity_field_valuestentity_field_composedRRtdtkdtvdt json_dictR((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pyR9âs.%%       (t__doc__t __future__RR6R4R t nltk.compatR2RR RR R RRR.R!RBR8R9(((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/twitter/common.pyt s"         + =