ó <¿CVc @sƒdZddlmZmZddlZddlZddlZddlZddlZ ddl m Z m Z idd6dgd6d d gd 6d d 6Z idd6dgd6d dgd 6dd 6Zidd6gd6dddgd 6dd 6Zidd6gd6dd ddddddgd 6dd 6Zidd6gd6ddgd 6dd 6Zidd6gd6ddgd 6dd 6Zid d6gd6d gd 6dd 6Zid!d6gd6d d gd 6dd 6Zid"d6gd6d"gd 6dd 6Zid#d6gd6d#gd 6dd 6Zdd$ddd%d&d d d"d#g ZeeeƒƒZi e d6ed$6ed6ed6ed%6ed&6ed 6ed 6ed"6ed#6ZejƒZd dgZe d'efd(„ƒYƒZgd)„Z e!e!d*„Z"d+„Z#d,„Z$d-„Z%d.„Z&d/„Z'e!e!d0„Z(d1„Z)d2„Z*e!d3„Z+d4„Z,ed5„Z-d6„Z.d7„Z/e0d8kre.ƒe/ƒndS(9u½ Overview ======== Chat-80 was a natural language system which allowed the user to interrogate a Prolog knowledge base in the domain of world geography. It was developed in the early '80s by Warren and Pereira; see ``http://www.aclweb.org/anthology/J82-3002.pdf`` for a description and ``http://www.cis.upenn.edu/~pereira/oldies.html`` for the source files. This module contains functions to extract data from the Chat-80 relation files ('the world database'), and convert then into a format that can be incorporated in the FOL models of ``nltk.sem.evaluate``. The code assumes that the Prolog input files are available in the NLTK corpora directory. The Chat-80 World Database consists of the following files:: world0.pl rivers.pl cities.pl countries.pl contain.pl borders.pl This module uses a slightly modified version of ``world0.pl``, in which a set of Prolog rules have been omitted. The modified file is named ``world1.pl``. Currently, the file ``rivers.pl`` is not read in, since it uses a list rather than a string in the second field. Reading Chat-80 Files ===================== Chat-80 relations are like tables in a relational database. The relation acts as the name of the table; the first argument acts as the 'primary key'; and subsequent arguments are further fields in the table. In general, the name of the table provides a label for a unary predicate whose extension is all the primary keys. For example, relations in ``cities.pl`` are of the following form:: 'city(athens,greece,1368).' Here, ``'athens'`` is the key, and will be mapped to a member of the unary predicate *city*. The fields in the table are mapped to binary predicates. The first argument of the predicate is the primary key, while the second argument is the data in the relevant field. Thus, in the above example, the third field is mapped to the binary predicate *population_of*, whose extension is a set of pairs such as ``'(athens, 1368)'``. An exception to this general framework is required by the relations in the files ``borders.pl`` and ``contains.pl``. These contain facts of the following form:: 'borders(albania,greece).' 'contains0(africa,central_africa).' We do not want to form a unary concept out the element in the first field of these records, and we want the label of the binary relation just to be ``'border'``/``'contain'`` respectively. In order to drive the extraction process, we use 'relation metadata bundles' which are Python dictionaries such as the following:: city = {'label': 'city', 'closures': [], 'schema': ['city', 'country', 'population'], 'filename': 'cities.pl'} According to this, the file ``city['filename']`` contains a list of relational tuples (or more accurately, the corresponding strings in Prolog form) whose predicate symbol is ``city['label']`` and whose relational schema is ``city['schema']``. The notion of a ``closure`` is discussed in the next section. Concepts ======== In order to encapsulate the results of the extraction, a class of ``Concept`` objects is introduced. A ``Concept`` object has a number of attributes, in particular a ``prefLabel`` and ``extension``, which make it easier to inspect the output of the extraction. In addition, the ``extension`` can be further processed: in the case of the ``'border'`` relation, we check that the relation is symmetric, and in the case of the ``'contain'`` relation, we carry out the transitive closure. The closure properties associated with a concept is indicated in the relation metadata, as indicated earlier. The ``extension`` of a ``Concept`` object is then incorporated into a ``Valuation`` object. Persistence =========== The functions ``val_dump`` and ``val_load`` are provided to allow a valuation to be stored in a persistent database and re-loaded, rather than having to be re-computed each time. Individuals and Lexical Items ============================= As well as deriving relations from the Chat-80 data, we also create a set of individual constants, one for each entity in the domain. The individual constants are string-identical to the entities. For example, given a data item such as ``'zloty'``, we add to the valuation a pair ``('zloty', 'zloty')``. In order to parse English sentences that refer to these entities, we also create a lexical item such as the following for each individual constant:: PropN[num=sg, sem=<\P.(P zloty)>] -> 'Zloty' The set of rules is written to the file ``chat_pnames.cfg`` in the current directory. iÿÿÿÿ(tprint_functiontunicode_literalsN(t string_typestpython_2_unicode_compatibleubordersurel_nameu symmetricuclosuresuregionuborderuschemau borders.plufilenameu contains0u transitiveucontainu contain.plucityucountryu populationu cities.plulatitudeu longitudeuareaucapitalucurrencyu countries.plucircle_of_latitudeudegreesu world1.plucircle_of_longitudeu continentu in_continentuoceanuseaucontainsu circle_of_latucircle_of_longtConceptcBsbeZdZggeƒd„Zd„Zd„Zd„Zd„Zd„Z d„Z d„Z RS( ub A Concept class, loosely based on SKOS (http://www.w3.org/TR/swbp-skos-core-guide/). cCsF||_||_||_||_||_tt|ƒƒ|_dS(u :param prefLabel: the preferred label for the concept :type prefLabel: str :param arity: the arity of the concept :type arity: int @keyword altLabels: other (related) labels :type altLabels: list @keyword closures: closure properties of the extension (list items can be ``symmetric``, ``reflexive``, ``transitive``) :type closures: list @keyword extension: the extensional value of the concept :type extension: set N(t prefLabeltarityt altLabelstclosurest _extensiontsortedtlistt extension(tselfRRRRR ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyt__init__Ýs      cCsd|j|j|jfS(Nu&Label = '%s' Arity = %s Extension = %s(RRR (R ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyt__str__ôscCs d|jS(Nu Concept('%s')(R(R ((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyt__repr__ÿscCs/|jj|ƒtt|jƒƒ|_|jS(u± Add more data to the ``Concept``'s extension set. :param data: a new semantic value :type data: string or pair of strings :rtype: set (R taddR R R (R tdata((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pytaugments cCsNi}xA|D]9\}}||kr9||j|ƒq |g||R?t ValueErrortOperationalErrortwarningstwarn(RBtqueryR<tpathRERFRO((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyt sql_query—s   cCs™g}tjjd|ddƒ}xp|jƒD]b}|j|ƒr/tj|dd|ƒ}tjdd|ƒ}|jdƒ}|j|ƒq/q/W|S(uO Read a file into memory and convert each relation clause into a list. ucorpora/chat80/%stformatutextu\(uu\)\.$u,( RJRtloadt splitlinest startswithtretsubtsplitR(R1treltrecstcontentstlinetrecord((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyR,ªscCsAt|dddtƒƒ}x|D]}|j||ƒq"W|S(uü Make a unary concept out of the primary key in a record. A record is a list of entities in some relation, such as ``['france', 'paris']``, where ``'france'`` is acting as the primary key. :param label: the preferred label for the concept :type label: string :param subj: position in the record of the subject of the predicate :type subj: int :param records: a list of records :type records: list of lists :return: ``Concept`` of arity 1 :rtype: Concept RiR (RRR(tlabelR5R8tcR_((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyR.¸s cCs‚|dk r'|dk r'|d}nt|ddd|dtƒƒ}x)|D]!}|j||||fƒqOW|jƒ|S(u$ Make a binary concept out of the primary key and another field in a record. A record is a list of entities in some relation, such as ``['france', 'paris']``, where ``'france'`` is acting as the primary key, and ``'paris'`` stands in the ``'capital_of'`` relation to ``'france'``. More generally, given a record such as ``['a', 'b', 'c']``, where label is bound to ``'B'``, and ``obj`` bound to 1, the derived binary concept will have label ``'B_of'``, and its extension will be a set of pairs such as ``('a', 'b')``. :param label: the base part of the preferred label for the concept :type label: str :param closures: closure properties for the extension of the concept :type closures: list :param subj: position in the record of the subject of the predicate :type subj: int :param obj: position in the record of the object of the predicate :type obj: int :param records: a list of records :type records: list of lists :return: ``Concept`` of arity 2 :rtype: Concept uborderucontainu_ofRiRR (RRRR((R`RR5R:R8RaR_((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyR0Îs !  c Cs¾i}x±|D]©}|d}|d}|d}|d}t||||ƒ}xc|D][}|j} | |kr¨x"|jD]} || j| ƒq|W|| jƒqW|||  Create lexical CFG rules for each individual symbol. Given a valuation with an entry of the form ``{'zloty': 'zloty'}``, create a lexical rule for the proper name 'Zloty'. :param symbols: a list of individual constants in the semantic representation :type symbols: sequence :rtype: list uË ################################################################## # Lexical rules automatically generated by running 'chat80.py -x'. ################################################################## u'PropN[num=sg, sem=<\P.(P %s)>] -> '%s' u_(RRZt capitalizetjoin( tsymbolsR€theaderttemplateRtpartstptcapstpnametrule((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyR}{s   cCsNt|tƒr|f}ng|D]}t|^q"}t|ƒ}|jƒS(u Build a list of concepts corresponding to the relation names in ``items``. :param items: names of the Chat-80 relations to extract :type items: list of strings :return: the ``Concept`` objects which are extracted from the relations :rtype: list (t isinstanceRt item_metadataRdRn(titemstrRbt concept_map((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyR4s   c Cs:ddl}ddlm}d}|d|ƒ}|jdtdtdtƒ|jdd d d d d ddƒ|jddd dd dddƒ|jddddd dƒ|jddd dd dddƒ|jdddd d d!d d"ƒ|jd#d$ddd d%d d&ƒ|jd'd(ddd d)d d*ƒ|jƒ\}}|jri|j ri|j d+ƒn|jr¹|j r™|jd,}t d-|ƒnt t|jƒ|jd.ƒn}|j dk r|j d,}tj|tjƒsþ|jd/|ƒq6t|j ƒ}n&ttƒ} | jƒ} |jrtg| D]} | j| jf^q;ƒ} x!| D]\} }t || ƒqcW|jd.ƒn|jr¾x"| D]} t | ƒt ƒq Wn|jrèt | |jƒ|jd.ƒnN|jr|j rt d0ƒnt| d1tƒnt| d2tƒ}t |ƒdS(3Niÿÿÿÿ(t OptionParseru€ Extract data from the Chat-80 Prolog files and convert them into a Valuation object for use in the NLTK semantics package. t descriptionRCR€tvocabu-su--storetdestuoutdbthelpustore a valuation in DBtmetavaruDBu-lu--loaduindbuload a stored valuation from DBu-cu --conceptstactionu store_trueu%print concepts instead of a valuationu-ru --relationulabeluEprint concept with label REL (check possible labels with '-v' option)uRELu-qu--quietu store_falseuverboseudon't print out progress infou-xu--lexulexu<write a file of lexical entries for country names, then exitu-vu--vocabuvocabuEprint out the vocabulary of concept labels and their arity, then exitu1Options --store and --load are mutually exclusiveu.dbuDumping a valuation to %siuCannot read file: %suWriting out lexical rulesRfRj(RxtoptparseR‘t set_defaultsRgtFalset add_optiont parse_argstoutdbtindbterrorRCR@RtRbRytNoneRuRvRwR{RdRnR“R RRR4R`R€Rm(RxR‘R’toptstoptionstargsRRBRrRR4RaRŽRR`((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pytmain³sj           +       cCs9tƒtdƒx!tddƒD]}t|ƒq!WdS(u: Print out every row from the 'city.db' database. u-Using SQL to extract rows from 'city.db' RDB.ucorpora/city_database/city.dbuSELECT * FROM city_tableN(R@RS(trow((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pytsql_demoÿs u__main__(1R+t __future__RRRXRoRuRxt nltk.dataRJt nltk.compatRRtborderstcontainstcitytcountryt circle_of_lattcircle_of_longt continenttregiontoceantseaRŽttupleR RRnRbR-tobjectRR;RšRIRSR,R.R0RdRmRtR{RiR}R4R¤R¦R)(((sa/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/sem/chat80.pyt|s¨                                       t '$    &   %  "  L