ó <¿CVc@ sèdZddlmZddlmZddlmZddlmZddlm Z ddlm Z ddlm Z dd l m Z dd l mZdd lZd e fd „ƒYZde fd„ƒYZdefd„ƒYZd S(sä Translation model that keeps track of vacant positions in the target sentence to decide where to place translated words. Translation can be viewed as a process where each word in the source sentence is stepped through sequentially, generating translated words for each source word. The target sentence can be viewed as being made up of ``m`` empty slots initially, which gradually fill up as generated words are placed in them. Models 3 and 4 use distortion probabilities to decide how to place translated words. For simplicity, these models ignore the history of which slots have already been occupied with translated words. Consider the placement of the last translated word: there is only one empty slot left in the target sentence, so the distortion probability should be 1.0 for that position and 0.0 everywhere else. However, the distortion probabilities for Models 3 and 4 are set up such that all positions are under consideration. IBM Model 5 fixes this deficiency by accounting for occupied slots during translation. It introduces the vacancy function v(j), the number of vacancies up to, and including, position j in the target sentence. Terminology: Maximum vacancy: The number of valid slots that a word can be placed in. This is not necessarily the same as the number of vacant slots. For example, if a tablet contains more than one word, the head word cannot be placed at the last vacant slot because there will be no space for the other words in the tablet. The number of valid slots has to take into account the length of the tablet. Non-head words cannot be placed before the head word, so vacancies to the left of the head word are ignored. Vacancy difference: For a head word: (v(j) - v(center of previous cept)) Can be positive or negative. For a non-head word: (v(j) - v(position of previously placed word)) Always positive, because successive words in a tablet are assumed to appear to the right of the previous word. Positioning of target words fall under three cases: (1) Words generated by NULL are distributed uniformly (2) For a head word t, its position is modeled by the probability v_head(dv | max_v,word_class_t(t)) (3) For a non-head word t, its position is modeled by the probability v_non_head(dv | max_v,word_class_t(t)) dv and max_v are defined differently for head and non-head words. The EM algorithm used in Model 5 is: E step - In the training data, collect counts, weighted by prior probabilities. (a) count how many times a source language word is translated into a target language word (b) for a particular word class and maximum vacancy, count how many times a head word and the previous cept's center have a particular difference in number of vacancies (b) for a particular word class and maximum vacancy, count how many times a non-head word and the previous target word have a particular difference in number of vacancies (d) count how many times a source word is aligned to phi number of target words (e) count how many times NULL is aligned to a target word M step - Estimate new probabilities based on the counts from the E step Like Model 4, there are too many possible alignments to consider. Thus, a hill climbing approach is used to sample good candidates. In addition, pruning is used to weed out unlikely alignments based on Model 4 scores. Notations: i: Position in the source sentence Valid values are 0 (for NULL), 1, 2, ..., length of source sentence j: Position in the target sentence Valid values are 1, 2, ..., length of target sentence l: Number of words in the source sentence, excluding NULL m: Number of words in the target sentence s: A word in the source language t: A word in the target language phi: Fertility, the number of target words produced by a source word p1: Probability that a target word produced by a source word is accompanied by another target word that is aligned to NULL p0: 1 - p1 max_v: Maximum vacancy dv: Vacancy difference, Δv The definition of v_head here differs from GIZA++, section 4.7 of [Brown et al., 1993], and [Koehn, 2010]. In the latter cases, v_head is v_head(v(j) | v(center of previous cept),max_v,word_class(t)). Here, we follow appendix B of [Brown et al., 1993] and combine v(j) with v(center of previous cept) to obtain dv: v_head(v(j) - v(center of previous cept) | max_v,word_class(t)). References: Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York. Peter E Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), 263-311. iÿÿÿÿ(tdivision(t defaultdict(t factorial(t AlignedSent(t Alignment(tIBMModel(t IBMModel4(tCounts(tlongest_target_sentence_lengthNt IBMModel5cB skeZdZdZd d„Zd„Zd„Zd„Zd„Z d„Z d d„Z d „Z d „Z RS( s˜ Translation model that keeps track of vacant positions in the target sentence to decide where to place translated words >>> bitext = [] >>> bitext.append(AlignedSent(['klein', 'ist', 'das', 'haus'], ['the', 'house', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus', 'war', 'ja', 'groß'], ['the', 'house', 'was', 'big'])) >>> bitext.append(AlignedSent(['das', 'buch', 'ist', 'ja', 'klein'], ['the', 'book', 'is', 'small'])) >>> bitext.append(AlignedSent(['ein', 'haus', 'ist', 'klein'], ['a', 'house', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus'], ['the', 'house'])) >>> bitext.append(AlignedSent(['das', 'buch'], ['the', 'book'])) >>> bitext.append(AlignedSent(['ein', 'buch'], ['a', 'book'])) >>> bitext.append(AlignedSent(['ich', 'fasse', 'das', 'buch', 'zusammen'], ['i', 'summarize', 'the', 'book'])) >>> bitext.append(AlignedSent(['fasse', 'zusammen'], ['summarize'])) >>> src_classes = {'the': 0, 'a': 0, 'small': 1, 'big': 1, 'house': 2, 'book': 2, 'is': 3, 'was': 3, 'i': 4, 'summarize': 5 } >>> trg_classes = {'das': 0, 'ein': 0, 'haus': 1, 'buch': 1, 'klein': 2, 'groß': 2, 'ist': 3, 'war': 3, 'ja': 4, 'ich': 5, 'fasse': 6, 'zusammen': 6 } >>> ibm5 = IBMModel5(bitext, 5, src_classes, trg_classes) >>> print(round(ibm5.head_vacancy_table[1][1][1], 3)) 1.0 >>> print(round(ibm5.head_vacancy_table[2][1][1], 3)) 0.0 >>> print(round(ibm5.non_head_vacancy_table[3][3][6], 3)) 1.0 >>> print(round(ibm5.fertility_table[2]['summarize'], 3)) 1.0 >>> print(round(ibm5.fertility_table[1]['book'], 3)) 1.0 >>> print(ibm5.p1) 0.033... >>> test_sentence = bitext[2] >>> test_sentence.words ['das', 'buch', 'ist', 'ja', 'klein'] >>> test_sentence.mots ['the', 'book', 'is', 'small'] >>> test_sentence.alignment Alignment([(0, 0), (1, 1), (2, 2), (3, None), (4, 3)]) gš™™™™™É?cC s>tt|ƒj|ƒ|jƒ||_||_|d kr«t||||ƒ}|j|_|j |_ |j |_ |j |_ |j |_ |j |_ |j|ƒnh|d|_|d|_ |d|_ |d|_ |d|_ |d|_ |d|_|d|_x$td |ƒD]}|j|ƒq#Wd S( s3 Train on ``sentence_aligned_corpus`` and create a lexical translation model, vacancy models, a fertility model, and a model for generating NULL-aligned words. Translation direction is from ``AlignedSent.mots`` to ``AlignedSent.words``. :param sentence_aligned_corpus: Sentence-aligned parallel corpus :type sentence_aligned_corpus: list(AlignedSent) :param iterations: Number of iterations to run training algorithm :type iterations: int :param source_word_classes: Lookup table that maps a source word to its word class, the latter represented by an integer id :type source_word_classes: dict[str]: int :param target_word_classes: Lookup table that maps a target word to its word class, the latter represented by an integer id :type target_word_classes: dict[str]: int :param probability_tables: Optional. Use this to pass in custom probability values. If not specified, probabilities will be set to a uniform distribution, or some other sensible value. If specified, all the following entries must be present: ``translation_table``, ``alignment_table``, ``fertility_table``, ``p1``, ``head_distortion_table``, ``non_head_distortion_table``, ``head_vacancy_table``, ``non_head_vacancy_table``. See ``IBMModel``, ``IBMModel4``, and ``IBMModel5`` for the type and purpose of these tables. :type probability_tables: dict[str]: object ttranslation_tabletalignment_tabletfertility_tabletp1thead_distortion_tabletnon_head_distortion_tablethead_vacancy_tabletnon_head_vacancy_tableiN(tsuperR t__init__treset_probabilitiest src_classest trg_classestNoneRR R R R RRtset_uniform_probabilitiesRRtrangettrain(tselftsentence_aligned_corpust iterationstsource_word_classesttarget_word_classestprobability_tablestibm4tn((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR¯s8$                    c sGttˆƒjƒt‡fd†ƒˆ_t‡fd†ƒˆ_dS(Nc st‡fd†ƒS(Nc st‡fd†ƒS(Nc sˆjS(N(tMIN_PROB((R(se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pytøs(R((R(se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR$øs(R((R(se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR$øsc st‡fd†ƒS(Nc st‡fd†ƒS(Nc sˆjS(N(R#((R(se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR$s(R((R(se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR$s(R((R(se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR$s(RR RRRR(R((Rse/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyRõs c s+t|ƒ}|dkrStdƒd|tjkrStjdt|ƒdƒnxÑtd|dƒD]¼}x³td|dƒD]ž}dd|‰t‡fd†ƒ|j ||           (RR#R7R.RdRRj( RRGt probabilityRiRlRoR}RJRL((R#RGRRKse/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR5ªs,  +      c C s9tj}|j}xŠ|jjƒD]y\}}xj|jƒD]\\}}xM|D]E}|j||||j||} t| |ƒ||||dS|d} |j| ƒ} |j| ƒ} |j| ƒ|j| ƒ} ||d} ||j| }|j| | |c|7<|j| |c|7<|j| ƒ|d8}xÁt d|ƒD]°}||d}|j|ƒ}||} |j| ƒ|} |||d|} ||j| }|j | | |c|7<|j | |c|7<|j| ƒ|d8}qúWdS(s£ :param count: Value to add to the vacancy counts :param alignment_info: Alignment under consideration :param i: Source word position under consideration :param trg_classes: Target word classes :param slots: Vacancy states of the slots in the target sentence. Output parameter that will be modified as new words are placed in the target sentence. iNi( RpR.RqRrRsRdRRRtRR‚Rƒ(RRHRGRLRRKRuRvRwRJRrRxR,R+RyRzR{R|((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR89s6        (R‡RˆR‰RR8(((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyR-)s R7cB s2eZdZd„Zd„Zd„Zd„ZRS(sq Represents positions in a target sentence. Used to keep track of which slot (position) is occupied. cC stg|d|_dS(Ni(tFalset_slots(Rttarget_sentence_length((se/private/var/folders/cc/xm4nqn811x9b50x1q_zpkmvdjlphkp/T/pip-build-FUwmDn/nltk/nltk/translate/ibm5.pyRkscC st|j|ps ÿ¬=