B @`o@sdZdZddlZddlZddlmZdZe ZddZdd Z d d Z d d Z ddZ ddZ e e ZZZe eZZe e ejZZe dZZZe dZZZe dZddZddZddZddZ ddZ!dd Z"d!d"Z#d#d$Z$e#Z%e$Z&d%d&Z'd'd(Z(d)d*Z)d+d,Z*d-d.Z+d/d0Z,e)ej-e&e Z.Z/Z0e)ej-e%eZ1Z2e*ej-e%eejZ3Z4e+ej5d1Z6e+ej7d2Z8e+ej5d3Z9e+ej7d4Z:e,ej5d5Z;e,ej7d6Zd;d<Z?d=d>Z@d?d@ZAdAdBZBdCdDZCdS)EaCode for encoding protocol message primitives. Contains the logic for encoding every logical protocol field type into one of the 5 physical wire types. This code is designed to push the Python interpreter's performance to the limits. The basic idea is that at startup time, for every field (i.e. every FieldDescriptor) we construct two functions: a "sizer" and an "encoder". The sizer takes a value of this field's type and computes its byte size. The encoder takes a writer function and a value. It encodes the value into byte strings and invokes the writer function to write those strings. Typically the writer function is the write() method of a BytesIO. We try to do as much work as possible when constructing the writer and the sizer rather than when calling them. In particular: * We copy any needed global functions to local variables, so that we do not need to do costly global table lookups at runtime. * Similarly, we try to do any attribute lookups at startup time if possible. * Every field's tag is encoded to bytes at startup, since it can't change at runtime. * Whatever component of the field size we can compute at startup, we do. * We *avoid* sharing code if doing so would make the code slower and not sharing does not burden us too much. For example, encoders for repeated fields do not just call the encoders for singular fields in a loop because this would add an extra function call overhead for every loop iteration; instead, we manually inline the single-value encoder into the loop. * If a Python function lacks a return statement, Python actually generates instructions to pop the result of the last statement off the stack, push None onto the stack, and then return that. If we really don't care what value is returned, then we can save two instructions by returning the result of the last statement. It looks funny but it helps. * We assume that type and bounds checking has happened at a higher level. z kenton@google.com (Kenton Varda)N) wire_formatgcCsp|dkr dS|dkrdS|dkr$dS|dkr0dS|d kr.SpecificSizer..PackedFieldSizecs*t|}x|D]}||7}qW|S)N)len)rrr)rrrrRepeatedFieldSizes  z>_SimpleSizer..SpecificSizer..RepeatedFieldSizecs |S)Nr)r)rrrr FieldSizesz6_SimpleSizer..SpecificSizer..FieldSize)rr)r is_repeated is_packedrrr)r)rrr SpecificSizersz#_SimpleSizer..SpecificSizerr)rr!r)rr _SimpleSizer~s r"csfdd}|S)zLike SimpleSizer, but modify_value is invoked on each value before it is passed to compute_value_size. modify_value is typically ZigZagEncode.csVt||r&tfdd}|S|r>fdd}|Sfdd}|SdS)Ncs2d}x|D]}||7}q W||S)Nrr)rrr)rr modify_valuerrrrs z>_ModifiedSizer..SpecificSizer..PackedFieldSizecs.t|}x|D]}||7}qW|S)N)r)rrr)rr#rrrrs  z@_ModifiedSizer..SpecificSizer..RepeatedFieldSizecs|S)Nr)r)rr#rrrrsz8_ModifiedSizer..SpecificSizer..FieldSize)rr)rrr rrr)rr#)rrrr!sz%_ModifiedSizer..SpecificSizerr)rr#r!r)rr#r_ModifiedSizersr$csfdd}|S)zWLike _SimpleSizer except for a fixed-size field. The input is the size of one value.cs\t||r$tfdd}|S|r@fdd}|Sfdd}|SdS)Ncst|}||S)N)r)rr)rr value_sizerrrs z;_FixedSizer..SpecificSizer..PackedFieldSizecs t|S)N)r)r) element_sizerrrsz=_FixedSizer..SpecificSizer..RepeatedFieldSizecsS)Nr)r) field_sizerrrsz5_FixedSizer..SpecificSizer..FieldSize)rr)rrr rrr)r%)r&r'rrrr!s  z"_FixedSizer..SpecificSizerr)r%r!r)r%r _FixedSizers r(rr rcsHt|tt|rt|r0fdd}|Sfdd}|SdS)z#Returns a sizer for a string field.cs<t|}x*|D]"}|d}|||7}qW|S)Nzutf-8)rencode)rrrl)r local_lenrrrrs   z&StringSizer..RepeatedFieldSizecs|d}||S)Nzutf-8)r))rr*)rr+rrrrszStringSizer..FieldSizeN)rrrAssertionError)rrr rrr)rr+rr StringSizersr-csHt|tt|rt|r0fdd}|Sfdd}|SdS)z"Returns a sizer for a bytes field.cs6t|}x$|D]}|}|||7}qW|S)N)r)rrrr*)rr+rrrrs   z%BytesSizer..RepeatedFieldSizecs|}||S)Nr)rr*)rr+rrrr szBytesSizer..FieldSizeN)rrrr,)rrr rrr)rr+rr BytesSizersr.cs<t|d|rt|r(fdd}|Sfdd}|SdS)z"Returns a sizer for a group field.rcs*t|}x|D]}||7}qW|S)N)rByteSize)rrr)rrrrs  z%GroupSizer..RepeatedFieldSizecs |S)N)r/)r)rrrrszGroupSizer..FieldSizeN)rr,)rrr rrr)rr GroupSizers   r0cs@t|t|rt|r*fdd}|Sfdd}|SdS)z$Returns a sizer for a message field.cs6t|}x$|D]}|}|||7}qW|S)N)rr/)rrrr*)rrrrr+s   z'MessageSizer..RepeatedFieldSizecs|}||S)N)r/)rr*)rrrrr3szMessageSizer..FieldSizeN)rrr,)rrr rrr)rrr MessageSizer$sr1cs:tddtdt|tdtfdd}|S)zReturns a sizer for extensions of MessageSet. The message set message looks like this: message MessageSet { repeated group Item = 1 { required int32 type_id = 2; required string message = 3; } } rrrcs|}||S)N)r/)rr*)r static_sizerrrLsz&MessageSetItemSizer..FieldSize)rr)rrr)rr2rMessageSetItemSizer=s  r3cs(|jt|jddfdd}|S)z Returns a sizer for a map field.FcsDd}x:|D]2}||}j||d}||7}r |q W|S)Nr)keyr)_concrete_classr/)Z map_valuetotalr4r entry_msg)is_message_map message_sizer message_typerrr_s   zMapSizer..FieldSize)r:r1number)field_descriptorr8rr)r8r9r:rMapSizerWsr=cstjdfdd }|S)zBReturn an encoder for a basic varint value (does not include tag).NcsF|d@}|dL}x(|r8|d|B|d@}|dL}qW||S)Nrr r)writerunused_deterministicbits)local_int2byterr EncodeVarintxs z$_VarintEncoder..EncodeVarint)N)sixint2byte)rCr)rBr_VarintEncoderts rFcstjdfdd }|S)zKReturn an encoder for a basic signed varint value (does not include tag).NcsV|dkr|d7}|d@}|dL}x(|rH|d|B|d@}|dL}q"W||S)Nrlrr r>r)r?rr@rA)rBrrEncodeSignedVarints z0_SignedVarintEncoder..EncodeSignedVarint)N)rDrE)rGr)rBr_SignedVarintEncoders rHcCsg}t|j|dd|S)zEncode the given integer as a varint and return the bytes. This is only called at startup time so it doesn't need to be fast.T) _EncodeVarintappendjoin)rpiecesrrr _VarintBytessrNcCsttt||S)zCEncode the given tag and return the bytes. Only called at startup.)rD binary_typerNrr)r wire_typerrrTagBytessrQcsfdd}|S)a_Return a constructor for an encoder for fields of a particular type. Args: wire_type: The field's wire type, for encoding tags. encode_value: A function which encodes an individual value, e.g. _EncodeVarint(). compute_value_size: A function which computes the size of an individual value, e.g. _VarintSize(). csj|r*t|tjtfdd}|S|rJt|fdd}|St|fdd}|SdS)NcsP|d}x|D]}||7}qW|||x|D]}|||q8WdS)Nrr)r?r deterministicsizer)r encode_valuelocal_EncodeVarint tag_bytesrrEncodePackedFields   zB_SimpleEncoder..SpecificEncoder..EncodePackedFieldcs&x |D]}||||qWdS)Nr)r?rrRr)rTrVrrEncodeRepeatedFields zD_SimpleEncoder..SpecificEncoder..EncodeRepeatedFieldcs||||S)Nr)r?rrR)rTrVrr EncodeFieldsz<_SimpleEncoder..SpecificEncoder..EncodeField)rQrWIRETYPE_LENGTH_DELIMITEDrJ)rrr rWrXrY)rrTrP)rUrVrSpecificEncoders   z'_SimpleEncoder..SpecificEncoderr)rPrTrr[r)rrTrPr_SimpleEncoders r\csfdd}|S)zLike SimpleEncoder but additionally invokes modify_value on every value before passing it to encode_value. Usually modify_value is ZigZagEncode.csp|r,t|tjtfdd}|S|rNt|fdd}|St|fdd}|SdS)NcsX|d}x|D]}||7}qW|||x|D]}|||q.SpecificEncoder..EncodePackedFieldcs*x$|D]}||||qWdS)Nr)r?rrRr)rTr#rVrrrXs zF_ModifiedEncoder..SpecificEncoder..EncodeRepeatedFieldcs||||S)Nr)r?rrR)rTr#rVrrrYsz>_ModifiedEncoder..SpecificEncoder..EncodeField)rQrrZrJ)rrr rWrXrY)rrTr#rP)rUrVrr[s   z)_ModifiedEncoder..SpecificEncoderr)rPrTrr#r[r)rrTr#rPr_ModifiedEncodersr]cstfdd}|S)zReturn a constructor for an encoder for a fixed-width field. Args: wire_type: The field's wire type, for encoding tags. format: The format string to pass to struct.pack(). csztj|r2t|tjtfdd}|S|rVt|dfdd }|St|dfdd }|SdS) Ncs<||t||x|D]}||q"WdS)N)r)r?rrRr)formatrUlocal_struct_packrVr%rrrW s zF_StructPackEncoder..SpecificEncoder..EncodePackedFieldcs(x"|D]}|||qWdS)Nr)r?rr@r)r^r_rVrrrXs zH_StructPackEncoder..SpecificEncoder..EncodeRepeatedFieldcs|||S)Nr)r?rr@)r^r_rVrrrYsz@_StructPackEncoder..SpecificEncoder..EncodeField)N)N)structpackrQrrZrJ)rrr rWrXrY)r^r%rP)rUr_rVrr[s   z+_StructPackEncoder..SpecificEncoder)r`calcsize)rPr^r[r)r^r%rPr_StructPackEncoders rccsPtdkrddndkr.ddn tdfdd}|S) aqReturn a constructor for an encoder for float fields. This is like StructPackEncoder, but catches errors that may be due to passing non-finite floating-point values to struct.pack, and makes a second attempt to encode those values. Args: wire_type: The field's wire type, for encoding tags. format: The format string to pass to struct.pack(). rcSs<|tkr|dn&|tkr$|dn||kr6|dndS)Nsss)_POS_INF_NEG_INF)r?rrrrEncodeNonFiniteOrRaise.s   z5_FloatingPointEncoder..EncodeNonFiniteOrRaiser cSs<|tkr|dn&|tkr$|dn||kr6|dndS)Nsss)rdre)r?rrrrrf9s   zGCan't encode floating-point values that are %d bytes long (only 4 or 8)cstj|r4t|tjtfdd}|S|rZt|dfdd }|St|dfdd }|SdS) Nc s`||t||x>|D]6}y||Wq"tk rV||Yq"Xq"WdS)N)r SystemError)r?rrRr)rfr^rUr_rVr%rrrWKs zI_FloatingPointEncoder..SpecificEncoder..EncodePackedFieldc sLxF|D]>}|y||Wqtk rB||YqXqWdS)N)rg)r?rr@r)rfr^r_rVrrrXXs  zK_FloatingPointEncoder..SpecificEncoder..EncodeRepeatedFieldcs>|y||Wntk r8||YnXdS)N)rg)r?rr@)rfr^r_rVrrrYbs zC_FloatingPointEncoder..SpecificEncoder..EncodeField)N)N)r`rarQrrZrJ)rrr rWrXrY)rfr^r%rP)rUr_rVrr[Fs    z._FloatingPointEncoder..SpecificEncoder)r`rb ValueError)rPr^r[r)rfr^r%rPr_FloatingPointEncoder s   $riz.EncodePackedFieldNcs0x*|D]"}||r |q|qWdS)Nr)r?rr@r)rlrVrmrrrXs   z(BoolEncoder..EncodeRepeatedFieldcs||r|S|S)Nr)r?rr@)rlrVrmrrrYsz BoolEncoder..EncodeField)N)N)rQrrZrJWIRETYPE_VARINT)rrr rWrXrYr)rlrUrVrmr BoolEncoders   rocsLt|tjtt|rt|r4fdd}|Sfdd}|SdS)z&Returns an encoder for a string field.cs<x6|D].}|d}||||||qWdS)Nzutf-8)r))r?rrRrencoded)rUr+tagrrrXs   z*StringEncoder..EncodeRepeatedFieldcs*|d}||||||S)Nzutf-8)r))r?rrRrp)rUr+rqrrrYs z"StringEncoder..EncodeFieldN)rQrrZrJrr,)rrr rXrYr)rUr+rqr StringEncoders rrcsLt|tjtt|rt|r4fdd}|Sfdd}|SdS)z%Returns an encoder for a bytes field.cs2x,|D]$}||||||qWdS)Nr)r?rrRr)rUr+rqrrrXs z)BytesEncoder..EncodeRepeatedFieldcs ||||||S)Nr)r?rrR)rUr+rqrrrYsz!BytesEncoder..EncodeFieldN)rQrrZrJrr,)rrr rXrYr)rUr+rqr BytesEncoders rscsLt|tjt|tj|r t|r6fdd}|Sfdd}|SdS)z%Returns an encoder for a group field.cs.x(|D] }|||||qWdS)N)_InternalSerialize)r?rrRr)end_tag start_tagrrrXs  z)GroupEncoder..EncodeRepeatedFieldcs|||||S)N)rt)r?rrR)rurvrrrYs z!GroupEncoder..EncodeFieldN)rQrWIRETYPE_START_GROUPWIRETYPE_END_GROUPr,)rrr rXrYr)rurvr GroupEncoders  rycsDt|tjt|rt|r.fdd}|Sfdd}|SdS)z'Returns an encoder for a message field.cs6x0|D](}|||||||qWdS)N)r/rt)r?rrRr)rUrqrrrXs z+MessageEncoder..EncodeRepeatedFieldcs$|||||||S)N)r/rt)r?rrR)rUrqrrrYsz#MessageEncoder..EncodeFieldN)rQrrZrJr,)rrr rXrYr)rUrqrMessageEncoders rzcsRdtdtjtdtjt|tdtjgtdtjtfdd}|S)zEncoder for extensions of MessageSet. The message set message looks like this: message MessageSet { repeated group Item = 1 { required int32 type_id = 2; required string message = 3; } } rIrrrcs,||||||||S)N)r/rt)r?rrR) end_bytesrU start_bytesrrrYs z*MessageSetItemEncoder..EncodeField) rLrQrrwrnrNrZrxrJ)rrYr)r{rUr|rMessageSetItemEncoder s    r}cs&|jt|jddfdd}|S)zEncoder for extensions of MessageSet. Maps always have a wire format like this: message MapEntry { key_type key = 1; value_type value = 2; } repeated MapEntry map = N; FcsD|rt|n|}x*|D]"}j|||d}|||qWdS)N)r4r)sortedkeysr5)r?rrRZ value_keysr4r7)encode_messager:rrrY8s zMapEncoder..EncodeField)r:rzr;)r<rYr)rr:r MapEncoder)s r)D__doc__ __author__r`rDZgoogle.protobuf.internalrrdrerrrr"r$r(Z Int32SizerZ Int64SizerZ EnumSizerZ UInt32SizerZ UInt64SizerZ ZigZagEncodeZ SInt32SizerZ SInt64SizerZ Fixed32SizerZ SFixed32SizerZ FloatSizerZ Fixed64SizerZ SFixed64SizerZ DoubleSizerZ BoolSizerr-r.r0r1r3r=rFrHrJZ_EncodeSignedVarintrNrQr\r]rcrirnZ Int32EncoderZ Int64EncoderZ EnumEncoderZ UInt32EncoderZ UInt64EncoderZ SInt32EncoderZ SInt64EncoderZWIRETYPE_FIXED32ZFixed32EncoderZWIRETYPE_FIXED64ZFixed64EncoderZSFixed32EncoderZSFixed64EncoderZ FloatEncoderZ DoubleEncoderrorrrsryrzr}rrrrrAsn !   )"&R       %