7ec@sTdgZddlZddlZddlZddlZddlZddlmZmZm Z ddl m Z m Z dZ dZdZejZdZd efd YZefed Zdefd YZd efdYZeZdefdYZdefdYZdefdYZdefdYZdS(tPooliN(tProcesst cpu_countt TimeoutError(tFinalizetdebugiiicCs t|S(N(tmap(targs((s,/usr/lib64/python2.7/multiprocessing/pool.pytmapstar@stMaybeEncodingErrorcBs)eZdZdZdZdZRS(sVWraps possible unpickleable errors, so they can be safely sent through the socket.cCsAt||_t||_tt|j|j|jdS(N(treprtexctvaluetsuperR t__init__(tselfR R ((s,/usr/lib64/python2.7/multiprocessing/pool.pyRKscCsd|j|jfS(Ns(Error sending result: '%s'. Reason: '%s'(R R (R((s,/usr/lib64/python2.7/multiprocessing/pool.pyt__str__Ps cCsdt|S(Ns(tstr(R((s,/usr/lib64/python2.7/multiprocessing/pool.pyt__repr__Ts(t__name__t __module__t__doc__RRR(((s,/usr/lib64/python2.7/multiprocessing/pool.pyR Gs  cCs|j}|j}t|dr>|jj|jjn|dk rW||nd}xI|dks~|r||kry |}Wn"ttfk rt dPnX|dkrt dPn|\} } } } } yt | | | f}Wnt k r}t |f}nXy|| | |fWnMt k r}t ||d}t d||| | t |ffnXd}} }} } } |d7}q`Wt d|dS(Nt_writeris)worker got EOFError or IOError -- exitingsworker got sentinel -- exitingis0Possible encoding error while sending result: %ssworker exiting after %d tasks(tputtgetthasattrRtcloset_readertNonetEOFErrortIOErrorRtTruet ExceptiontFalseR (tinqueuetoutqueuet initializertinitargstmaxtasksRRt completedttasktjobtitfuncRtkwdstresulttetwrapped((s,/usr/lib64/python2.7/multiprocessing/pool.pytworkerXs@     !    cBseZdZeZdddddZdZdZdZdZ didZ ddZ dd Z dd Z didd Zddd Zed ZedZedZedZdZdZdZdZedZedZRS(sH Class which supports an async version of the `apply()` builtin c Cs/|jtj|_i|_t|_||_||_||_|dkr|y t }Wq|t k rxd}q|Xn|dkrt dn|dk rt |d rtdn||_g|_|jtjdtjd|f|_t|j_t|j_|jjtjdtjd|j|j|j|j|jf|_t|j_t|j_|jjtjdtjd|j|j|jf|_ t|j _t|j _|j jt!||j"d|j|j#|j|j|j|j|j |jfdd|_$dS( Nis&Number of processes must be at least 1t__call__sinitializer must be a callablettargetRt exitpriorityi(%t _setup_queuestQueuet _taskqueuet_cachetRUNt_statet_maxtasksperchildt _initializert _initargsRRtNotImplementedErrort ValueErrorRt TypeErrort _processest_poolt_repopulate_poolt threadingtThreadRt_handle_workerst_worker_handlerRtdaemontstartt _handle_taskst _quick_putt _outqueuet _task_handlert_handle_resultst _quick_gett_result_handlerRt_terminate_poolt_inqueuet _terminate(Rt processesR$R%tmaxtasksperchild((s,/usr/lib64/python2.7/multiprocessing/pool.pyRsX                               cCswt}xjttt|jD]M}|j|}|jdk r"td||jt }|j|=q"q"W|S(sCleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. scleaning up worker %dN( R!treversedtrangetlenRAtexitcodeRRtjoinR(RtcleanedR*R0((s,/usr/lib64/python2.7/multiprocessing/pool.pyt_join_exited_workerss"  c Csxt|jt|jD]}|jdtd|j|j|j|j |j f}|jj ||j j dd|_ t|_|jtdqWdS(sBring the number of pool processes up to the specified number, for use after reaping workers which have exited. R2RRt PoolWorkers added workerN(RVR@RWRARR0RQRKR;R<R:tappendtnametreplaceRRGRHR(RR*tw((s,/usr/lib64/python2.7/multiprocessing/pool.pyRBs#   cCs|jr|jndS(sEClean up any exited workers and start replacements for them. N(R[RB(R((s,/usr/lib64/python2.7/multiprocessing/pool.pyt_maintain_pools cCsPddlm}||_||_|jjj|_|jjj|_ dS(Ni(t SimpleQueue( tqueuesRbRQRKRtsendRJRtrecvRN(RRb((s,/usr/lib64/python2.7/multiprocessing/pool.pyR4s   cCs|j|||jS(s1 Equivalent of `apply()` builtin (t apply_asyncR(RR+RR,((s,/usr/lib64/python2.7/multiprocessing/pool.pytapplyscCs|j|||jS(s/ Equivalent of `map()` builtin (t map_asyncR(RR+titerablet chunksize((s,/usr/lib64/python2.7/multiprocessing/pool.pyRsics|dkrQt|j|jjfdt|DjfStj||}t|j|jjfdt|DjfdDSdS(sZ Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()` ic3s0|]&\}}j||fifVqdS(N(t_job(t.0R*tx(R+R-(s,/usr/lib64/python2.7/multiprocessing/pool.pys sc3s0|]&\}}j|t|fifVqdS(N(RkR(RlR*Rm(R-(s,/usr/lib64/python2.7/multiprocessing/pool.pys scss"|]}|D] }|Vq qdS(N((Rltchunktitem((s,/usr/lib64/python2.7/multiprocessing/pool.pys sN(t IMapIteratorR7R6Rt enumeratet _set_lengthRt _get_tasks(RR+RiRjt task_batches((R+R-s,/usr/lib64/python2.7/multiprocessing/pool.pytimaps cs|dkrQt|j|jjfdt|DjfStj||}t|j|jjfdt|DjfdDSdS(sK Like `imap()` method but ordering of results is arbitrary ic3s0|]&\}}j||fifVqdS(N(Rk(RlR*Rm(R+R-(s,/usr/lib64/python2.7/multiprocessing/pool.pys sc3s0|]&\}}j|t|fifVqdS(N(RkR(RlR*Rm(R-(s,/usr/lib64/python2.7/multiprocessing/pool.pys scss"|]}|D] }|Vq qdS(N((RlRnRo((s,/usr/lib64/python2.7/multiprocessing/pool.pys !sN(tIMapUnorderedIteratorR7R6RRqRrRRs(RR+RiRjRt((R+R-s,/usr/lib64/python2.7/multiprocessing/pool.pytimap_unordereds cCsAt|j|}|jj|jd|||fgdf|S(s> Asynchronous equivalent of `apply()` builtin N(t ApplyResultR7R6RRkR(RR+RR,tcallbackR-((s,/usr/lib64/python2.7/multiprocessing/pool.pyRf#s+cst|dst|}n|dkrhtt|t|jd\}}|rh|d7}qhnt|dkrd}ntj|||}t|j |t|||j j fdt |DdfS(s< Asynchronous equivalent of `map()` builtin t__len__iiic3s0|]&\}}j|t|fifVqdS(N(RkR(RlR*Rm(R-(s,/usr/lib64/python2.7/multiprocessing/pool.pys =sN( RtlistRtdivmodRWRARRst MapResultR7R6RRq(RR+RiRjRytextraRt((R-s,/usr/lib64/python2.7/multiprocessing/pool.pyRh,s ( cCsotj}xB|jtks6|jrP|jtkrP|jtjdqW|j j dt ddS(Ng?sworker handler exiting( RCtcurrent_threadR9R8R7t TERMINATERattimetsleepR6RRR(tpooltthread((s,/usr/lib64/python2.7/multiprocessing/pool.pyREAs  * cCs tj}xt|jdD]q\}}d}d} zGyxt|D]\} }|jrmtdPny||WqJtk r} |d \} } y|| j | t | fWqt k rqXqJXqJW|rtd|| dnwPWn}tk r|} |r|d nd \} } | |krX|| j | dt | fn|r}td|| dq}nXWdd}}} XqWtdy@td|j dtd x|D]}|dqWWnt k rtd nXtd dS( Nis'task handler found thread._state != RUNisdoing set_length()iistask handler got sentinels/task handler sending sentinel to result handlers(task handler sending sentinel to workerss/task handler got IOError when sending sentinelsstask handler exiting(ii(RCRtiterRRRqR9RR t_setR!tKeyErrorRR(t taskqueueRR#RtcacheRttaskseqt set_lengthR(R*R.R)tindtextp((s,/usr/lib64/python2.7/multiprocessing/pool.pyRINsR       !       cCstj}xy |}Wn"ttfk r@tddSX|jrXtdPn|dkrrtdPn|\}}}y||j||Wntk rnXd}}}qWx|rq|jt krqy |}Wn"ttfk rtddSX|dkr%tdqn|\}}}y||j||Wntk r_nXd}}}qWt |drtdy5x.t dD] }|j j sPn|qWWqttfk rqXntdt||jdS( Ns.result handler got EOFError/IOError -- exitings,result handler found thread._state=TERMINATEsresult handler got sentinels&result handler ignoring extra sentinelRs"ensuring that outqueue is not fulli s7result handler exiting: len(cache)=%s, thread._state=%s(RCRRRRR9RRRRRRVRtpollRW(R#RRRR(R)R*tobj((s,/usr/lib64/python2.7/multiprocessing/pool.pyRMsZ              ccsDt|}x1ttj||}|s1dS||fVqWdS(N(Rttuplet itertoolstislice(R+tittsizeRm((s,/usr/lib64/python2.7/multiprocessing/pool.pyRss  cCstddS(Ns:pool objects cannot be passed between processes or pickled(R=(R((s,/usr/lib64/python2.7/multiprocessing/pool.pyt __reduce__scCs5td|jtkr1t|_t|j_ndS(Ns closing pool(RR9R8tCLOSERF(R((s,/usr/lib64/python2.7/multiprocessing/pool.pyRs  cCs-tdt|_t|j_|jdS(Nsterminating pool(RRR9RFRR(R((s,/usr/lib64/python2.7/multiprocessing/pool.pyt terminates   cCsStd|jj|jj|jjx|jD]}|jq;WdS(Ns joining pool(RRFRYRLRORA(RR((s,/usr/lib64/python2.7/multiprocessing/pool.pyRYs     cCsWtd|jjx9|jrR|jjrR|jjtjdqWdS(Ns7removing tasks from inqueue until task handler finishedi( Rt_rlocktacquiretis_aliveRRReRR(R"t task_handlerR((s,/usr/lib64/python2.7/multiprocessing/pool.pyt_help_stuff_finishs    c Cstdt|_t|_td|j||t|t|_|jdtdtj|k r|j dn|rt |ddrtdx-|D]"} | j dkr| j qqWntdtj|k r|j dntd tj|k r,|j dn|rt |ddrtd x;|D]0} | j rVtd | j| j qVqVWndS( Nsfinalizing pools&helping task handler/workers to finishsjoining worker handlerg}Ô%ITiRsterminating workerssjoining task handlersjoining result handlersjoining pool workersscleaning up worker %d(RRR9RRWRRRCRRYRRXRRtpid( tclsRR"R#Rtworker_handlerRtresult_handlerRR((s,/usr/lib64/python2.7/multiprocessing/pool.pyRPs6              N((((RRRRRRR[RBRaR4RgRRuRwRfRht staticmethodRERIRMRsRRRRYRt classmethodRP(((s,/usr/lib64/python2.7/multiprocessing/pool.pyRs0  :         4<     RxcBsDeZdZdZdZddZddZdZRS(cCsStjtj|_tj|_||_t|_ ||_ |||js  N( RRRRRRRRR(((s,/usr/lib64/python2.7/multiprocessing/pool.pyRxs      R}cBseZdZdZRS(cCstj|||t|_dg||_||_|dkr`d|_t|_||j =n||t |||_dS(Ni( RxRRRRRt _chunksizet _number_leftRRktbool(RRRjtlengthRy((s,/usr/lib64/python2.7/multiprocessing/pool.pyRRs      cCs|\}}|r||j||j|d|j+|jd8_|jdkr|jrn|j|jn|j|j=|jjzt|_ |jj Wd|jj XqnWt |_ ||_|j|j=|jjzt|_ |jj Wd|jj XdS(Nii(RRRRR7RkRRRRRRR!R(RR*tsuccess_resulttsuccessR-((s,/usr/lib64/python2.7/multiprocessing/pool.pyR^s* "         (RRRR(((s,/usr/lib64/python2.7/multiprocessing/pool.pyR}Ps RpcBs>eZdZdZddZeZdZdZRS(cCsktjtj|_tj|_||_tj |_ d|_ d|_ i|_|||j#s,        ,.-I