ó —Àv]c@@s˜dZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z d„Z d „ZdS( sR This is a script to submit dmlc job via Yarn dmlc will run as a Yarn application i(tabsolute_importN(tThreadi(topts(ttrackerc@sÅtjdk}tjdƒ}|d>k s6tdƒ‚tjj|ddƒ}tjj|ƒsltdƒ‚|jd>krÔ|j dkršd|j }nd |j |j f}||j dj d ƒd |_ntjj|j d ƒ}tjjtjjtjjtƒƒƒ} tjj| d ƒ} tjj|ƒsÍtjd|ƒdtjjtjjtƒtjdƒ|rdndf‰ˆGHtjˆdtdtjƒtjj|ƒsÍtdƒ‚ntjd|dtdtjƒjƒ\} } | j dƒdj ƒ} | ddks0tdƒ‚t| dj dƒdƒ} tjd|dtdtjƒjƒ\}} |jƒ}| dkrªtd| dƒ‚ntj |ƒ\}}|j!|ƒ|j!| ƒg}x7|j"D],}|j!|ƒ|j#tjj$|ƒƒqéWtjd ƒ}|d>kr=d!}ntjj|dd!ƒ}d"|||rjd#nd$|f‰tjj%ƒ‰x*|j&ƒD]\}}t'|ƒˆ|k r6t)j*ƒdd%krè|j(d&}n |j(d'}|j!|ƒd(ˆkrˆd(nd)}|j(d$|ˆd(Wˆd8jd9g|ƒ7‰t3j4d:||ƒ‡‡fd;†}t5d<|d=d?ƒ}|j6tƒ|j7ƒ|S(@sSubmission function for YARN.tntt HADOOP_HOMEs,Need to set HADOOP_HOME for YARN submission.tbinthadoops.HADOOP_HOME does not contain the hadoop binaryisDMLC[nworker=%d]:sDMLC[nworker=%d,nsever=%d]:t/iÿÿÿÿs dmlc-yarn.jars launcher.pys)cannot find "%s", I will try to run buildscd %s;./build.%styarntbattshtshelltenvs.failed to build dmlc-yarn.jar, try it manuallys %s versiontstdouts tHadoops"cannot parse hadoop version stringit.s %s classpathis:Hadoop Version is %s, dmlc_yarn will need Yarn(Hadoop 2.0)t JAVA_HOMEtjavas1%s -cp %s%s%s org.apache.hadoop.yarn.dmlc.Client t;t:t64bits/libstdc++.so.6s /libstdc++.sotLD_LIBRARY_PATHttDMLC_JOB_CLUSTERtDMLC_WORKER_COREStDMLC_WORKER_MEMORY_MBtDMLC_SERVER_COREStDMLC_SERVER_MEMORY_MBtDMLC_NUM_WORKERtDMLC_NUM_SERVERtDMLC_JOB_ARCHIVESs -file %ss -jobname %s s -tempdir %s s -queue %s s -appcp %s s -env %s t s ./launcher.pys)Submit job with %d workers and %d serversc@s*tjˆƒtjˆdtdˆƒdS(sinternal running function.R R N(tloggingtdebugt subprocesst check_calltTrue((tcmdR (s2/tmp/pip-install-Qvdv_2/mxnet/dmlc_tracker/yarn.pytrunns ttargettargsN((8tostnametgetenvtNonetAssertionErrortpathtjointexiststjobnamet num_serverst num_workerstcommandtsplitt yarn_app_dirtdirnametabspatht expandusert__file__twarningstwarntpardirR#R$R%tenvirontPopentPIPEt communicatetinttstript RuntimeErrorRtget_cache_file_settaddtarchivestappendtbasenametcopytitemststrt ship_libcxxtplatformt architecturet worker_corestworker_memory_mbt server_corestserver_memory_mbt hdfs_tempdirtqueuetyarn_app_classpathR R!R"Rt setDaemontstart(R)tnworkertnservertpass_envt is_windowst hadoop_homet hadoop_binarytprefixt YARN_JAR_PATHt curr_patht YARN_BOOT_PYtoutt_thadoop_versiont classpathtfsett new_commandtar_listtfnameRtJAVAtktvtlibcxxRtftentryR'tthread((R&R s2/tmp/pip-install-Qvdv_2/mxnet/dmlc_tracker/yarn.pyt yarn_submitsœ$*$!              c @sg‰‡‡fd†}tjjtjjtjjtƒƒƒ}tjj|dƒ}tjˆj ˆj d|ddj|gˆj ƒƒdS(Nc@s ˆjtˆ|||ƒƒdS(N(RIRs(RZR[R\(R)t submit_thread(s2/tmp/pip-install-Qvdv_2/mxnet/dmlc_tracker/yarn.pytyarn_submit_passzss launcher.pyt fun_submittpscmdR ( R*R/R8R9R:R;R0RtsubmitR4R3R5(R)RuRbRc((R)Rts2/tmp/pip-install-Qvdv_2/mxnet/dmlc_tracker/yarn.pyRxxs*(t__doc__t __future__RR*R#R<R!ROt threadingRRRRRsRx(((s2/tmp/pip-install-Qvdv_2/mxnet/dmlc_tracker/yarn.pyts      h