ó „ćRc@sÄddlZddlZddlZddlZddlZddlZddlZd„Zd„Zd„Z d„Z d„Z d„Z d„Z d „Zd „Zd „Zd „Zd „ZdS(i’’’’NcCsŠtƒ}t|dƒ}d}x[|D]S}|j|ƒ}|dkre|dkrXq%n|| }n|j|jƒƒq%W|jƒ|S(sŚ Reads in links that are considered safe Given a filename, reads in links, expecting them to be new-line separated. Ignores any content on a line after a '#' sign to allow any comments that might be added. trt#i’’’’i(tsettopentfindtaddtstriptclose(tfilenamet safe_linkst safe_filet comment_signtlinetcpos((sH/vol/project/2012/wmproject2013/chandra/web-project/Crawler/functions.pytread_safe_linkss      cCsætƒ}t|dƒ}d}x|D]ˆ}|j|ƒ}|dkre|dkrXq%n|| }n|jƒ}|jdƒ r |jdƒ r d|GHq%n|j|ƒq%W|jƒ|S(s? Reads in link prefixes that should also be Given a filename, reads in prefixes, starting with either 'www' or 'http', such that any page found during crawling containing the given prefix will also be included in the check, rather than skipped if not having the same prefix as the initial starting page. RRi’’’’ithttptwwws%Ignoring malformed extra prefix %s...(RRRRt startswithRR(Rt extra_preft extra_fileR R R ((sH/vol/project/2012/wmproject2013/chandra/web-project/Crawler/functions.pytread_extra_prefixes$s          cCsmxf|jdƒdkrh|jdƒ}|jd|ƒ}|dkrItS|| ||tdƒ}qW|S(sō Removes certain code from a HTML source Given a HTML source, it clear certain code that is of no interest, i.e. comments, and returns that string. Can be extended to exclude any code that should not be parsed for URLs, as desired. s(RtFalsetlen(tsourcetpostepos((sH/vol/project/2012/wmproject2013/chandra/web-project/Crawler/functions.pyt clear_source=s  cCs¢|jdƒ}|dkrtS|jd|ƒ}|jd|ƒ}|dkrš||krš|tdƒ7}||}||d|j||dƒ!}|StSdS(sRetrieves the base tag if foundsthrefshref=iN(RRR(RRRtupostdcharturl((sH/vol/project/2012/wmproject2013/chandra/web-project/Crawler/functions.pyt find_baseOs  !cCs‘g}x„tjd|ƒD]p}|jd|jƒƒ}|jd|jƒƒ|kr|jd|jƒƒdkr|j|jƒƒqqW|S(sa Finds anchor positions in source string Given a HTML source, returns a list of positions in the given source where anchor tags with a valid href have been identified. The way in which this is done is simply finding a "" symbol (meaning the href has to be set for this particular instance of an anchor) and returns the string contained within the two following instances of either single or double quotes. RRshref="it't"t sVs cSs#i|]\}}|d|“qS(i((RƒR„R…((sH/vol/project/2012/wmproject2013/chandra/web-project/Crawler/functions.pys Ys ii”ii(R=R>R?tMODE_SERVER_VERIFIEDRwt MODE_STRICT(t err_linkst strict_lnksRRbRh((sH/vol/project/2012/wmproject2013/chandra/web-project/Crawler/functions.pyt parse_errorsRs) %c Cs‰t|dƒdks>t|dƒdkr…tjtjkr…d|}t|dƒdkr£|d7}x8|djƒD]#\}}tjtjkr|dkr{|d||tj|f7}|d||d||jd |d|ƒd!d 7}q{n{ytj|}Wnt k r*d }nX|d|||f7}|d||d||jd |d|ƒd!d 7}||s’|d 7}n|d 7}qyWntjtjkrgt|dƒdkrg|d7}x|djƒD]z\} } |d| | f7}|d||d| |jd |d| ƒd!d 7}|| sV|d 7}n|d 7}qęWn|dd 7}|j |ƒndS(sč Records broken URL information in a file This function is to be called on one page at a time, giving its URL and source, a file object where to write the information and a tuple of HTTP and URL errors. It records all links that issued errors when checking and are according the mode set from the command line (more information can be found in crawler.py regarding this variable), as well as the corresponding anchor tag within the source, allowing for fast searching. iis@On page at %s, the following broken links have been identified: s"[Links broken due to HTTP errors] i”s %s: status %s: %s s [ iRs ] sUnknown status retrieveds [ EMPTY anchor detected ] s s![Links broken due to URL errors] s %s: reason %s RpiNs--------------------( RR=R>R?t iteritemsR‡R_t responsesRtKeyErrortwrite( terrlnksRRtwfileRBtmessagethttpurlRhtdescturlurlRi((sH/vol/project/2012/wmproject2013/chandra/web-project/Crawler/functions.pyt record_pagebs8 >   ?  9  ( 9  (R!RPRZR\R_R=RuRRRR R(R+R7RLRoR‚RŠR•(((sH/vol/project/2012/wmproject2013/chandra/web-project/Crawler/functions.pyts$            7 / 7 7