Nexus File Manager
v2.0
🏠
Dashboard
⚡
System
🌐
›
opt
›
alt
›
python38
›
lib64
›
python3.8
›
test
›
__pycache__
Quick:
⬆️ Parent
🌐 Root
🏠 Home
🌍 WWW
📁 Temp
⚙️ Etc
📤 Upload
📁 Create
⚡ WordPress Admin
🔄 Refresh
✏️ test_robotparser.cpython-38.opt-2.pyc
← Back
U i�f�* � @ s d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlm Z m Z G dd� d�Z G dd� de ej �Z G dd � d e ej �ZG d d � d e ej �ZG d d � d e ej �ZG dd� de �ZG dd� deej �ZG dd� deej �ZG dd� de�ZG dd� de ej �ZG dd� de ej �ZG dd� de ej �ZG d d � d e ej �ZG d d � d e�ZG d d!� d!e ej �ZG d"d#� d#e ej �ZG d$d%� d%e ej �Z G d&d'� d'e ej �Z G d(d)� d)eej �Z G d*d+� d+e ej �Z G d,d-� d-e �Z G d.d/� d/ej �Z!G d0d1� d1ej �Z"e#d2k�re�$� dS )3� N)�support)�BaseHTTPRequestHandler� HTTPServerc @ sH e Zd ZdZdZg Zg ZdZdd� Zdd� Z dd � Z d d � Z d d � Z dS )� BaseRobotTest� Ztest_robotparserNc C s, t �| j��� }tj�� | _| j�|� d S �N) �io�StringIO� robots_txt� readlines�urllib� robotparser�RobotFileParser�parser�parse)�self�lines� r �:/opt/alt/python38/lib64/python3.8/test/test_robotparser.py�setUp s zBaseRobotTest.setUpc C s$ t |t�r|\}}||fS | j|fS r )� isinstance�tuple�agent�r �urlr r r r �get_agent_and_url s z BaseRobotTest.get_agent_and_urlc C sL | j D ]@}| �|�\}}| j||d�� | �| j�||�� W 5 Q R X qd S �N)r r )�goodr �subTest� assertTruer � can_fetchr r r r �test_good_urls s z BaseRobotTest.test_good_urlsc C sL | j D ]@}| �|�\}}| j||d�� | �| j�||�� W 5 Q R X qd S r )�badr r � assertFalser r r r r r � test_bad_urls" s zBaseRobotTest.test_bad_urlsc C s | � | j�� | j� d S r )� assertEqualr � site_maps�r r r r �test_site_maps( s z BaseRobotTest.test_site_maps) �__name__� __module__� __qualname__r r r r" r& r r r! r$ r( r r r r r s r c @ s" e Zd ZdZddgZdddgZdS )�UserAgentWildcardTestz�User-agent: * Disallow: /cyberworld/map/ # This is an infinite virtual URL space Disallow: /tmp/ # these will soon disappear Disallow: /foo.html �/� /test.html�/cyberworld/map/index.htmlz/tmp/xxx� /foo.htmlN�r) r* r+ r r r" r r r r r, , s r, c @ s e Zd ZdZdddgZdgZdS )� CrawlDelayAndCustomAgentTestz�# robots.txt for http://www.example.com/ User-agent: * Crawl-delay: 1 Request-rate: 3/15 Disallow: /cyberworld/map/ # This is an infinite virtual URL space # Cybermapper knows where to go. User-agent: cybermapper Disallow: r- r. )Z cybermapperr/ r/ Nr1 r r r r r2 7 s r2 c @ s& e Zd ZdZddgZdgZddgZdS )� SitemapTesta # robots.txt for http://www.example.com/ User-agent: * Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml Sitemap: http://www.google.com/hostednews/sitemap_index.xml Request-rate: 3/15 Disallow: /cyberworld/map/ # This is an infinite virtual URL space r- r. r/ z7http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xmlz2http://www.google.com/hostednews/sitemap_index.xmlN)r) r* r+ r r r" r& r r r r r3 H s �r3 c @ s e Zd ZdZg ZdddgZdS )�RejectAllRobotsTestz(# go away User-agent: * Disallow: / r/ r- �/tmp/Nr1 r r r r r4 Y s r4 c @ s e Zd ZdZdZdd� ZdS )�BaseRequestRateTestNc C s� | j }| j| j D ]�}| �|�\}}| j||d��p | �|�|�| j� |�|�}| �|| j� | jd k r�| �|t j j � | �|j | jj � | �|j | jj � W 5 Q R X qd S r )r r r" r r r% � crawl_delay� request_rateZassertIsInstancer r � RequestRateZrequestsZseconds)r r r r Zparsed_request_rater r r �test_request_rateg s( ���z%BaseRequestRateTest.test_request_rate)r) r* r+ r8 r7 r: r r r r r6 c s r6 c @ s e Zd ZdZdgZdS )� EmptyFileTestr z/fooN)r) r* r+ r r r r r r r; s r; c @ s> e Zd ZdZdZej�dd�ZdZ dgZ ddd d d d d gZ dS )� CrawlDelayAndRequestRateTestz�User-agent: figtree Crawl-delay: 3 Request-rate: 9/30 Disallow: /tmp Disallow: /a%3cd.html Disallow: /a%2fb.html Disallow: /%7ejoe/index.html �figtree� � � )r= r0 �/tmpz /tmp.html� /tmp/a.html� /a%3cd.html� /a%3Cd.htmlz /a%2fb.htmlz/~joe/index.htmlN) r) r* r+ r r r r r9 r8 r7 r r" r r r r r< � s �r< c @ s e Zd ZdZdS )�DifferentAgentTestz FigTree Robot libwww-perl/5.04N�r) r* r+ r r r r r rE � s rE c @ s* e Zd ZdZdgZddddddgZd Zd S ) �InvalidRequestRateTestz�User-agent: * Disallow: /tmp/ Disallow: /a%3Cd.html Disallow: /a/b.html Disallow: /%7ejoe/index.html Crawl-delay: 3 Request-rate: 9/banana rA r5 rB rC rD z /a/b.htmlz/%7Ejoe/index.htmlr@ N)r) r* r+ r r r" r7 r r r r rG � s �rG c @ s e Zd ZdZdgZg ZdS )�InvalidCrawlDelayTestz2User-Agent: * Disallow: /. Crawl-delay: pears r0 Nr1 r r r r rH � s rH c @ s e Zd ZdZdZdgZdgZdS )� AnotherInvalidRequestRateTestzeUser-agent: Googlebot Allow: /folder1/myfile.html Disallow: /folder1/ Request-rate: whale/banana � Googlebot�/folder1/myfile.html�/folder1/anotherfile.htmlN�r) r* r+ r r r r" r r r r rI � s rI c @ s e Zd ZdZdZdgZdS )�UserAgentOrderingTestzMUser-agent: Googlebot Disallow: / User-agent: Googlebot-Mobile Allow: / rJ z/something.jpgN)r) r* r+ r r r" r r r r rN � s rN c @ s e Zd ZdZdS )�UserAgentGoogleMobileTestzGooglebot-MobileNrF r r r r rO � s rO c @ s e Zd ZdZdZdgZdgZdS )�GoogleURLOrderingTestzJUser-agent: Googlebot Allow: /folder1/myfile.html Disallow: /folder1/ Z googlebotrK rL NrM r r r r rP � s rP c @ s e Zd ZdZdgZdgZdS )�DisallowQueryStringTestz2User-agent: * Disallow: /some/path?name=value � /some/pathz/some/path?name=valueNr1 r r r r rQ � s rQ c @ s e Zd ZdZdgZdgZdS )� UseFirstUserAgentWildcardTestzNUser-agent: * Disallow: /some/path User-agent: * Disallow: /another/path z /another/pathrR Nr1 r r r r rS � s rS c @ s e Zd ZdZdgZdgZdS )�EmptyQueryStringTestz>User-agent: * Allow: /some/path? Disallow: /another/path? z /some/path?z/another/path?Nr1 r r r r rT � s rT c @ s0 e Zd ZdZej�dd�ZdZddgZ dgZ dS ) �DefaultEntryTestzOUser-agent: * Crawl-delay: 1 Request-rate: 3/15 Disallow: /cyberworld/map/ r@ � � r- r. r/ N) r) r* r+ r r r r9 r8 r7 r r" r r r r rU s rU c @ s e Zd ZdZdZdd� ZdS )�StringFormattingTestz�User-agent: * Crawl-delay: 1 Request-rate: 3/15 Disallow: /cyberworld/map/ # This is an infinite virtual URL space # Cybermapper knows where to go. User-agent: cybermapper Disallow: /some/path zxUser-agent: cybermapper Disallow: /some/path User-agent: * Crawl-delay: 1 Request-rate: 3/15 Disallow: /cyberworld/map/c C s | � t| j�| j� d S r )r% �strr �expected_outputr' r r r �test_string_formatting( s z+StringFormattingTest.test_string_formattingN)r) r* r+ r rZ r[ r r r r rX s rX c @ s e Zd Zdd� Zdd� ZdS )� RobotHandlerc C s | � dd� d S )Ni� zForbidden access)Z send_errorr' r r r �do_GET. s zRobotHandler.do_GETc G s d S r r )r �format�argsr r r � log_message1 s zRobotHandler.log_messageN)r) r* r+ r] r` r r r r r\ , s r\ c @ s* e Zd Zdd� Zdd� Zejdd� �ZdS )� PasswordProtectedSiteTestCasec C sB t tjdft�| _tjd| jjddid�| _d| j_ | j� � d S )Nr zHTTPServer servingZ poll_intervalg{�G�z�?)�name�target�kwargsT) r r �HOSTr\ �server� threadingZThreadZ serve_forever�tZdaemon�startr' r r r r 7 s �z#PasswordProtectedSiteTestCase.setUpc C s" | j �� | j�� | j �� d S r )rf Zshutdownrh �joinZ server_closer' r r r �tearDownD s z&PasswordProtectedSiteTestCase.tearDownc C s\ | j j}dtj d t|d � }|d }tj�� }|�|� |� � | � |� d|�� d S )Nzhttp://�:rW z /robots.txt�*) rf Zserver_addressr re rY r r r Zset_url�readr# r )r Zaddrr Z robots_urlr r r r �testPasswordProtectedSiteI s z7PasswordProtectedSiteTestCase.testPasswordProtectedSiteN)r) r* r+ r rk r Z reap_threadsro r r r r ra 5 s ra c @ sF e Zd ZdZd�e�Zedd� �Zdd� Zdd� Z d d � Z d d � Z d S )�NetworkTestCasezhttp://www.pythontest.net/z{}elsewhere/robots.txtc C s@ t �d� t �| j�� tj�| j�| _| j� � W 5 Q R X d S )NZnetwork) r ZrequiresZtransient_internet�base_urlr r r r r rn )�clsr r r � setUpClassY s zNetworkTestCase.setUpClassc C s$ d� | j|tj�|�d s dnd�S )Nz{}{}{}rW r- r )r^ rq �os�path�splitext)r ru r r r r ` s �zNetworkTestCase.urlc C sV | � | jj� | � | jj� | �| j�� d� | � | j�d�� | � | j�d�� d S )Nr rm )r# r � disallow_all� allow_allZ assertGreater�mtimer7 r8 r' r r r � test_basice s zNetworkTestCase.test_basicc C s� | � | j�d| �d��� | �| j�d| j�� | �| j�d| �d��� | �| j�d| �d��� | �| j�d| �d��� | � | j�d| j�� d S )Nrm Z elsewhereZNutchZbrianZwebstats)r r r r r# rq r' r r r �test_can_fetchl s z NetworkTestCase.test_can_fetchc C sf t j�| �d��}|�� | �|j� | �|j� | � |� � d� | � |� d�� | � |� d�� d S )Nz i-robot.txtr rm )r r r r rn r rx r# rw r% ry Z assertIsNoner7 r8 )r r r r r � test_read_404t s z NetworkTestCase.test_read_404N) r) r* r+ rq r^ r � classmethodrs r rz r{ r| r r r r rp T s rp �__main__)%r rt rg ZunittestZurllib.robotparserr Ztestr Z http.serverr r r ZTestCaser, r2 r3 r4 r6 r; r< rE rG rH rI rN rO rP rQ rS rT rU rX r\ ra rp r) �mainr r r r �<module> s> " )
💻
System Info
Current path
/opt/alt/python38/lib64/python3.8/test/__pycache__
Contents
0 folders, 1461 files
Disk free
99.54 GB
PHP version
8.3.30
🚀
Quick Actions
📍 Script location
🌐 Web root
🗑️ Temp
📁 /opt/alt/python38/lib64/python3.8/test/__pycache__
⚡ Nexus File Manager • 1461 items