Name	Size	Mode	Actions
big5freq.cpython-39.opt-1.pyc	27130	0644	editdlrm
big5freq.cpython-39.pyc	27130	0644	editdlrm
big5prober.cpython-39.opt-1.pyc	1085	0644	editdlrm
big5prober.cpython-39.pyc	1085	0644	editdlrm
chardistribution.cpython-39.opt-1.pyc	6171	0644	editdlrm
chardistribution.cpython-39.pyc	6171	0644	editdlrm
charsetgroupprober.cpython-39.opt-1.pyc	2212	0644	editdlrm
charsetgroupprober.cpython-39.pyc	2212	0644	editdlrm
charsetprober.cpython-39.opt-1.pyc	3434	0644	editdlrm
charsetprober.cpython-39.pyc	3434	0644	editdlrm
codingstatemachine.cpython-39.opt-1.pyc	2861	0644	editdlrm
codingstatemachine.cpython-39.pyc	2861	0644	editdlrm
compat.cpython-39.opt-1.pyc	350	0644	editdlrm
compat.cpython-39.pyc	350	0644	editdlrm
cp949prober.cpython-39.opt-1.pyc	1092	0644	editdlrm
cp949prober.cpython-39.pyc	1092	0644	editdlrm
enums.cpython-39.opt-1.pyc	2599	0644	editdlrm
enums.cpython-39.pyc	2599	0644	editdlrm
escprober.cpython-39.opt-1.pyc	2584	0644	editdlrm
escprober.cpython-39.pyc	2584	0644	editdlrm
escsm.cpython-39.opt-1.pyc	7033	0644	editdlrm
escsm.cpython-39.pyc	7033	0644	editdlrm
eucjpprober.cpython-39.opt-1.pyc	2398	0644	editdlrm
eucjpprober.cpython-39.pyc	2398	0644	editdlrm
euckrfreq.cpython-39.opt-1.pyc	12014	0644	editdlrm
euckrfreq.cpython-39.pyc	12014	0644	editdlrm
euckrprober.cpython-39.opt-1.pyc	1093	0644	editdlrm
euckrprober.cpython-39.pyc	1093	0644	editdlrm
euctwfreq.cpython-39.opt-1.pyc	27134	0644	editdlrm
euctwfreq.cpython-39.pyc	27134	0644	editdlrm
euctwprober.cpython-39.opt-1.pyc	1093	0644	editdlrm
euctwprober.cpython-39.pyc	1093	0644	editdlrm
gb2312freq.cpython-39.opt-1.pyc	19058	0644	editdlrm
gb2312freq.cpython-39.pyc	19058	0644	editdlrm
gb2312prober.cpython-39.opt-1.pyc	1101	0644	editdlrm
gb2312prober.cpython-39.pyc	1101	0644	editdlrm
hebrewprober.cpython-39.opt-1.pyc	2970	0644	editdlrm
hebrewprober.cpython-39.pyc	2970	0644	editdlrm
jisfreq.cpython-39.opt-1.pyc	22086	0644	editdlrm
jisfreq.cpython-39.pyc	22086	0644	editdlrm
jpcntx.cpython-39.opt-1.pyc	37559	0644	editdlrm
jpcntx.cpython-39.pyc	37559	0644	editdlrm
langbulgarianmodel.cpython-39.opt-1.pyc	21749	0644	editdlrm
langbulgarianmodel.cpython-39.pyc	21749	0644	editdlrm
langgreekmodel.cpython-39.opt-1.pyc	20425	0644	editdlrm
langgreekmodel.cpython-39.pyc	20425	0644	editdlrm
langhebrewmodel.cpython-39.opt-1.pyc	20493	0644	editdlrm
langhebrewmodel.cpython-39.pyc	20493	0644	editdlrm
langhungarianmodel.cpython-39.opt-1.pyc	21694	0644	editdlrm
langhungarianmodel.cpython-39.pyc	21694	0644	editdlrm
langrussianmodel.cpython-39.opt-1.pyc	26297	0644	editdlrm
langrussianmodel.cpython-39.pyc	26297	0644	editdlrm
langthaimodel.cpython-39.opt-1.pyc	20669	0644	editdlrm
langthaimodel.cpython-39.pyc	20669	0644	editdlrm
langturkishmodel.cpython-39.opt-1.pyc	20509	0644	editdlrm
langturkishmodel.cpython-39.pyc	20509	0644	editdlrm
latin1prober.cpython-39.opt-1.pyc	2906	0644	editdlrm
latin1prober.cpython-39.pyc	2906	0644	editdlrm
mbcharsetprober.cpython-39.opt-1.pyc	2213	0644	editdlrm
mbcharsetprober.cpython-39.pyc	2213	0644	editdlrm
mbcsgroupprober.cpython-39.opt-1.pyc	1082	0644	editdlrm
mbcsgroupprober.cpython-39.pyc	1082	0644	editdlrm
mbcssm.cpython-39.opt-1.pyc	15669	0644	editdlrm
mbcssm.cpython-39.pyc	15669	0644	editdlrm
sbcharsetprober.cpython-39.opt-1.pyc	3066	0644	editdlrm
sbcharsetprober.cpython-39.pyc	3066	0644	editdlrm
sbcsgroupprober.cpython-39.opt-1.pyc	1651	0644	editdlrm
sbcsgroupprober.cpython-39.pyc	1651	0644	editdlrm
sjisprober.cpython-39.opt-1.pyc	2434	0644	editdlrm
sjisprober.cpython-39.pyc	2434	0644	editdlrm
universaldetector.cpython-39.opt-1.pyc	5782	0644	editdlrm
universaldetector.cpython-39.pyc	5782	0644	editdlrm
utf8prober.cpython-39.opt-1.pyc	1943	0644	editdlrm
utf8prober.cpython-39.pyc	1943	0644	editdlrm
version.cpython-39.opt-1.pyc	390	0644	editdlrm
version.cpython-39.pyc	390	0644	editdlrm
__init__.cpython-39.opt-1.pyc	1851	0644	editdlrm
__init__.cpython-39.pyc	1851	0644	editdlrm

a ��=b�0�@s�dZd
dl
Z
d
dlZd
dlZddlmZ
ddlmZmZm Z
ddl mZ
ddlm Z
ddlmZ
dd lmZ
Gd d�de�ZdS)a
Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco �N�
)
�CharSetGroupProber)� InputState�LanguageFilter�ProbingState)
�EscCharSetProber)
�Latin1Prober)
�MBCSGroupProber)
�SBCSGroupProberc @sneZ
dZd
ZdZe�d�
Ze�d�
Ze�d�
Z dddd d ddd d�Z ejf
dd�
Z dd�Zdd�Zdd�ZdS)�UniversalDetectoraq
The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result g�������?s[�-�]s(|~{)s[�-�]zWindows-1252zWindows-1250zWindows-1251zWindows-1256zWindows-1253zWindows-1255zWindows-1254zWindows-1257)z iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8z iso-8859-9ziso-8859-13cCsNd|_g|_
d|_d|_d|_d|_d|_|
|_t� t �
|_d|_|� �
dS)
N)�_esc_charset_prober�_charset_probers�result�done� _got_data�_input_state� _last_char�lang_filter�loggingZ getLogger�__name__�logger�_has_win_bytes�reset)�selfr�r�=/usr/lib/python3.9/site-packages/chardet/universaldetector.py�__init__Qs










zUniversalDetector.__init__c
CsVd
dd
d�|_d|_
d|_d|_tj|_d|_|jr>|j� �
|j D]}
|
� �
qDd
S)z� Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. N���encoding� confidence�languageF�)rrrrr� PURE_ASCIIrrrrr )r�proberrrrr^s








zUniversalDetector.resetcCs>|jr d
St
|
�
sd
St|
t�s(t|
�
}
|js�|
�tj�
rJdddd�|_nv|
�tj tj f�
rldddd�|_nT|
�d�
r�dddd�|_n:|
�d �
r�d ddd�|_n |
�tjtjf�
r�dddd�|_d|_|jd d
u
r�d|_d
S|j tjk�
r.|j�|
�
�
rtj|_ n*|j tjk�
r.|j�|j|
�
�
r.tj|_ |
dd
�|_|j tjk�
r�|j�
s^t|j�
|_|j�|
�
tjk�r:|jj|j��|jjd�|_d|_n�|j tjk�r:|j�
s�t |j�
g
|_|jt!j"@�
r�|j�#t$��

|j�#t%��

|jD]:}|�|
�
tjk�
r�|j|��|jd�|_d|_
�q&�
q�|j&�|
�
�r:d|_'d
S)a�
Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIG��?�rzUTF-32s��zX-ISO-10646-UCS-4-3412s��zX-ISO-10646-UCS-4-2143zUTF-16Tr�����)(r�len� isinstance� bytearrayr� startswith�codecs�BOM_UTF8r�BOM_UTF32_LE�BOM_UTF32_BE�BOM_LE�BOM_BErrr#�HIGH_BYTE_DETECTOR�search� HIGH_BYTE�ESC_DETECTORrZ ESC_ASCIIrrr�feedrZFOUND_IT�charset_name�get_confidencer!r r rZNON_CJK�appendr r�WIN_BYTE_DETECTORr)rZbyte_strr$rrrr6os�




� 
�

�  

�  

� 

�






�


�








�


zUniversalDetector.feedc
Cst
|jr|j
Sd
|_|js&|j�d�

n�|jtjkrBdddd�|_
n�|jtjkr�d}
d}d}|j D]"}|sjq`|� �}
|
|kr`|
}|}q`|r�||jkr�|j}|j� �}|� �}|�d �
r�|jr�|j�||�}|||jd�|_
|j��tjk
�
rn|j
d du�
rn|j�d�

|j D]`}|�
s�
qt|t��
rP|jD] }|j�d|j|j|� ��
�
q,n|j�d|j|j|� ��
�
q|j
S) z� Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!�asciir%r&rNrziso-8859rz no probers hit minimum thresholdz%s %s confidence = %s)rrrr�debugrrr#r4r r8�MINIMUM_THRESHOLDr7�lowerr+r�ISO_WIN_MAP�getr!ZgetEffectiveLevelr�DEBUGr)rZprobers) rZprober_confidenceZmax_prober_confidenceZ max_proberr$r7Zlower_charset_namer Zgroup_proberrrr�close�sj 





� 
















�

�










� 


�zUniversalDetector.closeN)r� __module__�__qualname__�__doc__r=�re�compiler2r5r:r?rZALLrrr6rBrrrrr3s$











�  mr)rEr,rrFZcharsetgroupproberrZenumsrrrZ escproberrZlatin1proberrZmbcsgroupproberr Zsbcsgroupproberr �objectrrrrr�s