Has a list of probers.
Mostly abstract parent class.
Runs a bunch of related probers at the same time and decides which is best.
SBCS = Single-ByteCharSet. Runs a bunch of SingleByteCharSetProbers. Always contains the same SingleByteCharSetProbers.
A CharSetProber that is used for detecting single-byte encodings by using a "precedence matrix" (i.e., a character bigram model).
Runs a bunch of MultiByteCharSetProbers. It also uses a UTF8Prober, which is essentially a MultiByteCharSetProber that only has a state machine. Always contains the same MultiByteCharSetProbers.
A CharSetProber that uses both a character unigram model (or "character distribution analysis") and an independent state machine for trying to detect and encoding.
Used for "coding scheme" detection, where we just look for either invalid byte sequences or sequences that only occur for that particular encoding.
Used for character unigram distribution encoding detection. Takes a mapping from characters to a "frequency order" (i.e., what frequency rank that byte has in the given encoding) and a "typical distribution ratio", which is the number of occurrences of the 512 most frequently used characters divided by the number of occurrences of the rest of the characters for a typical document. The "characters" in this case are 2-byte sequences and they are first converted to an "order" (name comes from ord() function, I believe). This "order" is used to index into the frequency order table to determine the frequency rank of that byte sequence. The reason this extra step is necessary is that the frequency rank table is language-specific (and not encoding-specific).
hebrewprober.py
jpcntxprober.py
langbulgarianmodel.py
langcyrillicmodel.py
langgreekmodel.py
langhebrewmodel.py
langhungarianmodel.py
langthaimodel.py
latin1prober.py
sbcharsetprober.py
sbcsgroupprober.py
escprober.py
escsm.py
utf8prober.py
codingstatemachine.py
mbcssmprober.py
big5freqprober.py
chardistribution.py
euckrfreqprober.py
euctwfreqprober.py
gb2312freqprober.py
jisfreqprober.py
big5prober.py
cp949prober.py
eucjpprober.py
euckrprober.py
euctwprober.py
gb2312prober.py
mbcharsetprober.py
mbcsgroupprober.py
sjisprober.py
__init__.py
(currently has detect
function in it)compat.py
enums.py
universaldetector.py
version.py
This is just a collection of information that I've found useful or thought might be useful in the future:
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。