U HI^@sdZddlZddlmZddlZddlZddlZddlZddlZddl Z ddl Z ej dkrle ee dedadZdZGdddZGd d d ZdS) z7 Module for ibus-table to access the sqlite3 databases N)rrutf-81.00u!“”‘’《》〈〉〔〕「」『』【】〖〗（）［］｛｝．。，、；：？！…—·ˉˇ¨々～‖∶＂＇｀｜⒈⒉⒊⒋⒌⒍⒎⒏⒐⒑⒒⒓⒔⒕⒖⒗⒘⒙⒚⒛АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁⅠⅡⅢⅣⅤⅥⅦⅧⅨⅩⅪⅫ⒈⒉⒊⒋⒌⒍⒎⒏⒐⒑⒒⒓⒔⒕⒖⒗⒘⒙⒚⒛㎎㎏㎜㎝㎞㎡㏄㏎㏑㏒㏕ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ⑴⑵⑶⑷⑸⑹⑺⑻⑼⑽⑾⑿⒀⒁⒂⒃⒄⒅⒆⒇€＄￠￡￥¤→↑←↓↖↗↘↙ァアィイゥウェエォオカガキギクグケゲコゴサザシジスズセゼソゾタダチヂッツヅテデトドナニヌネノハバパヒビピフブプヘベペホボポマミムメモャヤュユョヨラリルレロヮワヰヱヲンヴヵヶーヽヾぁあぃいぅうぇえぉおかがきぎぱくぐけげこごさざしじすずせぜそぞただちぢっつづてでとどなにぬねのはばひびぴふぶぷへべぺほぼぽまみむめもゃやゅゆょよらりるれろゎわゐゑをん゛゜ゝゞ勹灬冫艹屮辶刂匚阝廾丨虍彐卩钅冂冖宀疒肀丿攵凵犭亻彡饣礻扌氵纟亠囗忄讠衤廴尢夂丶āáǎàōóǒòêēéěèīíǐìǖǘǚǜüūúǔù＋－＜＝＞±×÷∈∏∑∕√∝∞∟∠∣∥∧∨∩∪∫∮∴∵∶∷∽≈≌≒≠≡≤≥≦≧≮≯⊕⊙⊥⊿℃°‰♂♀§№☆★○●◎◇◆□■△▲※〓＃＆＠＼＾＿￣абвгдежзийклмнопрстуфхцчшщъыьэюяёⅰⅱⅲⅳⅴⅵⅶⅷⅸⅹβγδεζηαικλμνξοπρστυφθψω①②③④⑤⑥⑦⑧⑨⑩①②③④⑤⑥⑦⑧⑨⑩㈠㈡㈢㈣㈤㈥㈦㈧㈨㈩㈠㈡㈢㈣㈤㈥㈦㈧㈨㈩ㄅㄆㄇㄈㄉㄊㄋㄌㄍㄎㄏㄐㄑㄒㄓㄔㄕㄖㄗㄘㄙㄧㄨㄩㄚㄛㄜㄝㄞㄟㄠㄡㄢㄣㄤㄥㄦc@s*eZdZdZd ddZddZddZdS) ImePropertiesz= A class to cache the properties of an input method. NcCsn|dkr i}|sdS||_d}z||}Wnddl}|YnX|D]}|d|j|d<qRdS)ul “db” is the handle of the sqlite3 database file obtained by sqlite3.connect(). NzSELECT attr, val FROM main.ime;r)ime_property_cacheexecutefetchall traceback print_exc)selfdbdefault_propertiessqlstrresultsr resultr+/usr/share/ibus-table/engine/tabsqlitedb.py__init__PszImeProperties.__init__cCs||jkr|j|SdS)z Return the value for a key from the property cache :param key: The key to lookup in the property cache :type key: String :rtype: String N)r)r keyrrrgetcs  zImeProperties.getcCsdt|jS)Nzime_property_cache = %s)reprrr rrr__str__oszImeProperties.__str__)NN)__name__ __module__ __qualname____doc__rrrrrrrrLs  rc@sveZdZdZd^ddZd_d d Zd d ZddZd`ddZddZ ddZ ddZ ddZ ddZ ddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zdad-d.Zdbd/d0Zd1d2Zdcd3d4Zd5d6Zd7d8Zddd9d:Zd;d<Zded?d@ZdfdAdBZdgdCdDZ dEdFZ!dGdHZ"dIdJZ#dKdLZ$dMdNZ%dOdPZ&dhdQdRZ'didSdTZ(djdUdVZ)dWdXZ*dkdYdZZ+dld\d]Z,dS)m TabSqliteDbuPhrase database for tables The phrases table in the database has columns with the names: “id”, “tabkeys”, “phrase”, “freq”, “user_freq” There are 2 databases, sysdb, userdb. sysdb: System database for the input method, for example something like /usr/share/ibus-table/tables/wubi-jidian86.db “user_freq” is always 0 in a system database. “freq” is some number in a system database indicating a frequency of use of that phrase relative to the other phrases in that database. user_db: Database on disk where the phrases used or defined by the user are stored. “user_freq” is a counter which counts how many times that combination of “tabkeys” and “phrase” has been used. “freq” is equal to 0 for all combinations of “tabkeys” and “phrase” where an entry for that phrase is already in the system database which starts with the same “tabkeys”. For combinations of “tabkeys” and “phrase” which do not exist at all in the system database, “freq” is equal to -1 to indidated that this is a user defined phrase. NFc#CszttdaWn ttfk r2tdaYnXg|_||_||_| |s`tj |jrpt |j|_ntd|jzX|jd|jd|jd|jd|jd|jd |jd Wn$ddl}|td YnX|jd d d d d ddtdtdddd d ddddddd dddddddddd d dd dddd"|_|rd }d!}t|jD]8}||j|d"}|j||s~|j||q~t|j|jd#|_t|jd$|_|jd%|_| |_!|"|_#|jd&|_$|j$r*t%|j$&dk|_$ntd'd(|_$|jd)|_'|j'rdt%|j'&dk|_'ntd*d(|_'|(|_)|*|_+|,|_-ddl.} t /| 0d+} tj 1|j2d,d-} t /| | |_3|4|r|rdS|d.krt 5| stj /td/d0} t 5| rt6tj /| d1tj7rHt8tj /| d1t6tj /| d2tj7rvt8tj /| d2t9:| | t9;| t<| | ntj=| d3d4t /| |}t >|st?j@Ad5d6|inz|B|} d7d8d9d:d;g}| dks | d<tCks |D|tE|krt?j@Ad=|| dkrXt?j@Ad>|jF|d?d@|_n| d<tCkrt?j@AdAdBtCdC| d<|jF|| d<d@|_nL|D|tE|krt?j@AdDdEtE|dF|D|dGdHdId|_ddJlm}|dK}t?j@AdL||tj >|r.tG|||tj >|dMrXtG|dM|dM|tj >|dNrtG|dN|dN|t?j@AdO||H|t?j@AdPnt?j@AdQ|Wnddl}|YnXz(t?j@AdRdS|i|jdT|Wnt?j@AdU|ddJlm}|dK}t?j@AdL||tj >|r`tG|||tj >|dMrtG|dM|dM|tj >|dNrtG|dN|dN|t?j@AdV||H||jdT|YnX|IdW|jrvg}|jD]*}|J|d|dX|dY|dZd[qd\}z|jK||Wnddl}|YnX|jL|jd]|MdW|NdS)^NZIBUS_TABLE_DEBUG_LEVELrzCannot open database file %szPRAGMA encoding = "UTF-8";z"PRAGMA case_sensitive_like = true;zPRAGMA page_size = 4096;zPRAGMA cache_size = 20000;zPRAGMA temp_store = MEMORY;z$PRAGMA journal_size_limit = 1000000;zPRAGMA synchronous = NORMAL;z"Error while initializing database.z:CREATE TABLE IF NOT EXISTS main.ime (attr TEXT, val TEXT);Zsomebodyz%sz%Y%m%dzibus-table.svgZLGPLZabcdefghijklmnopqrstuvwxyz4Zspace1,2,3,4,5,6,7,8,9,0z Page_Up,minuszPage_Down,equaltrueZfalsezA IME under IBus Tableus0z{})"namez name.zh_cnz name.zh_hkz name.zh_twZauthoruuid serial_numberZiconlicense languageslanguage_filterZvalid_input_charsmax_key_lengthZ commit_keys select_keysZ page_up_keysZpage_down_keysZ status_promptZdef_full_width_punctZdef_full_width_letteruser_can_define_phraseZ pinyin_modedynamic_adjustZ auto_selectZ auto_commitZ descriptionZlayoutZsymbolrulesleast_commit_length start_chars orientationZalways_show_lookupZ char_promptsz9 SELECT val FROM main.ime WHERE attr = :attr;zC INSERT INTO main.ime (attr, val) VALUES (:attr, :val);attrvalr rr+r'r-zXCould not find "user_can_define_phrase" entry from database, is it an outdated database?Fr.zPCould not find "dynamic_adjust" entry from database, is it an outdated database?Ztables.dbz.cache:memory:HOMEz .ibus/tablesz debug.logzsetup-debug.logT)exist_okz.The user database %(udb)s does not exist yet. Zudbidtabkeysphrasefreq user_freqversionz/The user database %s seems to be incompatible. z1There is no version information in the database. 0.0)old_database_versionz+The version of the database does not match z3(too old or too new?). ibus-table wants version=%s z*But the database actually has version=%s z6The number of columns of the database does not match. zibus-table expects %s columns. z*But the database actually has %s columns. z&But the versions of the databases are z identical. zThis should never happen! )strftimez-%Y-%m-%d_%H:%M:%Sz,Renaming the incompatible database to "%s". z-shmz-walz$Creating a new, empty database "s". zIf user phrases were successfully recovered from the old, incompatible database, they will be used to initialize the new database. zCompatible database %s found. z"Connect to the database %(name)s. r%a ATTACH DATABASE "%s" AS user_db; PRAGMA user_db.encoding = "UTF-8"; PRAGMA user_db.case_sensitive_like = true; PRAGMA user_db.page_size = 4096; PRAGMA user_db.cache_size = 20000; PRAGMA user_db.temp_store = MEMORY; PRAGMA user_db.journal_mode = WAL; PRAGMA user_db.journal_size_limit = 1000000; PRAGMA user_db.synchronous = NORMAL; z Could not open the database %s. z%Creating a new, empty database "%s". user_dbrrr<r=r>r?z INSERT INTO user_db.phrases (tabkeys, phrase, freq, user_freq) VALUES (:tabkeys, :phrase, :freq, :user_freq) PRAGMA wal_checkpoint;)Ointosgetenv DEBUG_LEVEL TypeError ValueErrorZ old_phrasesfilename_user_dbreset_phrases_cachepathisfilesqlite3connectr printrr r executescriptr&Zuuid4timerC_default_ime_attributessortedr rime_propertiesr_mlen_snum is_chinese _is_chineseis_cjkZ_is_cjkr-boollowerr. get_rulesr/get_possible_tabkeys_lengthsZpossible_tabkeys_lengthsget_start_charsZ startcharsibus_table_locationjoinZ data_homebasenamereplace cache_pathload_phrases_cacheisdiraccessF_OKunlinkshutilZcopytreeZrmtreesymlinkmakedirsexistssysstderrwriteget_database_descDATABASE_VERSION%get_number_of_columns_of_phrase_tablelenextract_user_phrasesrename init_user_db create_tablesappend executemanycommitcreate_indexesgenerate_userdb_desc)r rNrDZcreate_databaser select_sqlstr insert_sqlstrr4sqlargsreZ tables_pathZ cache_nameZold_tables_pathdescZphrase_table_column_namesrCZ timestampr=rrrrrsH        )                                     zTabSqliteDb.__init__rrrDTc Cstdkr,tjd||dd||d|r4|s8dSd|}|||d}z*|j|||rj|j||Wnd dl}| YnXdS) zupdate phrase freqsrz+update_phrase() tabkeys=%(t)s phrase=%(p)s tpzuser_freq=%(u)s database=%(d)s )udNzu UPDATE %s.phrases SET user_freq = :user_freq WHERE tabkeys = :tabkeys AND phrase = :phrase ;)r?r<r=r) rKrsrtrur rrinvalidate_phrases_cacher r ) r r<r=r?databaserrrr rrr update_phrases6 zTabSqliteDb.update_phrasecCs0||jdkrdS|j|jddS)z1 Trigger a checkpoint operation. NrG)save_phrases_cacherOr rrrrrr sync_usrdbs   zTabSqliteDb.sync_usrdbcCstdkrtjdi|_dS)z. Make the phrases cache empty rzreset_phrases_cache() N)rKrsrtru_phrases_cacherrrrrPs zTabSqliteDb.reset_phrases_cachecCsVtdkrtjdtd|jdD],}|j|d|r$|j|d|q$dS)u Delete all phrases starting with “tabkeys” from the phrases cache. :param tabkeys: The keys typed :type tabkeys: String rzinvalidate_phrases_cache() rN) rKrsrtruranger[rrpop)r r<irrrrs  z$TabSqliteDb.invalidate_phrases_cachecCstdkrtjdz@ddl}|t|jd|_|j d}|rL||j krRi|_Wnvt k rtdkr~tjd|jYnJt k rtdkrtjd|jYntjd |jYnXdS) z. Load phrases cache from disk rzload_phrases_cache() rNrr'z'load_phrases_cache() File %s not found z1load_phrases_cache() Permission error reading %s z.load_phrases_cache() Unknown error reading %s ) rKrsrtrujsonloadopenrirrr\FileNotFoundErrorPermissionError)r rZsnumrrrrjs4   zTabSqliteDb.load_phrases_cachecCsztdkrtjdzDddl}|j|jd<|jd}||jt |dt ||jWnddl }| YnXdS)z. Save phrases cache from disk rzsave_phrases_cache() rNr'z.tmpw)rKrsrtrurr\rridumprrIrhr r )r rZ _cache_pathr rrrrs   zTabSqliteDb.save_phrases_cachecCs@|jd}|r<|d}|D]}|ddkrdSqdS)zg Check whether this input method is classified as Chinese in the the database. r),zhTF)rZrsplitrafind)r r)Zlangslangrrrr]$s  zTabSqliteDb.is_chinesecCsH|jd}|rD|d}|D]$}dD]}||r&dSq&qdS)zx Check whether this input method is classified as Chinese, Japanese, or Korean in the database. r)r)rZjaZkoTF)rZrrstrip startswith)r r)Zlanguagerrrrr_1s   zTabSqliteDb.is_cjkcCs$|jd}|dkr t|dSdS)a Get the default Chinese mode from the database 0 means to show simplified Chinese only 1 means to show traditional Chinese only 2 means to show all characters but show simplified Chinese first 3 means to show all characters but show traditional Chinese first 4 means to show all characters If no mode is specified in the database, return 4 to avoid all filtering of characters. :rtype: Integer r*)Zcm0Zcm1Zcm2Zcm3Zcm4r)rZrrH)r r*rrrget_chinese_mode?s  zTabSqliteDb.get_chinese_modecCs|jd}|r|SdS)zc Get the keys used to select a candidate from the database :rtype: String r,r!rZr)r Zretrrrget_select_keysSs zTabSqliteDb.get_select_keysc Cs2zt|jdWSttfk r,YdSXdS)zl Get the default orientation of the lookup table from the database :rtype: Integer r2rN)rHrZrrLrMrrrrget_orientation^szTabSqliteDb.get_orientationcCsR|dkr0d|}|j|d|}|j|d|}|j||jdS)z%Create tables that contain all phrasemainzp CREATE TABLE IF NOT EXISTS %s.goucima (zi TEXT PRIMARY KEY, goucima TEXT); zp CREATE TABLE IF NOT EXISTS %s.pinyin (pinyin TEXT, zi TEXT, freq INTEGER); z CREATE TABLE IF NOT EXISTS %s.phrases (id INTEGER PRIMARY KEY, tabkeys TEXT, phrase TEXT, freq INTEGER, user_freq INTEGER); N)r rr)r rrrrrr}is   zTabSqliteDb.create_tablescCsd}d}d}|D]B\}}||d}|j||rD|j||q|j||q|jt|j|jd|_t|jd|_ | |_ |jd|_ |j rt |j dk|_ ntd d |_ ||_d S) zUpdate or insert attributes in ime table, attrs is a iterable object Like [(attr,val), (attr,val), ...] This is called only by tabcreatedb.py. z+SELECT val from main.ime WHERE attr = :attrz2UPDATE main.ime SET val = :val WHERE attr = :attr;z6INSERT INTO main.ime (attr, val) VALUES (:attr, :val);r3r6r+r-r"zWCould not find "user_can_define_phrase" entry from database, is it a outdated database?FN)r rr rrrXrZrHrr[r]r^r-r`rarUrbr/)r ZattrsrZ update_sqlstrrr4r5rrrr update_imes6     zTabSqliteDb.update_imec Cs2i}td}td}|js"dSz|jd}|rB|d}|D]}||}|rg}|ddkr|t |d|d <|d d }t ||j krt d |q|D]0} || } | t | dt | dfq||t |d<qFt d |qFWn&tk r,ddl} | YnX|S)zGet phrase construct ruleszc([ea])(\d):(.*)zp(-{0,1}\d)(-{0,1}\d)rr/;rarEabover+zrule: "%s" over max key lengthznot a legal rule: "%s"rN)recompiler-rZrrrmatchgrouprHryr[rUr~ Exceptionr r ) r r/Zpatt_rZpatt_pZ_rulesruleresZcmsZ_cmsZ_cmZcm_resr rrrrbs<       zTabSqliteDb.get_rulesc sjr4jd}fddtd|dDddSztjd}Wnttfk rdd}YnX|dkrtt|jdSgSdS) uReturn a list of the possible lengths for tabkeys in this table. Example: If the table source has rules like: RULES = ce2:p11+p12+p21+p22;ce3:p11+p21+p22+p31;ca4:p11+p21+p31+p41 self._rules will be set to self._rules = { 2: [(1, 1), (1, 2), (2, 1), (2, 2)], 3: [(1, 1), (1, 2), (2, 1), (3, 1)], 4: [(1, 1), (2, 1), (3, 1), (-1, 1)], 'above': 4} and then this function returns “[4, 4, 4]” Or, if the table source has no RULES but LEAST_COMMIT_LENGTH=2 and MAX_KEY_LENGTH = 4, then it returns “[2, 3, 4]” I cannot find any tables which use LEAST_COMMIT_LENGTH though. rcsg|]}tj|qSr)ryr/.0xrrr sz.rErNr0r) r/rrHrZrrLrMlistr[)r Zmax_lenZleast_commit_lenrrrrcs $  z(TabSqliteDb.get_possible_tabkeys_lengthscCs |jdS)z"return possible start chars of IMEr1rrrrrrdszTabSqliteDb.get_start_charscCs*|jd}t|tdkr&|d}|S)z6Get the characters which engine should not change freqZno_check_charsrr)rZrtypedecode)r Z_charsrrrget_no_check_charss  zTabSqliteDb.get_no_check_charsrc Cstdkrtjdt|dd|i}g}|D]*\}}}}|||||d||q0|j|||j |j ddS)uAdd many phrases to database fast. Used by tabcreatedb.py when creating the system database from scratch. “phrases” is a iterable object which looks like: [(tabkeys, phrase, freq ,user_freq), (tabkeys, phrase, freq, user_freq), ...] This function does not check whether phrases are already there. As this function is only used while creating the system database, it is not really necessary to check whether phrases are already there because the database is initially empty anyway. And the caller should take care that the “phrases” argument does not contain duplicates. rzadd_phrases() len(phrases)=%s INSERT INTO %(database)s.phrases (tabkeys, phrase, freq, user_freq) VALUES (:tabkeys, :phrase, :freq, :user_freq); rrFrGN) rKrsrtruryr~rr rrr) r phrasesrrinsert_sqlargsr<r=r>r?rrr add_phrasess&  zTabSqliteDb.add_phrasesc Cstdkr,tjd||dd||d|r4|s8dSdd|i}||d }|j||} | rtdkrtjd d ||d d d| idSdd|i} ||||d} tdkrtjd| | d z*|j| | |r|j||Wnddl } | YnXdS)zaAdd phrase to database, phrase is a object of (tabkeys, phrase, freq ,user_freq) rz&add_phrase tabkeys=%(t)s phrase=%(p)s rzfreq=%(f)s user_freq=%(u)s )frNzk SELECT * FROM %(database)s.phrases WHERE tabkeys = :tabkeys AND phrase = :phrase; rr<r=z add_phrase() z-select_sqlstr=%(sql)s select_sqlargs=%(arg)s )Zsqlargzalready there!: results=%(r)s rrrFz:add_phrase() insert_sqlstr=%(sql)s insert_sqlargs=%(arg)s r) rKrsrtrur rr rrr r ) r r<r=r>r?rrrZselect_sqlargsrrrr rrr add_phrasesj  zTabSqliteDb.add_phrasecCszd}g}|D]\}}|||dq z2|j|j|||j|jdWnddl}|YnXdS)zyAdd goucima into database, goucimas is iterable object Like goucimas = [(zi,goucima), (zi,goucima), ...] zP INSERT INTO main.goucima (zi, goucima) VALUES (:zi, :goucima); )zigoucimarGrN)r~r rrrr r )r Zgoucimasrrrrr rrr add_goucimaOs   zTabSqliteDb.add_goucimac Csd|}d}|D]\}}}|d7}|dddddd d d d d }z|j||||dWqtk rtjdd||||dddl}|YqXq|j dS)zxAdd pinyin to database, pinyins is a iterable object Like: [(zi,pinyin, freq), (zi, pinyin, freq), ...] zX INSERT INTO %s.pinyin (pinyin, zi, freq) VALUES (:pinyin, :zi, :freq); rr1!2@3#r $5%)pinyinrr>z(Error when inserting into pinyin table. z-count=%(c)s pinyin=%(p)s zi=%(z)s freq=%(f)s )crzrN) rhr rrrsrtrur r r) r Zpinyinsrrcountrrr>r rrr add_pinyinbsL  zTabSqliteDb.add_pinyincCs*d}|j||jd|jdS)ze Optimize the database by copying the contents to temporary tables and back. a CREATE TABLE tmp AS SELECT * FROM main.phrases; DELETE FROM main.phrases; INSERT INTO main.phrases SELECT * FROM tmp ORDER BY tabkeys ASC, phrase ASC, user_freq DESC, freq DESC, id ASC; DROP TABLE tmp; CREATE TABLE tmp AS SELECT * FROM main.goucima; DELETE FROM main.goucima; INSERT INTO main.goucima SELECT * FROM tmp ORDER BY zi, goucima; DROP TABLE tmp; CREATE TABLE tmp AS SELECT * FROM main.pinyin; DELETE FROM main.pinyin; INSERT INTO main.pinyin SELECT * FROM tmp ORDER BY pinyin ASC, freq DESC; DROP TABLE tmp; zVACUUM;N)r rVr)r rrrroptimize_database~s  zTabSqliteDb.optimize_databasecCstdkrtjddS)zDrop the indexes in the database to reduce its size We do not use any indexes at the moment, therefore this function does nothing. rzdrop_indexes() NrKrsrtru)r _databaserrr drop_indexess zTabSqliteDb.drop_indexescCstdkrtjddS)uCreate indexes for the database. We do not use any indexes at the moment, therefore this function does nothing. We used indexes before, but benchmarking showed that none of them was really speeding anything up, therefore we deleted all of them to get much smaller databases (about half the size). If some index turns out to be very useful in future, it could be created here (and dropped in “drop_indexes()”). rzcreate_indexes() Nr)r rZ_commitrrrrs  zTabSqliteDb.create_indexescCs,z|d}Wntk r&d}YnX|S)z Encode a string in Big5 or, if that is not possible, return something higher than any Big5 code. :param phrase: String to be encoded in Big5 encoding :type phrase: String :rtype: Byte array ZBig5s)encodeUnicodeEncodeError)r r=big5rrr big5_codes  zTabSqliteDb.big5_coderrcsd}tj|jdd}|dkr*|jndd|dkrr|jrr|dkrNd ndt|fd dd d |St|fd dd d |S)u  “candidates” is an array containing something like: [(tabkeys, phrase, freq, user_freq), ...] “typed_tabkeys” is key sequence the user really typed, which maybe only the beginning part of the “tabkeys” in a matched candidate. dr7r)Zcangjie3Zcangjie5z cangjie-bigz quick-classicZquick3Zquick5cSsdS)Nrrrrrrz-TabSqliteDb.best_candidates..)rErrErc sft|dk d|dt|d@ d|dt|d|d|ddt|ddfS)NrrrrrE)rHchinese_variantsdetect_chinese_categoryryordrbitmaskcode_point_function typed_tabkeysrrrs    rNc sTt|dk d|dd|dt|d|d|ddt|ddfS)NrrrrEr)rHryrr)rrrrrs    )rIrQrgrNrhrr^rY)r r candidates chinese_modeZmaximum_number_of_candidatesZ engine_namerrrbest_candidatess&   zTabSqliteDb.best_candidatesc Cs.|sgS|j|}|r|Sd}|r(d}|js4|jrBdd|i} n dd|i} d} dD]} | ||fkrV| } qV|} | | | | } d||fkr| d| d} d ||fkr| d | d } |r| |d } |r| |d } |r| d 7} | | d } td krtjd | t | f|j | |  }d}|dkr6d }n|d krDd}|sP|}n.g}|D]$}|t |d @rX||qXi}|D]f}|d|d f}||kr|||<n<|||t|d||dt|d||dffgq|j|||d}td kr tjdt |||j|<|S)zE Get matching phrases for tabkeys from the database. rz AND length(phrase)=1 a SELECT tabkeys, phrase, freq, user_freq FROM ( SELECT tabkeys, phrase, freq, user_freq FROM main.phrases WHERE tabkeys LIKE :tabkeys ESCAPE :escapechar %(one_char_condition)s UNION ALL SELECT tabkeys, phrase, freq, user_freq FROM user_db.phrases WHERE tabkeys LIKE :tabkeys ESCAPE :escapechar %(one_char_condition)s ) one_char_conditionz SELECT tabkeys, phrase, freq, user_freq FROM main.phrases WHERE tabkeys LIKE :tabkeys ESCAPE :escapechar %(one_char_condition)s u☺z!@#r_%%)r< escapecharrz$select_words() sqlstr=%s sqlargs=%s NrrErrrrzselect_words() best=%s )rrr-r.rhrKrsrtrurr rr rrr~updatemaxrvalues)r r<Zonecharrsingle_wildcard_charmulti_wildcard_charZ auto_wildcardZbestrrrchartabkeys_for_likerZunfiltered_resultsrrrphrase_frequenciesrrrr select_wordss                  zTabSqliteDb.select_wordscCs|sgSd}|}|r ||d}|r0||d}|d7}d|i}|j||}d} |dkrdd} n |dkrpd} g} |D]J\} } } | s| t| | | dgqx| t| @rx| t| | | dgqx|j|| |d S) zh Get Chinese characters matching the pinyin given by tabkeys from the database. z} SELECT pinyin, zi, freq FROM main.pinyin WHERE pinyin LIKE :tabkeys ORDER BY freq DESC, pinyin ASC ;rrr<NrrrEr) rhr rr r~tuplerrr)r r<rrrrrrrrrrrr>rrr#select_chinese_characters_by_pinyinYsBz/TabSqliteDb.select_chinese_characters_by_pinyincCshzFd}|j|d}|j|dtfd}|j|d|jWnddl}|YnXdS)z{ Add a description table to the user database This adds the database version and the create time zBCREATE TABLE IF NOT EXISTS user_db.desc (name PRIMARY KEY, value);z2INSERT OR IGNORE INTO user_db.desc VALUES (?, ?);r@zMINSERT OR IGNORE INTO user_db.desc VALUES (?, DATETIME("now", "localtime"));)z create-timerN)r rVrrwrr r )r Z sqlstringr rrrrs z TabSqliteDb.generate_userdb_desccCs6|dkr dSt|s2t|}|d|dS)z Initialize the user database unless it is an in-memory database :param db_file: Full path of the database file. :type db_file: String r8Nax PRAGMA encoding = "UTF-8"; PRAGMA case_sensitive_like = true; PRAGMA page_size = 4096; PRAGMA cache_size = 20000; PRAGMA temp_store = MEMORY; PRAGMA journal_mode = WAL; PRAGMA journal_size_limit = 1000000; PRAGMA synchronous = NORMAL; )rQrrrSrTrVr)r db_filer rrrr|s    zTabSqliteDb.init_user_dbcCsbt|sdSz@t|}i}|dD]}|d||d<q,||WSYdSXdS)z Get the description table from the database :param db_file: Full path of the database file. :type db_file: String :rtype: Dictionary NzSELECT * FROM desc;rr)rQrrrSrTrr close)r rr rrowrrrrvs  zTabSqliteDb.get_database_desccCst|sdSzbt|}|d}d|dd}t d|}|rh| d d}t |WSWdSWnYdSXdS)a1 Get the number of columns in the 'phrases' table in the database in db_file. Determines the number of columns by parsing this: sqlite> select sql from sqlite_master where name='phrases'; CREATE TABLE phrases (id INTEGER PRIMARY KEY, tabkeys TEXT, phrase TEXT, freq INTEGER, user_freq INTEGER) sqlite> This result could be on a single line, as above, or on multiple lines. :param db_file: Full path of the database file. :rtype: Integer rz3select sql from sqlite_master where name='phrases'; z .*\((.*)\)rrN) rQrrrSrTrr rf splitlinesrrrrry)r rr Ztp_resstringrtprrrrxs     z1TabSqliteDb.get_number_of_columns_of_phrase_tablecCsT|sdSd}|j|d|i}|r4|dd}nd}tdkrPtjd||S)zGet goucima of given characterrz0SELECT goucima FROM main.goucima WHERE zi = :zi;rrrzget_goucima() goucima=%s )r rr rKrsrtru)r rrrrrrr get_goucimaszTabSqliteDb.get_goucimacCshtdkr tjd||jdt|tdkr:|d}|sBdSt|dkrX||S|jsbdSt||jkr|jt|}n@t||jdkr|j|jd}ntjd||jddSt||j krtjdd ||j d dSd}|D]R\}}|d kr|d8}|d kr"|d8}||||}|s@dS||7}qtdkrdtjd ||S) u'Parse phrase to get its table code Example: Let’s assume we use wubi-jidian86. The rules in the source of that table are: RULES = ce2:p11+p12+p21+p22;ce3:p11+p21+p31+p32;ca4:p11+p21+p31+p-11 “ce2” is a rule for phrases of length 2, “ce3” is a rule for phrases of length 3, “ca4” is a rule for phrases of length 4 *and* for all phrases with a length greater then 4. “pnm” in such a rule means to use the n-th character of the phrase and take the m-th character of the table code of that character. I.e. “p-11” is the first character of the table code of the last character in the phrase. Let’s assume the phrase is “天下大事”. The goucima (構詞碼 = “word formation keys”) for these 4 characters are: character goucima 天 gdi 下 ghi 大 dddd 事 gkvh (If no special goucima are defined by the user, the longest encoding for a single character in a table is the goucima for that character). The length of the phrase “天下大事” is 4 characters, therefore the rule ca4:p11+p21+p31+p-11 applies, i.e. the table code for “天下大事” is calculated by using the first, second, third and last character of the phrase and taking the first character of the goucima for each of these. Therefore, the table code for “天下大事” is “ggdg”. rz'parse_phrase() phrase=%(p)s rules%(r)s )rrrzUTF-8rz9No rule for this phrase length. phrase=%(p)s rules=%(r)s z!Rule exceeds maximum key length. zrule=%(r)s self._mlen=%(m)s )rmrzparse_phrase() tabkeys=%s ) rKrsrtrur/rrryrr[)r r=rr<rZmaZtabkeyrrr parse_phrases\'           zTabSqliteDb.parse_phrasecCstdkrtjd||d|r&|s*dSd}|d|d}|j||}tdkrxtjd||dd d |it|S) u Checks whether “phrase” can be matched in the system database with a key sequence *starting* with “tabkeys”. rz3is_in_system_database() tabkeys=%(t)s phrase=%(p)s rFzf SELECT * FROM main.phrases WHERE tabkeys LIKE :tabkeys AND phrase = :phrase; rrz3is_in_system_database() tabkeys=%(t)s phrase=%(p)s zresults=%(r)s r)rKrsrtrur rr r`)r r<r=rrrrrris_in_system_databaseNs,z!TabSqliteDb.is_in_system_databasecCsvtdkrtjd||d|r&|s*dSd}||d}|j||}tdkrbtjd||rr|ddSdS)uV Return how often a conversion result “phrase” for the typed keys “tabkeys” has been happened by checking the user database. :param tabkeys: The keys typed :type tabkeys: String :param phrase: A conversion result for these tabkeys :type phrase: String :rtype: Integer rz,user_frequency() tabkeys=%(t)s phrase=%(p)s rrz SELECT sum(user_freq) FROM user_db.phrases WHERE tabkeys = :tabkeys AND phrase = :phrase GROUP BY tabkeys, phrase; rzuser_frequency() result=%s )rKrsrtrur rr )r r<r=rrrrrruser_frequencygs   zTabSqliteDb.user_frequencycCstdkrtjd||dt|tdkr8|d}t|tdkrR|d}|rZ|s^dS|jrp|tkrpdS|js|j r|j sdS| |}|sdS|j ||drdS|j ||ddkrdS|j||d dd d n|j ||dr*|j ||d}|dkr|j|||dd n|j||ddd d nj|j r:|j s>dS| |}|sRdS|j ||d}|dkr|j|||dd n|j||d dd d dS) aAdjust user_freq in user database if necessary. Also, if the phrase is not in the system database, and it is a Chinese table, and defining user phrases is allowed, add it as a user defined phrase to the user database if it is not yet there. rz2check_phrase_internal() tabkey=%(t)s phrase=%(p)s rrutf8NrrrrD)r<r=r>r?r)r<r=r?)rKrsrtrurrr^CHINESE_NOCHECK_CHARSr.r-r]rr r rr)r r<r=r?rrr check_phrases       zTabSqliteDb.check_phrasecCsJt|tdkr|d}d}d|i}|j||}dd|D}|S)uu Return the list of possible tabkeys for a phrase. For example, if “phrase” is “你” and the table is wubi-jidian.86.txt, the result will be ['wq', 'wqi', 'wqiy'] because that table contains the following 3 lines matching that phrase exactly: wq 你 597727619 wqi 你 1490000000 wqiy 你 1490000000 rr zo SELECT tabkeys FROM main.phrases WHERE phrase = :phrase ORDER by length(tabkeys) ASC; r=cSsg|] }|dqS)rrrrrrrsz,TabSqliteDb.find_zi_code..)rrr rr )r r=rrrZlist_of_possible_tabkeysrrr find_zi_codes  zTabSqliteDb.find_zi_codecCsZ|sdS|rdd|i}n dd|i}||d}|j|||rL|j||dS)z$Remove phrase from database Nzu DELETE FROM %(database)s.phrases WHERE tabkeys = :tabkeys AND phrase = :phrase; rz^ DELETE FROM %(database)s.phrases WHERE phrase = :phrase; r)r rrr)r r<r=rrZ delete_sqlstrZdelete_sqlargsrrr remove_phrases  zTabSqliteDb.remove_phraserAc Csltjdz8t|}|d|dkrt|d}|t|ddd}tjdt ||d d WSg}|d }|D]z}d }d |d i}|j ||} | r| | d d |d d |dfq| |d } | r| | |d d|dfq|t|ddd}tjddt ||d d WSWn d d l } | gYSXd S)z"extract user phrases from databasezCTrying to recover the phrases from the old, incompatible database. rGrz SELECT tabkeys, phrase, freq, sum(user_freq) FROM phrases GROUP BY tabkeys, phrase, freq; cSs|d|d|d|dfSNrrrErrrrrrrrz2TabSqliteDb.extract_user_phrases..rz4Recovered phrases from the old database: phrases=%s Nz;SELECT phrase, sum(user_freq) FROM phrases GROUP BY phrase;z SELECT tabkeys FROM main.phrases WHERE phrase = :phrase ORDER BY length(tabkeys) DESC; r=rrrcSs|d|d|d|dfSrrrrrrr,rz.Recovered phrases from the very old database: z phrases=%s )rsrtrurSrTrr rrYrr r~rr r ) r Z database_filerBr rrrrrZtabkeys_resultsr<r rrrrzsp   z TabSqliteDb.extract_user_phrases)NNF)rrrrDT)r)r)rrrrrT)r)T)rrr)rFrrrF)rrrr)rr)rr)rr)rrrDT)rrA)-rrrrrrrrPrrjrr]r_rrrr}rrbrcrdrrrrrrrrrrrrrr|rvrxrrr r r rrrzrrrrrrs 2      #!% & 7   8 d *&U   A r)rrIos.pathrQrsrorSr&rWrr version_inforeloadZsetdefaultencodingrHrKrwr rrrrrrs"    &