mirror of
https://github.com/fxsjy/jieba.git
synced 2025-07-10 00:01:33 +08:00
merge from master
This commit is contained in:
ZoeyYoung
parent
2857ae45cc
commit
dce353f88b
14
Changelog
14
Changelog
@ -1,9 +1,9 @@
|
||||
2013-07-01: version 0.31
|
||||
1. 修改了代码缩进格式，遵循PEP8标准
|
||||
2. 支持Jython解析器，感谢 @piaolingxue
|
||||
2. 支持Jython解析器，感谢 @piaolingxue
|
||||
3. 修复中英混合词汇不能识别数字在前词语的Bug
|
||||
4. 部分代码重构，感谢 @chao78787
|
||||
5. 多进程并行分词模式下自动检测CPU个数设置合适的进程数，感谢@linkerlin
|
||||
4. 部分代码重构，感谢 @chao78787
|
||||
5. 多进程并行分词模式下自动检测CPU个数设置合适的进程数，感谢@linkerlin
|
||||
6. 修复了0.3版中jieba.extra_tags方法对whoosh模块的错误依赖
|
||||
|
||||
|
||||
@ -42,8 +42,8 @@
|
||||
2013-04-27: version 0.28
|
||||
========================
|
||||
1) 新增词典lazy load功能，用户可以在'import jieba'后再改变词典的路径. 感谢hermanschaaf
|
||||
2) 显示词典加载异常时错误的词条信息. 感谢neuront
|
||||
3) 修正了词典被vim编辑后会加载失败的bug. 感谢neuront
|
||||
2) 显示词典加载异常时错误的词条信息. 感谢neuront
|
||||
3) 修正了词典被vim编辑后会加载失败的bug. 感谢neuront
|
||||
|
||||
2013-04-22: version 0.27
|
||||
========================
|
||||
@ -80,7 +80,7 @@
|
||||
2012-11-28: version 0.22
|
||||
========================
|
||||
1) 新增jieba.cut_for_search方法， 该方法在精确分词的基础上对“长词”进行再次切分，适用于搜索引擎领域的分词，比精确分词模式有更高的召回率。
|
||||
2) 开始支持Python3.x版。 之前一直是只支持Python2.x系列，从这个版本起有一个单独的jieba3k
|
||||
2) 开始支持Python3.x版。 之前一直是只支持Python2.x系列，从这个版本起有一个单独的jieba3k
|
||||
|
||||
|
||||
2012-11-23: version 0.21
|
||||
@ -91,7 +91,7 @@
|
||||
|
||||
2012-11-06: version 0.20
|
||||
========================
|
||||
1) 新增词性标注功能
|
||||
1) 新增词性标注功能
|
||||
|
||||
|
||||
2012-10-25: version 0.19
|
||||
|
||||
176
README.md
176
README.md
@ -14,9 +14,9 @@ jieba
|
||||
Feature
|
||||
========
|
||||
* 支持三种分词模式：
|
||||
* 精确模式，试图将句子最精确地切开，适合文本分析；
|
||||
* 全模式，把句子中所有的可以成词的词语都扫描出来, 速度非常快，但是不能解决歧义；
|
||||
* 搜索引擎模式，在精确模式的基础上，对长词再次切分，提高召回率，适合用于搜索引擎分词。
|
||||
* 精确模式，试图将句子最精确地切开，适合文本分析；
|
||||
* 全模式，把句子中所有的可以成词的词语都扫描出来, 速度非常快，但是不能解决歧义；
|
||||
* 搜索引擎模式，在精确模式的基础上，对长词再次切分，提高召回率，适合用于搜索引擎分词。
|
||||
|
||||
* 支持繁体分词
|
||||
* 支持自定义词典
|
||||
@ -29,19 +29,31 @@ http://jiebademo.ap01.aws.af.cm/
|
||||
|
||||
(Powered by Appfog)
|
||||
|
||||
Python Version
|
||||
==============
|
||||
* 目前master分支是只支持Python2.x 的
|
||||
* Python3.x 版本的分支也已经基本可用： https://github.com/fxsjy/jieba/tree/jieba3k
|
||||
网站代码：https://github.com/fxsjy/jiebademo
|
||||
|
||||
|
||||
Usage
|
||||
========
|
||||
Python 2.x 下的安装
|
||||
===================
|
||||
* 全自动安装：`easy_install jieba` 或者 `pip install jieba`
|
||||
* 半自动安装：先下载http://pypi.python.org/pypi/jieba/ ，解压后运行python setup.py install
|
||||
* 手动安装：将jieba目录放置于当前目录或者site-packages目录
|
||||
* 通过import jieba 来引用 （第一次import时需要构建Trie树，需要几秒时间）
|
||||
|
||||
|
||||
Python 3.x 下的安装
|
||||
====================
|
||||
* 目前master分支是只支持Python2.x 的
|
||||
* Python3.x 版本的分支也已经基本可用： https://github.com/fxsjy/jieba/tree/jieba3k
|
||||
|
||||
git clone https://github.com/fxsjy/jieba.git
|
||||
git checkout jieba3k
|
||||
python setup.py install
|
||||
|
||||
结巴分词Java版本
|
||||
================
|
||||
作者：piaolingxue
|
||||
地址：https://github.com/huaban/jieba-analysis
|
||||
|
||||
Algorithm
|
||||
========
|
||||
* 基于Trie树结构实现高效的词图扫描，生成句子中汉字所有可能成词情况所构成的有向无环图（DAG)
|
||||
@ -76,13 +88,13 @@ Algorithm
|
||||
|
||||
Output:
|
||||
|
||||
【全模式】: 我/ 来到/ 北京/ 清华/ 清华大学/ 华大/ 大学
|
||||
【全模式】: 我/ 来到/ 北京/ 清华/ 清华大学/ 华大/ 大学
|
||||
|
||||
【精确模式】: 我/ 来到/ 北京/ 清华大学
|
||||
【精确模式】: 我/ 来到/ 北京/ 清华大学
|
||||
|
||||
【新词识别】：他, 来到, 了, 网易, 杭研, 大厦 (此处，“杭研”并没有在词典中，但是也被Viterbi算法识别出来了)
|
||||
【新词识别】：他, 来到, 了, 网易, 杭研, 大厦 (此处，“杭研”并没有在词典中，但是也被Viterbi算法识别出来了)
|
||||
|
||||
【搜索引擎模式】： 小明, 硕士, 毕业, 于, 中国, 科学, 学院, 科学院, 中国科学院, 计算, 计算所, 后, 在, 日本, 京都, 大学, 日本京都大学, 深造
|
||||
【搜索引擎模式】： 小明, 硕士, 毕业, 于, 中国, 科学, 学院, 科学院, 中国科学院, 计算, 计算所, 后, 在, 日本, 京都, 大学, 日本京都大学, 深造
|
||||
|
||||
功能 2) ：添加自定义词典
|
||||
================
|
||||
@ -92,16 +104,16 @@ Output:
|
||||
* 词典格式和`dict.txt`一样，一个词占一行；每一行分三部分，一部分为词语，另一部分为词频，最后为词性（可省略），用空格隔开
|
||||
* 范例：
|
||||
|
||||
* 自定义词典：https://github.com/fxsjy/jieba/blob/master/test/userdict.txt
|
||||
|
||||
* 用法示例：https://github.com/fxsjy/jieba/blob/master/test/test_userdict.py
|
||||
|
||||
* 自定义词典：https://github.com/fxsjy/jieba/blob/master/test/userdict.txt
|
||||
|
||||
* 之前： 李小福 / 是 / 创新 / 办 / 主任 / 也 / 是 / 云 / 计算 / 方面 / 的 / 专家 /
|
||||
* 用法示例：https://github.com/fxsjy/jieba/blob/master/test/test_userdict.py
|
||||
|
||||
|
||||
* 之前： 李小福 / 是 / 创新 / 办 / 主任 / 也 / 是 / 云 / 计算 / 方面 / 的 / 专家 /
|
||||
|
||||
* 加载自定义词库后： 李小福 / 是 / 创新办 / 主任 / 也 / 是 / 云计算 / 方面 / 的 / 专家 /
|
||||
|
||||
* 加载自定义词库后： 李小福 / 是 / 创新办 / 主任 / 也 / 是 / 云计算 / 方面 / 的 / 专家 /
|
||||
|
||||
|
||||
* "通过用户自定义词典来增强歧义纠错能力" --- https://github.com/fxsjy/jieba/issues/14
|
||||
|
||||
功能 3) ：关键词提取
|
||||
@ -112,36 +124,80 @@ Output:
|
||||
|
||||
代码示例 （关键词提取）
|
||||
|
||||
https://github.com/fxsjy/jieba/blob/master/test/extract_tags.py
|
||||
https://github.com/fxsjy/jieba/blob/master/test/extract_tags.py
|
||||
|
||||
功能 4) : 词性标注
|
||||
================
|
||||
* 标注句子分词后每个词的词性，采用和ictclas兼容的标记法
|
||||
* 用法示例
|
||||
|
||||
>>> import jieba.posseg as pseg
|
||||
>>> words =pseg.cut("我爱北京天安门")
|
||||
>>> for w in words:
|
||||
... print(w.word,w.flag)
|
||||
...
|
||||
我 r
|
||||
爱 v
|
||||
北京 ns
|
||||
天安门 ns
|
||||
|
||||
>>> import jieba.posseg as pseg
|
||||
>>> words = pseg.cut("我爱北京天安门")
|
||||
>>> for w in words:
|
||||
... print w.word, w.flag
|
||||
...
|
||||
我 r
|
||||
爱 v
|
||||
北京 ns
|
||||
天安门 ns
|
||||
|
||||
功能 5) : 并行分词
|
||||
==================
|
||||
* 原理：将目标文本按行分隔后，把各行文本分配到多个python进程并行分词，然后归并结果，从而获得分词速度的可观提升
|
||||
* 基于python自带的multiprocessing模块，目前暂不支持windows
|
||||
* 用法：
|
||||
* `jieba.enable_parallel(4)` # 开启并行分词模式，参数为并行进程数
|
||||
* `jieba.disable_parallel()` # 关闭并行分词模式
|
||||
* `jieba.enable_parallel(4)` # 开启并行分词模式，参数为并行进程数
|
||||
* `jieba.disable_parallel()` # 关闭并行分词模式
|
||||
|
||||
* 例子：
|
||||
https://github.com/fxsjy/jieba/blob/master/test/parallel/test_file.py
|
||||
https://github.com/fxsjy/jieba/blob/master/test/parallel/test_file.py
|
||||
|
||||
* 实验结果：在4核3.4GHz Linux机器上，对金庸全集进行精确分词，获得了1MB/s的速度，是单进程版的3.3倍。
|
||||
|
||||
|
||||
功能 6) : Tokenize：返回词语在原文的起始位置
|
||||
============================================
|
||||
* 注意，输入参数只接受unicode
|
||||
* 默认模式
|
||||
|
||||
```python
|
||||
result = jieba.tokenize('永和服装饰品有限公司')
|
||||
for tk in result:
|
||||
print("word %s\t\t start: %d \t\t end:%d" % (tk[0], tk[1], tk[2]))
|
||||
```
|
||||
|
||||
```
|
||||
word 永和 start: 0 end:2
|
||||
word 服装 start: 2 end:4
|
||||
word 饰品 start: 4 end:6
|
||||
word 有限公司 start: 6 end:10
|
||||
|
||||
```
|
||||
|
||||
* 搜索模式
|
||||
|
||||
```python
|
||||
result = jieba.tokenize('永和服装饰品有限公司', mode='search')
|
||||
for tk in result:
|
||||
print("word %s\t\t start: %d \t\t end:%d" % (tk[0], tk[1], tk[2]))
|
||||
```
|
||||
|
||||
```
|
||||
word 永和 start: 0 end:2
|
||||
word 服装 start: 2 end:4
|
||||
word 饰品 start: 4 end:6
|
||||
word 有限 start: 6 end:8
|
||||
word 公司 start: 8 end:10
|
||||
word 有限公司 start: 6 end:10
|
||||
```
|
||||
|
||||
|
||||
功能 7) : ChineseAnalyzer for Whoosh搜索引擎
|
||||
============================================
|
||||
* 引用： `from jieba.analyse import ChineseAnalyzer `
|
||||
* 用法示例：https://github.com/fxsjy/jieba/blob/master/test/test_whoosh.py
|
||||
|
||||
|
||||
其他词典
|
||||
========
|
||||
1. 占用内存较小的词典文件
|
||||
@ -182,14 +238,14 @@ jieba采用延迟加载，"import jieba"不会立即触发词典的加载，一
|
||||
常见问题
|
||||
=========
|
||||
1）模型的数据是如何生成的？https://github.com/fxsjy/jieba/issues/7
|
||||
|
||||
|
||||
2）这个库的授权是? https://github.com/fxsjy/jieba/issues/2
|
||||
|
||||
|
||||
更多问题请点击：https://github.com/fxsjy/jieba/issues?sort=updated&state=closed
|
||||
|
||||
|
||||
Change Log
|
||||
==========
|
||||
http://www.oschina.net/p/jieba/news#list
|
||||
https://github.com/fxsjy/jieba/blob/master/Changelog
|
||||
|
||||
jieba
|
||||
========
|
||||
@ -224,30 +280,30 @@ Function 1): cut
|
||||
Code example: segmentation
|
||||
==========
|
||||
|
||||
#encoding=utf-8
|
||||
import jieba
|
||||
#encoding=utf-8
|
||||
import jieba
|
||||
|
||||
seg_list = jieba.cut("我来到北京清华大学",cut_all=True)
|
||||
print("Full Mode:", "/ ".join(seg_list)) #全模式
|
||||
seg_list = jieba.cut("我来到北京清华大学", cut_all=True)
|
||||
print("Full Mode:", "/ ".join(seg_list)) # 全模式
|
||||
|
||||
seg_list = jieba.cut("我来到北京清华大学",cut_all=False)
|
||||
print("Default Mode:", "/ ".join(seg_list)) #默认模式
|
||||
seg_list = jieba.cut("我来到北京清华大学", cut_all=False)
|
||||
print("Default Mode:", "/ ".join(seg_list)) # 默认模式
|
||||
|
||||
seg_list = jieba.cut("他来到了网易杭研大厦")
|
||||
print(", ".join(seg_list))
|
||||
seg_list = jieba.cut("他来到了网易杭研大厦")
|
||||
print(", ".join(seg_list))
|
||||
|
||||
seg_list = jieba.cut_for_search("小明硕士毕业于中国科学院计算所，后在日本京都大学深造") #搜索引擎模式
|
||||
print(", ".join(seg_list))
|
||||
seg_list = jieba.cut_for_search("小明硕士毕业于中国科学院计算所，后在日本京都大学深造") # 搜索引擎模式
|
||||
print(", ".join(seg_list))
|
||||
|
||||
Output:
|
||||
|
||||
[Full Mode]: 我/ 来到/ 北京/ 清华/ 清华大学/ 华大/ 大学
|
||||
[Full Mode]: 我/ 来到/ 北京/ 清华/ 清华大学/ 华大/ 大学
|
||||
|
||||
[Accurate Mode]: 我/ 来到/ 北京/ 清华大学
|
||||
[Accurate Mode]: 我/ 来到/ 北京/ 清华大学
|
||||
|
||||
[Unknown Words Recognize] 他, 来到, 了, 网易, 杭研, 大厦 (In this case, "杭研" is not in the dictionary, but is identified by the Viterbi algorithm)
|
||||
[Unknown Words Recognize] 他, 来到, 了, 网易, 杭研, 大厦 (In this case, "杭研" is not in the dictionary, but is identified by the Viterbi algorithm)
|
||||
|
||||
[Search Engine Mode]： 小明, 硕士, 毕业, 于, 中国, 科学, 学院, 科学院, 中国科学院, 计算, 计算所, 后, 在
|
||||
[Search Engine Mode]： 小明, 硕士, 毕业, 于, 中国, 科学, 学院, 科学院, 中国科学院, 计算, 计算所, 后, 在
|
||||
, 日本, 京都, 大学, 日本京都大学, 深造
|
||||
|
||||
|
||||
@ -259,13 +315,13 @@ Function 2): Add a custom dictionary
|
||||
* The dictionary format is the same as that of `analyse/idf.txt`: one word per line; each line is divided into two parts, the first is the word itself, the other is the word frequency, separated by a space
|
||||
* Example：
|
||||
|
||||
云计算 5
|
||||
李小福 2
|
||||
创新办 3
|
||||
云计算 5
|
||||
李小福 2
|
||||
创新办 3
|
||||
|
||||
之前： 李小福 / 是 / 创新 / 办 / 主任 / 也 / 是 / 云 / 计算 / 方面 / 的 / 专家 /
|
||||
之前： 李小福 / 是 / 创新 / 办 / 主任 / 也 / 是 / 云 / 计算 / 方面 / 的 / 专家 /
|
||||
|
||||
加载自定义词库后： 李小福 / 是 / 创新办 / 主任 / 也 / 是 / 云计算 / 方面 / 的 / 专家 /
|
||||
加载自定义词库后： 李小福 / 是 / 创新办 / 主任 / 也 / 是 / 云计算 / 方面 / 的 / 专家 /
|
||||
|
||||
Function 3): Keyword Extraction
|
||||
================
|
||||
@ -275,7 +331,7 @@ Function 3): Keyword Extraction
|
||||
|
||||
Code sample (keyword extraction)
|
||||
|
||||
https://github.com/fxsjy/jieba/blob/master/test/extract_tags.py
|
||||
https://github.com/fxsjy/jieba/blob/master/test/extract_tags.py
|
||||
|
||||
Using Other Dictionaries
|
||||
========
|
||||
@ -296,10 +352,10 @@ Initialization
|
||||
By default, Jieba employs lazy loading to only build the trie once it is necessary. This takes 1-3 seconds once, after which it is not initialized again. If you want to initialize Jieba manually, you can call:
|
||||
|
||||
import jieba
|
||||
jieba.initialize() #(optional)
|
||||
jieba.initialize() # (optional)
|
||||
|
||||
You can also specify the dictionary (not supported before version 0.28) :
|
||||
|
||||
|
||||
jieba.set_dictionary('data/dict.txt.big')
|
||||
|
||||
Segmentation speed
|
||||
|
||||
@ -1,13 +1,9 @@
|
||||
from __future__ import with_statement
|
||||
__version__ = '0.31'
|
||||
__license__ = 'MIT'
|
||||
|
||||
import re
|
||||
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
import pprint
|
||||
from . import finalseg
|
||||
import time
|
||||
|
||||
@ -32,7 +28,7 @@ def gen_trie(f_name):
|
||||
trie = {}
|
||||
ltotal = 0.0
|
||||
with open(f_name, 'rb') as f:
|
||||
lineno = 0
|
||||
lineno = 0
|
||||
for line in f.read().rstrip().decode('utf-8').split('\n'):
|
||||
lineno += 1
|
||||
try:
|
||||
@ -42,7 +38,7 @@ def gen_trie(f_name):
|
||||
ltotal+=freq
|
||||
p = trie
|
||||
for c in word:
|
||||
if not c in p:
|
||||
if c not in p:
|
||||
p[c] ={}
|
||||
p = p[c]
|
||||
p['']='' #ending flag
|
||||
@ -127,7 +123,7 @@ def __cut_all(sentence):
|
||||
for k,L in dag.items():
|
||||
if len(L)==1 and k>old_j:
|
||||
yield sentence[k:L[0]+1]
|
||||
old_j = L[0]
|
||||
old_j = L[0]
|
||||
else:
|
||||
for j in L:
|
||||
if j>k:
|
||||
@ -153,7 +149,7 @@ def get_DAG(sentence):
|
||||
if c in p:
|
||||
p = p[c]
|
||||
if '' in p:
|
||||
if not i in DAG:
|
||||
if i not in DAG:
|
||||
DAG[i]=[]
|
||||
DAG[i].append(j)
|
||||
j+=1
|
||||
@ -166,7 +162,7 @@ def get_DAG(sentence):
|
||||
i+=1
|
||||
j=i
|
||||
for i in range(len(sentence)):
|
||||
if not i in DAG:
|
||||
if i not in DAG:
|
||||
DAG[i] =[i]
|
||||
return DAG
|
||||
|
||||
@ -189,7 +185,7 @@ def __cut_DAG(sentence):
|
||||
yield buf
|
||||
buf=''
|
||||
else:
|
||||
if not (buf in FREQ):
|
||||
if (buf not in FREQ):
|
||||
regognized = finalseg.cut(buf)
|
||||
for t in regognized:
|
||||
yield t
|
||||
@ -197,14 +193,14 @@ def __cut_DAG(sentence):
|
||||
for elem in buf:
|
||||
yield elem
|
||||
buf=''
|
||||
yield l_word
|
||||
yield l_word
|
||||
x =y
|
||||
|
||||
if len(buf)>0:
|
||||
if len(buf)==1:
|
||||
yield buf
|
||||
else:
|
||||
if not (buf in FREQ):
|
||||
if (buf not in FREQ):
|
||||
regognized = finalseg.cut(buf)
|
||||
for t in regognized:
|
||||
yield t
|
||||
@ -213,7 +209,7 @@ def __cut_DAG(sentence):
|
||||
yield elem
|
||||
|
||||
def cut(sentence,cut_all=False):
|
||||
if( type(sentence) is bytes):
|
||||
if isinstance(sentence, bytes):
|
||||
try:
|
||||
sentence = sentence.decode('utf-8')
|
||||
except UnicodeDecodeError:
|
||||
@ -230,8 +226,9 @@ def cut(sentence,cut_all=False):
|
||||
if cut_all:
|
||||
cut_block = __cut_all
|
||||
for blk in blocks:
|
||||
if len(blk)==0:
|
||||
continue
|
||||
if re_han.match(blk):
|
||||
#pprint.pprint(__cut_DAG(blk))
|
||||
for word in cut_block(blk):
|
||||
yield word
|
||||
else:
|
||||
@ -287,7 +284,7 @@ def add_word(word, freq, tag=None):
|
||||
user_word_tag_tab[word] = tag.strip()
|
||||
p = trie
|
||||
for c in word:
|
||||
if not c in p:
|
||||
if c not in p:
|
||||
p[c] = {}
|
||||
p = p[c]
|
||||
p[''] = '' # ending flag
|
||||
@ -307,7 +304,7 @@ def __lcut_for_search(sentence):
|
||||
def enable_parallel(processnum=None):
|
||||
global pool,cut,cut_for_search
|
||||
if os.name=='nt':
|
||||
raise Exception("parallel mode only supports posix system")
|
||||
raise Exception("jieba: parallel mode only supports posix system")
|
||||
if sys.version_info[0]==2 and sys.version_info[1]<6:
|
||||
raise Exception("jieba: the parallel feature needs Python version>2.5 ")
|
||||
from multiprocessing import Pool,cpu_count
|
||||
@ -318,7 +315,7 @@ def enable_parallel(processnum=None):
|
||||
def pcut(sentence,cut_all=False):
|
||||
parts = re.compile(b'([\r\n]+)').split(sentence)
|
||||
if cut_all:
|
||||
result = pool.map(__lcut_all,parts)
|
||||
result = pool.map(__lcut_all,parts)
|
||||
else:
|
||||
result = pool.map(__lcut,parts)
|
||||
for r in result:
|
||||
@ -348,7 +345,7 @@ def set_dictionary(dictionary_path):
|
||||
with DICT_LOCK:
|
||||
abs_path = os.path.normpath( os.path.join( os.getcwd(), dictionary_path ) )
|
||||
if not os.path.exists(abs_path):
|
||||
raise Exception("path does not exists:" + abs_path)
|
||||
raise Exception("jieba: path does not exists:" + abs_path)
|
||||
DICTIONARY = abs_path
|
||||
initialized = False
|
||||
|
||||
@ -360,8 +357,8 @@ def get_abs_path_dict():
|
||||
def tokenize(unicode_sentence,mode="default"):
|
||||
#mode ("default" or "search")
|
||||
if not isinstance(unicode_sentence, str):
|
||||
raise Exception("jieba: the input parameter should string.")
|
||||
start = 0
|
||||
raise Exception("jieba: the input parameter should unicode.")
|
||||
start = 0
|
||||
if mode=='default':
|
||||
for w in cut(unicode_sentence):
|
||||
width = len(w)
|
||||
|
||||
@ -2,9 +2,9 @@ import jieba
|
||||
import os
|
||||
|
||||
try:
|
||||
from analyzer import ChineseAnalyzer
|
||||
from analyzer import ChineseAnalyzer
|
||||
except ImportError:
|
||||
pass
|
||||
pass
|
||||
|
||||
_curpath=os.path.normpath( os.path.join( os.getcwd(), os.path.dirname(__file__) ) )
|
||||
f_name = os.path.join(_curpath,"idf.txt")
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
#encoding=utf-8
|
||||
from whoosh.analysis import RegexAnalyzer,LowercaseFilter,StopFilter
|
||||
from whoosh.analysis import Tokenizer,Token
|
||||
from whoosh.analysis import Tokenizer,Token
|
||||
|
||||
import jieba
|
||||
import re
|
||||
@ -31,4 +31,4 @@ class ChineseTokenizer(Tokenizer):
|
||||
yield token
|
||||
|
||||
def ChineseAnalyzer(stoplist=STOP_WORDS,minsize=1):
|
||||
return ChineseTokenizer() | LowercaseFilter() | StopFilter(stoplist=stoplist,minsize=minsize)
|
||||
return ChineseTokenizer() | LowercaseFilter() | StopFilter(stoplist=stoplist,minsize=minsize)
|
||||
|
||||
@ -25,13 +25,13 @@ def load_model():
|
||||
with open(abs_path, mode='rb') as f:
|
||||
start_p = marshal.load(f)
|
||||
f.closed
|
||||
|
||||
|
||||
trans_p = {}
|
||||
abs_path = os.path.join(_curpath, PROB_TRANS_P)
|
||||
with open(abs_path, 'rb') as f:
|
||||
trans_p = marshal.load(f)
|
||||
f.closed
|
||||
|
||||
|
||||
emit_p = {}
|
||||
abs_path = os.path.join(_curpath, PROB_EMIT_P)
|
||||
with file(abs_path, 'rb') as f:
|
||||
@ -41,7 +41,7 @@ def load_model():
|
||||
return start_p, trans_p, emit_p
|
||||
|
||||
if sys.platform.startswith("java"):
|
||||
start_P, trans_P, emit_P = load_model()
|
||||
start_P, trans_P, emit_P = load_model()
|
||||
else:
|
||||
import prob_start,prob_trans,prob_emit
|
||||
start_P, trans_P, emit_P = prob_start.P, prob_trans.P, prob_emit.P
|
||||
@ -61,9 +61,9 @@ def viterbi(obs, states, start_p, trans_p, emit_p):
|
||||
V[t][y] =prob
|
||||
newpath[y] = path[state] + [y]
|
||||
path = newpath
|
||||
|
||||
|
||||
(prob, state) = max([(V[len(obs) - 1][y], y) for y in ('E','S')])
|
||||
|
||||
|
||||
return (prob, path[state])
|
||||
|
||||
|
||||
|
||||
@ -26,19 +26,19 @@ def load_model(f_name,isJython=True):
|
||||
f.closed
|
||||
if not isJython:
|
||||
return result
|
||||
|
||||
|
||||
start_p = {}
|
||||
abs_path = os.path.join(_curpath, PROB_START_P)
|
||||
with open(abs_path, mode='rb') as f:
|
||||
start_p = marshal.load(f)
|
||||
f.closed
|
||||
|
||||
|
||||
trans_p = {}
|
||||
abs_path = os.path.join(_curpath, PROB_TRANS_P)
|
||||
with open(abs_path, 'rb') as f:
|
||||
trans_p = marshal.load(f)
|
||||
f.closed
|
||||
|
||||
|
||||
emit_p = {}
|
||||
abs_path = os.path.join(_curpath, PROB_EMIT_P)
|
||||
with file(abs_path, 'rb') as f:
|
||||
@ -121,7 +121,7 @@ def __cut_detail(sentence):
|
||||
def __cut_DAG(sentence):
|
||||
DAG = jieba.get_DAG(sentence)
|
||||
route ={}
|
||||
|
||||
|
||||
jieba.calc(sentence,DAG,0,route=route)
|
||||
|
||||
x = 0
|
||||
@ -138,7 +138,7 @@ def __cut_DAG(sentence):
|
||||
yield pair(buf,word_tag_tab.get(buf,'x'))
|
||||
buf=''
|
||||
else:
|
||||
if not (buf in jieba.FREQ):
|
||||
if (buf not in jieba.FREQ):
|
||||
regognized = __cut_detail(buf)
|
||||
for t in regognized:
|
||||
yield t
|
||||
@ -153,7 +153,7 @@ def __cut_DAG(sentence):
|
||||
if len(buf)==1:
|
||||
yield pair(buf,word_tag_tab.get(buf,'x'))
|
||||
else:
|
||||
if not (buf in jieba.FREQ):
|
||||
if (buf not in jieba.FREQ):
|
||||
regognized = __cut_detail(buf)
|
||||
for t in regognized:
|
||||
yield t
|
||||
@ -162,7 +162,7 @@ def __cut_DAG(sentence):
|
||||
yield pair(elem,word_tag_tab.get(elem,'x'))
|
||||
|
||||
def __cut_internal(sentence):
|
||||
if not ( type(sentence) is str):
|
||||
if not isinstance(sentence, str):
|
||||
try:
|
||||
sentence = sentence.decode('utf-8')
|
||||
except:
|
||||
@ -199,7 +199,7 @@ def cut(sentence):
|
||||
yield w
|
||||
else:
|
||||
parts = re.compile('([\r\n]+)').split(sentence)
|
||||
result = jieba.pool.map(__lcut_internal,parts)
|
||||
result = jieba.pool.map(__lcut_internal,parts)
|
||||
for r in result:
|
||||
for w in r:
|
||||
yield w
|
||||
|
||||
@ -6,14 +6,16 @@ import jieba
|
||||
jieba.enable_parallel()
|
||||
|
||||
url = sys.argv[1]
|
||||
content = open(url,"rb").read()
|
||||
t1 = time.time()
|
||||
words = "/ ".join(jieba.cut(content))
|
||||
with open(url,"rb") as content:
|
||||
content = content.read()
|
||||
t1 = time.time()
|
||||
words = "/ ".join(jieba.cut(content))
|
||||
t2 = time.time()
|
||||
tm_cost = t2-t1
|
||||
print('cost',tm_cost)
|
||||
print('speed' , len(content)/tm_cost, " bytes/second")
|
||||
|
||||
t2 = time.time()
|
||||
tm_cost = t2-t1
|
||||
with open("1.log","wb") as log_f:
|
||||
log_f.write(words.encode('utf-8'))
|
||||
|
||||
log_f = open("1.log","wb")
|
||||
log_f.write(words.encode('utf-8'))
|
||||
print('speed' , len(content)/tm_cost, " bytes/second")
|
||||
|
||||
|
||||
@ -5,19 +5,15 @@ import jieba
|
||||
jieba.initialize()
|
||||
|
||||
url = sys.argv[1]
|
||||
content = open(url,"rb").read()
|
||||
t1 = time.time()
|
||||
words = "/ ".join(jieba.cut(content))
|
||||
|
||||
t2 = time.time()
|
||||
tm_cost = t2-t1
|
||||
|
||||
log_f = open("1.log","wb")
|
||||
log_f.write(words.encode('utf-8'))
|
||||
|
||||
|
||||
log_f.write(bytes("/ ".join(words),'utf-8'))
|
||||
|
||||
print('cost',tm_cost)
|
||||
print('speed' , len(content)/tm_cost, " bytes/second")
|
||||
with open(url,"rb") as content:
|
||||
content = content.read()
|
||||
t1 = time.time()
|
||||
words = "/ ".join(jieba.cut(content))
|
||||
t2 = time.time()
|
||||
tm_cost = t2-t1
|
||||
print('cost',tm_cost)
|
||||
print('speed' , len(content)/tm_cost, " bytes/second")
|
||||
|
||||
with open("1.log","wb") as log_f:
|
||||
log_f.write(words.encode('utf-8'))
|
||||
log_f.write(bytes("/ ".join(words),'utf-8'))
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user