lzh
486 lines
21 KiB
Plaintext
486 lines
21 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Q2"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"and daughter (V.W.).\n",
|
|
")[Gal(beta 1,3/4)Glc-NAc(beta 1,?)]\n",
|
|
"[5] is to jointly learn an em- bedding of words into an n-dimensional vector space and to use these vectors to predict how likely a word is given its context.\n",
|
|
".\n",
|
|
".\n",
|
|
", xm ).\n",
|
|
"(1992) J. Biol.\n",
|
|
"267, 968-974; Zhou et al.\n",
|
|
"(1995) Mol.\n",
|
|
"9, 208-218).\n",
|
|
"41, 115-123) largely failed to stain tubular basement membranes, suggesting the presence of heparan sulfate chains lacking the specific JM-403 epitope.\n",
|
|
"['and daughter (V.W.).', ')[Gal(beta 1,3/4)Glc-NAc(beta 1,?)]', '[5] is to jointly learn an em- bedding of words into an n-dimensional vector space and to use these vectors to predict how likely a word is given its context.', '.', '.', ', xm ).', '(1992) J. Biol.', '267, 968-974; Zhou et al.', '(1995) Mol.', '9, 208-218).', '41, 115-123) largely failed to stain tubular basement membranes, suggesting the presence of heparan sulfate chains lacking the specific JM-403 epitope.']\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import nltk\n",
|
|
"from nltk.tokenize import sent_tokenize\n",
|
|
"from nltk.tokenize import word_tokenize\n",
|
|
"f=open('sentence_splitter_input.txt','r',encoding='utf-8')\n",
|
|
"ssi=f.read()\n",
|
|
"split_result=sent_tokenize(ssi)\n",
|
|
"under=[]\n",
|
|
"over=[]\n",
|
|
"for i in split_result:\n",
|
|
" if i[0][0].istitle()==False:\n",
|
|
" print(i)\n",
|
|
" over.append(i)\n",
|
|
"print(over)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Q3"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 96,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"E:\\anaconda3\\lib\\site-packages\\ipykernel_launcher.py:49: SettingWithCopyWarning: \n",
|
|
"A value is trying to be set on a copy of a slice from a DataFrame\n",
|
|
"\n",
|
|
"See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"===============================hello====================================\n",
|
|
"e is the three most likely characters after the bigram h and the rank of the actual observed character e is 5.26%\n",
|
|
"t is the three most likely characters after the bigram e and the rank of the actual observed character l is 1.09%\n",
|
|
"e is the three most likely characters after the bigram l and the rank of the actual observed character l is 1.04%\n",
|
|
"e is the three most likely characters after the bigram l and the rank of the actual observed character o is 0.64%\n",
|
|
"f is the three most likely characters after the bigram o and the rank of the actual observed character is 0.64%\n",
|
|
"e is the three most likely characters after the bigram he and the rank of the actual observed character e is 0.03%\n",
|
|
"i is the three most likely characters after the bigram el and the rank of the actual observed character l is 0.08%\n",
|
|
"l is the three most likely characters after the bigram ll and the rank of the actual observed character l is 0.04%\n",
|
|
"===============================weather====================================\n",
|
|
"a is the three most likely characters after the bigram w and the rank of the actual observed character e is 0.60%\n",
|
|
"t is the three most likely characters after the bigram e and the rank of the actual observed character a is 1.54%\n",
|
|
"n is the three most likely characters after the bigram a and the rank of the actual observed character t is 2.25%\n",
|
|
"h is the three most likely characters after the bigram t and the rank of the actual observed character h is 5.70%\n",
|
|
"e is the three most likely characters after the bigram h and the rank of the actual observed character e is 5.26%\n",
|
|
"t is the three most likely characters after the bigram e and the rank of the actual observed character r is 3.35%\n",
|
|
"e is the three most likely characters after the bigram r and the rank of the actual observed character is 3.35%\n",
|
|
"e is the three most likely characters after the bigram we and the rank of the actual observed character e is 0.01%\n",
|
|
"a is the three most likely characters after the bigram ea and the rank of the actual observed character a is 0.09%\n",
|
|
"t is the three most likely characters after the bigram at and the rank of the actual observed character t is 0.05%\n",
|
|
"e is the three most likely characters after the bigram th and the rank of the actual observed character h is 1.95%\n",
|
|
"e is the three most likely characters after the bigram he and the rank of the actual observed character e is 0.33%\n",
|
|
"===============================mystique====================================\n",
|
|
"e is the three most likely characters after the bigram m and the rank of the actual observed character y is 0.09%\n",
|
|
"b is the three most likely characters after the bigram y and the rank of the actual observed character s is 0.36%\n",
|
|
"i is the three most likely characters after the bigram s and the rank of the actual observed character t is 2.18%\n",
|
|
"h is the three most likely characters after the bigram t and the rank of the actual observed character i is 2.27%\n",
|
|
"n is the three most likely characters after the bigram i and the rank of the actual observed character q is 0.02%\n",
|
|
"u is the three most likely characters after the bigram q and the rank of the actual observed character u is 0.21%\n",
|
|
"l is the three most likely characters after the bigram u and the rank of the actual observed character e is 0.21%\n",
|
|
"t is the three most likely characters after the bigram e and the rank of the actual observed character is 0.21%\n",
|
|
"y is the three most likely characters after the bigram my and the rank of the actual observed character y is 0.00%\n",
|
|
"s is the three most likely characters after the bigram ys and the rank of the actual observed character s is 0.04%\n",
|
|
"t is the three most likely characters after the bigram st and the rank of the actual observed character t is 0.12%\n",
|
|
"o is the three most likely characters after the bigram ti and the rank of the actual observed character i is 0.00%\n",
|
|
"u is the three most likely characters after the bigram iq and the rank of the actual observed character q is 0.01%\n",
|
|
"u is the three most likely characters after the bigram qu and the rank of the actual observed character u is 0.04%\n",
|
|
"===============================catcher====================================\n",
|
|
"o is the three most likely characters after the bigram c and the rank of the actual observed character a is 0.83%\n",
|
|
"n is the three most likely characters after the bigram a and the rank of the actual observed character t is 2.25%\n",
|
|
"h is the three most likely characters after the bigram t and the rank of the actual observed character c is 0.23%\n",
|
|
"o is the three most likely characters after the bigram c and the rank of the actual observed character h is 0.92%\n",
|
|
"e is the three most likely characters after the bigram h and the rank of the actual observed character e is 5.26%\n",
|
|
"t is the three most likely characters after the bigram e and the rank of the actual observed character r is 3.35%\n",
|
|
"e is the three most likely characters after the bigram r and the rank of the actual observed character is 3.35%\n",
|
|
"a is the three most likely characters after the bigram ca and the rank of the actual observed character a is 0.06%\n",
|
|
"t is the three most likely characters after the bigram at and the rank of the actual observed character t is 0.02%\n",
|
|
"o is the three most likely characters after the bigram tc and the rank of the actual observed character c is 0.03%\n",
|
|
"h is the three most likely characters after the bigram ch and the rank of the actual observed character h is 0.08%\n",
|
|
"e is the three most likely characters after the bigram he and the rank of the actual observed character e is 0.33%\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"from nltk import FreqDist\n",
|
|
"from nltk import ngrams\n",
|
|
"\n",
|
|
"\n",
|
|
"def to_df(FQ):\n",
|
|
" '''\n",
|
|
" Translate FreqDist to DataFrame\n",
|
|
" '''\n",
|
|
" key_word=[]\n",
|
|
" frequence=[]\n",
|
|
" for key,value in FQ.items():\n",
|
|
" key=''.join(key)\n",
|
|
" key_word.append(key)\n",
|
|
" frequence.append(value)\n",
|
|
"\n",
|
|
" data=pd.DataFrame({\n",
|
|
" 'key_words':key_word,\n",
|
|
" 'frequence':frequence\n",
|
|
" })\n",
|
|
" return data\n",
|
|
"\n",
|
|
"def get_word1(key_word):\n",
|
|
" all_count=freq['frequence'].sum()\n",
|
|
" if len(key_word)==1:\n",
|
|
" next_c=freq[freq['key_words'].str[0]==key_word]['key_words'].tolist()[0][1]\n",
|
|
" new_key=key_word+next_c\n",
|
|
" key_p=freq[freq['key_words'].str.contains(new_key)]['frequence'].sum()\n",
|
|
" return next_c,key_p/all_count\n",
|
|
" \n",
|
|
" elif(len(key_word)==2):\n",
|
|
" next_c=freq[freq['key_words'].str.contains(key_word)]['key_words'].tolist()[0][2]\n",
|
|
" new_key=key_word+next_c\n",
|
|
" key_p=freq[freq['key_words'].str.contains(new_key)]['frequence'].sum()\n",
|
|
" return next_c,key_p/all_count\n",
|
|
"\n",
|
|
"def get_word2(current_word,next_word):\n",
|
|
" all_count=freq['frequence'].sum()\n",
|
|
" new_key=current_word+next_word\n",
|
|
" key_p=freq[freq['key_words'].str.contains(new_key)]['frequence'].sum()\n",
|
|
" return key_p/all_count\n",
|
|
"data=pd.read_csv('training set.txt',sep='\\t',header=None)\n",
|
|
"\n",
|
|
"words=\"\"\n",
|
|
"data[2]=data[0]*data[1]\n",
|
|
"# words+=''.join(data[0]*data[1][i])\n",
|
|
"data[2]=data[2].str.strip()\n",
|
|
"for i in range(len(data)):\n",
|
|
" data[2][i]=word_tokenize(data[2][i])\n",
|
|
" \n",
|
|
"words=[]\n",
|
|
"for i in data[2]:\n",
|
|
" words.extend(i)\n",
|
|
"words=''.join(words)\n",
|
|
"bigrams = ngrams(words, 3)\n",
|
|
"bigramsDist = FreqDist(bigrams)\n",
|
|
"\n",
|
|
"freq=to_df(bigramsDist)\n",
|
|
"freq=freq.sort_values('frequence',ascending=False)\n",
|
|
"train_set=['hello','weather','mystique','catcher']\n",
|
|
"for t_s in train_set:\n",
|
|
" my_word=t_s\n",
|
|
" print(\"===============================\"+my_word+\"====================================\")\n",
|
|
" for i in range(len(my_word)):\n",
|
|
"\n",
|
|
" next_c,p_value=get_word1(my_word[i])\n",
|
|
" try:\n",
|
|
" p_value2=get_word2(my_word[i],my_word[i+1])\n",
|
|
" except:\n",
|
|
" print('{} is the three most likely characters after the bigram {} and the rank of the actual observed character {} is {:.2f}%'.format(next_c,my_word[i],'',p_value2*100))\n",
|
|
" continue \n",
|
|
" print('{} is the three most likely characters after the bigram {} and the rank of the actual observed character {} is {:.2f}%'.format(next_c,my_word[i],my_word[i+1],p_value2*100))\n",
|
|
" for i in range(0,len(my_word)):\n",
|
|
" words=my_word[i:i+2]\n",
|
|
" next_c,p_value=get_word1(words)\n",
|
|
" try:\n",
|
|
" p_value2=get_word2(words,my_word[i+2])\n",
|
|
" except:\n",
|
|
" continue\n",
|
|
" print('{} is the three most likely characters after the bigram {} and the rank of the actual observed character {} is {:.2f}%'.format(next_c,words,my_word[i+1],p_value2*100))\n",
|
|
" \n",
|
|
" "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Q4"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" stat gsp \\\n",
|
|
"0 Arizona $259 billion \n",
|
|
"1 California $3.0 trillion \n",
|
|
"2 Florida $1.0 trillion \n",
|
|
"3 Idaho None \n",
|
|
"4 Maryland $382.4 billion \n",
|
|
"5 Nevada None \n",
|
|
"6 New_Jersey None \n",
|
|
"7 North_Carolina $496 billion \n",
|
|
"8 Pennsylvania $803 billion \n",
|
|
"9 Vermont None \n",
|
|
"\n",
|
|
" nickname \n",
|
|
"0 The Grand Canyon State; The Copper State;The V... \n",
|
|
"1 The Golden State \n",
|
|
"2 The Sunshine State \n",
|
|
"3 Gem State \n",
|
|
"4 Old Line State; Free State; Little America; Am... \n",
|
|
"5 None \n",
|
|
"6 The Garden State \n",
|
|
"7 Old North State; Tar Heel State \n",
|
|
"8 Keystone State; Quaker State \n",
|
|
"9 The Green Mountain State \n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import docx\n",
|
|
"import re\n",
|
|
"import os\n",
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"\n",
|
|
"def nickname_para(file):\n",
|
|
" children = file.element.body.iter()\n",
|
|
" child_iters = []\n",
|
|
" for child in children:\n",
|
|
" # 通过类型判断目录\n",
|
|
" if child.tag.endswith('textbox'):\n",
|
|
" for ci in child.iter():\n",
|
|
" if ci.tag.endswith('main}r'):\n",
|
|
" child_iters.append(ci)\n",
|
|
" textbox = [ci.text for ci in child_iters]\n",
|
|
" p=''.join(textbox)\n",
|
|
" pattern=re.compile('Nickname\\(s\\): (.*?)\\(s\\)')\n",
|
|
" return re.findall(pattern,p)\n",
|
|
"def gsp(file):\n",
|
|
" for p in file.paragraphs:\n",
|
|
" xml = p.paragraph_format.element.xml\n",
|
|
" xml_str = str(xml)\n",
|
|
" wt_list = re.findall('<w:t[\\S\\s]*?</w:t>', xml_str)\n",
|
|
" hyperlink = u''\n",
|
|
" for wt in wt_list:\n",
|
|
" wt_content = re.sub('<[\\S\\s]*?>', u'', wt)\n",
|
|
" hyperlink += wt_content\n",
|
|
" gsp=re.findall('gross state product.*?(\\$\\d+\\.\\d+)',str(hyperlink))\n",
|
|
" \n",
|
|
" if 'gross state product' in hyperlink:\n",
|
|
" res=re.findall('(\\$[\\d+\\.]+ billion|\\$[\\d+\\.]+ trillion)',hyperlink)\n",
|
|
" return res\n",
|
|
"#获取文档对象\n",
|
|
"dir_path=\"./state/\"\n",
|
|
"stats=os.listdir(dir_path)\n",
|
|
"\n",
|
|
"stat_name=[]\n",
|
|
"nickname=[]\n",
|
|
"gross=[]\n",
|
|
"\n",
|
|
"for stat in stats:\n",
|
|
" try:\n",
|
|
" file=docx.Document(dir_path+stat)\n",
|
|
" except:\n",
|
|
" continue\n",
|
|
" stat_name.append(stat.split('.')[0])\n",
|
|
" nickname.append(nickname_para(file))\n",
|
|
" gross.append(gsp(file))\n",
|
|
"\n",
|
|
"res=pd.DataFrame({\n",
|
|
" 'stat':stat_name,\n",
|
|
" 'gsp':gross,\n",
|
|
" 'nickname':nickname\n",
|
|
"})\n",
|
|
"\n",
|
|
"for i in range(len(res)):\n",
|
|
" try:\n",
|
|
" res['nickname'][i]=res['nickname'][i][0]\n",
|
|
" res['nickname'][i]=res['nickname'][i].replace(\"\\\"\",\"\")\n",
|
|
" res['nickname'][i]=res['nickname'][i].replace(\",\",\";\")\n",
|
|
" res['nickname'][i]=res['nickname'][i].replace(\"Motto\",\"\")\n",
|
|
" res['nickname'][i]=res['nickname'][i].replace(\"[1]\",\"\")\n",
|
|
" res['nickname'][i]=res['nickname'][i].replace(\"[2]\",\"\")\n",
|
|
" except:\n",
|
|
" res['nickname'][i]='None'\n",
|
|
" try:\n",
|
|
" res['gsp'][i]=res['gsp'][i][0]\n",
|
|
" except:\n",
|
|
" res['gsp'][i]='None'\n",
|
|
" \n",
|
|
"res.to_csv('res.tsv',sep='\\t',index=None)\n",
|
|
"print(res) \n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>stat</th>\n",
|
|
" <th>gsp</th>\n",
|
|
" <th>nickname</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <td>0</td>\n",
|
|
" <td>Arizona</td>\n",
|
|
" <td>$259 billion</td>\n",
|
|
" <td>The Grand Canyon State; The Copper State;The V...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>1</td>\n",
|
|
" <td>California</td>\n",
|
|
" <td>$3.0 trillion</td>\n",
|
|
" <td>The Golden State</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>2</td>\n",
|
|
" <td>Florida</td>\n",
|
|
" <td>$1.0 trillion</td>\n",
|
|
" <td>The Sunshine State</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>3</td>\n",
|
|
" <td>Idaho</td>\n",
|
|
" <td>None</td>\n",
|
|
" <td>Gem State</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>4</td>\n",
|
|
" <td>Maryland</td>\n",
|
|
" <td>$382.4 billion</td>\n",
|
|
" <td>Old Line State; Free State; Little America; Am...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>5</td>\n",
|
|
" <td>Nevada</td>\n",
|
|
" <td>None</td>\n",
|
|
" <td>None</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>6</td>\n",
|
|
" <td>New_Jersey</td>\n",
|
|
" <td>None</td>\n",
|
|
" <td>The Garden State</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>7</td>\n",
|
|
" <td>North_Carolina</td>\n",
|
|
" <td>$496 billion</td>\n",
|
|
" <td>Old North State; Tar Heel State</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>8</td>\n",
|
|
" <td>Pennsylvania</td>\n",
|
|
" <td>$803 billion</td>\n",
|
|
" <td>Keystone State; Quaker State</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>9</td>\n",
|
|
" <td>Vermont</td>\n",
|
|
" <td>None</td>\n",
|
|
" <td>The Green Mountain State</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>"
|
|
],
|
|
"text/plain": [
|
|
" stat gsp \\\n",
|
|
"0 Arizona $259 billion \n",
|
|
"1 California $3.0 trillion \n",
|
|
"2 Florida $1.0 trillion \n",
|
|
"3 Idaho None \n",
|
|
"4 Maryland $382.4 billion \n",
|
|
"5 Nevada None \n",
|
|
"6 New_Jersey None \n",
|
|
"7 North_Carolina $496 billion \n",
|
|
"8 Pennsylvania $803 billion \n",
|
|
"9 Vermont None \n",
|
|
"\n",
|
|
" nickname \n",
|
|
"0 The Grand Canyon State; The Copper State;The V... \n",
|
|
"1 The Golden State \n",
|
|
"2 The Sunshine State \n",
|
|
"3 Gem State \n",
|
|
"4 Old Line State; Free State; Little America; Am... \n",
|
|
"5 None \n",
|
|
"6 The Garden State \n",
|
|
"7 Old North State; Tar Heel State \n",
|
|
"8 Keystone State; Quaker State \n",
|
|
"9 The Green Mountain State "
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"res"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.1"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|