TensorFlow による Character-Level 言語モデル
RNN & Character-Level 言語モデル
RNN による言語モデルについても少し試しておきます。
以下は TensorFlow のチュートリアルでも参照されている、Stanford の Andrej Karpathy（現在は OpenAI らしい）による RNN の紹介記事ですが :
この中で Character-Level 言語モデルが紹介されています。
We’ll train RNN character-level language models. That is, we’ll give the RNN a huge chunk of text and ask it to model the probability distribution of the next character in the sequence given a sequence of previous characters. This will then allow us to generate new text one character at a time.
ついでに python による簡単な実装例も示されています :
TensorFlow による Character-Level 言語モデルの実装
そして TensorFlow で実装するのですが、実際には A. Karpathy の Torch による実装があるのでそちらを参考にしています。character-level 言語モデルで訓練/サンプリングするためのマルチ層の Recurrent Neural Network (RNN, LSTM そして GRU) を実装していて、具体的にはモデルは一つのテキストファイルを入力として、sequence の次の文字を predict することを学習する RNN を訓練します。
（試した限りでは LSTM よりも GRU の方が良い感じでした。）
題材としては取りあえず、Gutenberg から子供向けの古代エジプトについての歴史書を選んでみました。
で、訓練開始。最初のサンプリングはもちろん意味不明 :
z5L8JqA7ôPPêN:t,yUXaw!7.6àÔâ"ç:Uï7Büâœæàbôæ8XZ46:ï6Z*L MX1ïrâ4JÔôü5.GU8œ8Wê[5çtz.yuMfAt7GToüâPbL"9TzZEUXNmMzzqüæq4UzK*ïG7Dïb4ZZ]ôÔXŒïUâMïæfXWPÔ4'ês4êj6êbGU8MA°]ôêœWZUXWïôZFôJà4ôGYOüPA4âNA4ôPGçT*NZê(übob_KnUtw!Uz8üvfUG9 PT_âwé")kb:0"ôMôG:ÔUDâ.zKdTZoP(AXr?,ôUZ4SiXïU*f"s4WÔqü"564](_4àbUÏZzb4ôXp7_LW",HLkT(,8rxübïôUWbUæZpZrX7.TMWê_X,K(DASELGTü_JPTi5âzGb3PzUHvë44ôzfrMô8ræPtXÔUôbM,UjXê"êi4J3Xdüzlé4,P.WÔP-ôL"ÏU*ZêufrZut'z]"4ék?7âRMï6ô_D*oE4M5ï"o1wUHïP,tê:qqâ7
200 epochs。単語としては成立していますが、英語としてはいまいち…
"Herow? We may be proved the Greek lines for Mamessioh, the hands fall into complete determining roubt what it was slow thee, under Tigolathy, engradus the month, 12, the trade had regulication with a more inclier feet; which is consuit and involved, Whit heart vine upon the Hittite people of the god and in mystery, and a pezit of a sixteen three more enjoyovies. The east the government of "Harehæ). The march. The face of the monarch remained below the two favous. He hadbendy that h
250 epochs。少し英語らしくなってきた？
In whohe,
And two Nile Egypt, Psamatik had
the Levinites from their monument to land in different broadling the
blossom of the Egyptian conqueror.
I came which determined, and had carried had
probably beauty and remove mid. The production of Assour, a
general towns reducted the King"--campte a point of each of his wife and brought
between the circumstances of his others, and to have been the dimensions
war, with difficues of Lakes, Naphuled, and hasing his bounth fine
Ablimn the bas a long god-gual-cylem in expedition, the missult
fronty of kings received afterwards amperiescent feet, and we must, must expenditues
that revolt at Karnak was barbarly one of Caired and Greek might be respected to
very remaining a period may be more him and to cut confine it, in his
worst, a deities of the western city of men cometthing of enarxer till shall be calamities
that here indeed in it thought taken pillarners, between their pas
500 epochs。このモデルではこのあたりが限界でしょうか :
Egypt possistration obtaining his limbs, lief force. The same monarch were told as much invade brich emblec, and to make all the purpose of all period. The like took off the artigious smaller commonly to ultime a hast to the good delights. Let stand who were modern Asiatic doing countries will but a fewer hundred _shupps and Shumad Bœlusaa, Skirwaan; no one-who had ever pieces is cut and left between the Greek nine sides traverse, a certain pure, you adm; but man had been remotle. Pent The two hordes the first chariots landed history Esya-space of the Nile in the crown of Egyptian terrible hand, in the jude timed Horuss to south herself. xpresidrages of successure strengthen after a live death, and all their cities labour is a considerable establishment of the podtis foreshonous, the Arab to Tash and in the imagine food; and a residence Ramesses, came long, and forty numerous in 56°z, in his kindlim, and after ordered the piece of the Nile, much to the sea, which make it scarcely skin a bettive grew lakes of the other." It is designated his general. Antefaadich cemertions must advantage of the number, there were models, or with the sovereignty of Sheshom and Shabak. The head of considerable quiltly ashess fell into the other wind submitted by maintain matiries in each year and Cyrus in the old temple battle--Egypt. Only further recounces who had like agels out at an indecener and accomplishez, and the first limbuted had yet reigned and carri
以上