TensorFlow 2.0 Beta : ガイド : Keras : カスタム callback (翻訳/解説)
翻訳 : (株)クラスキャット セールスインフォメーション
作成日時 : 06/20/2019
* 本ページは、TensorFlow の本家サイトの TF 2.0 Beta の以下のページを翻訳した上で適宜、補足説明したものです：
* サンプルコードの動作確認はしておりますが、必要な場合には適宜、追加改変しています。
* ご自由にリンクを張って頂いてかまいませんが、sales-info@classcat.com までご一報いただけると嬉しいです。
ガイド : Keras : カスタム callback
カスタム callback は Keras モデルを読み/変更することを含む、訓練、評価や推論の間の Keras モデルの挙動をカスタマイズするためのパワフルなツールです。サンプルは訓練進捗と結果が TensorBoard でエクスポートされて可視化できる tf.keras.callbacks.TensorBoard や、モデルが訓練の間に自動的にセーブされる tf.keras.callbacks.ModelCheckpoint 等を含みます。このガイドでは、Keras callback が何か、それはいつ呼び出されるか、それは何ができるか、そして貴方自身のものをどのように構築できるかを学習します。ガイドの終わりに向けて、貴方のカスタム callback を始めるための 2, 3 の単純な callback アプリケーションを作成するデモがあります。
Setup
from __future__ import absolute_import, division, print_function, unicode_literals !pip install -q tensorflow==2.0.0-beta1 import tensorflow as tf
Keras callback へのイントロダクション
Keras では、callback は特定の機能を提供するためにサブクラス化されることを意図した python クラスで、(batch/epoch start と end を含む) 訓練、テストと予測の様々な段階で呼び出されるメソッドのセットを持ちます。callback は訓練の間モデルの内部状態と統計上のビューを得るために有用です。tf.keras.Model.fit(), tf.keras.Model.evaluate() と tf.keras.Model.predict() メソッドのいずれかに (キーワード引数 callbacks として) callback のリストを渡すことができます。そして callback のメソッドが訓練/評価/推論の異なるステージで呼び出されます。
始めるために、tensorflow をインポートして単純な Sequential Keras モデルを定義しましょう :
# Define the Keras model to add callbacks to def get_model(): model = tf.keras.Sequential() model.add(tf.keras.layers.Dense(1, activation = 'linear', input_dim = 784)) model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.1), loss='mean_squared_error', metrics=['mae']) return model
それから訓練とテストのために Keras datasets API から MNIST データをロードします :
# Load example MNIST data and pre-process it
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 784).astype('float32') / 255
x_test = x_test.reshape(10000, 784).astype('float32') / 255
今、データの総てのバッチの開始と終わりを追跡するために単純なカスタム callback を定義します。これらの呼び出しの間、それは現在のバッチのインデックスをプリントします。
import datetime
class MyCustomCallback(tf.keras.callbacks.Callback):
def on_train_batch_begin(self, batch, logs=None):
print('Training: batch {} begins at {}'.format(batch, datetime.datetime.now().time()))
def on_train_batch_end(self, batch, logs=None):
print('Training: batch {} ends at {}'.format(batch, datetime.datetime.now().time()))
def on_test_batch_begin(self, batch, logs=None):
print('Evaluating: batch {} begins at {}'.format(batch, datetime.datetime.now().time()))
def on_test_batch_end(self, batch, logs=None):
print('Evaluating: batch {} ends at {}'.format(batch, datetime.datetime.now().time()))
tf.keras.Model.fit() のような model メソッドに callback を提供することはメソッドがそれらのステージで呼び出されることを保証します :
model = get_model()
_ = model.fit(x_train, y_train,
batch_size=64,
epochs=1,
steps_per_epoch=5,
verbose=0,
callbacks=[MyCustomCallback()])
Training: batch 0 begins at 15:12:18.987617 Training: batch 0 ends at 15:12:19.379915 Training: batch 1 begins at 15:12:19.380453 Training: batch 1 ends at 15:12:19.660801 Training: batch 2 begins at 15:12:19.661332 Training: batch 2 ends at 15:12:19.943575 Training: batch 3 begins at 15:12:19.944135 Training: batch 3 ends at 15:12:20.226410 Training: batch 4 begins at 15:12:20.226943 Training: batch 4 ends at 15:12:20.480595
callback を取る Model メソッド
ユーザは次のメソッド tf.keras.Model に callback のリストを供給できます :
固定数のエポックのためのモデル訓練する (データセットに渡る iteration、または Python generator によるバッチ毎に yield されたデータ)。
evaluate(), evaluate_generator()
与えられたデータかデータ generator のためのモデルを評価します。評価から損失とメトリック値を出力します。
predict(), predict_generator()
入力データやデータ generator のための出力予測を生成します。
_ = model.evaluate(x_test, y_test, batch_size=128, verbose=0, steps=5,
callbacks=[MyCustomCallback()])
Evaluating: batch 0 begins at 15:12:20.534213 Evaluating: batch 0 ends at 15:12:20.579315 Evaluating: batch 1 begins at 15:12:20.579683 Evaluating: batch 1 ends at 15:12:20.609873 Evaluating: batch 2 begins at 15:12:20.610312 Evaluating: batch 2 ends at 15:12:20.624019 Evaluating: batch 3 begins at 15:12:20.624337 Evaluating: batch 3 ends at 15:12:20.637601 Evaluating: batch 4 begins at 15:12:20.637857 Evaluating: batch 4 ends at 15:12:20.650800
callback メソッドの概要
訓練/テスト/予測のための一般的なメソッド
訓練、テストと予測のために、次のメソッドが override されるために提供されます。
on_(train|test|predict)_begin(self, logs=None)
fit/evaluate/predict の最初に呼び出されます。
on_(train|test|predict)_end(self, logs=None)
fit/evaluate/predict の最後に呼び出されます。
on_(train|test|predict)_batch_begin(self, batch, logs=None)
訓練/テスト/予測の間にバッチを処理する直前に呼び出されます。このメソッド内で、logs は batch と size を利用可能なキーとする辞書で、現在のバッチ数とバッチサイズを表わします。
on_(train|test|predict)_batch_end(self, batch, logs=None)
バッチの訓練/テスト/予測の終わりに呼び出されます。このメソッド内で、logs はステートフルなメトリクス結果を含む辞書です。
訓練用の特定のメソッド
加えて、訓練のために、次が提供されます。
on_epoch_begin(self, epoch, logs=None)
訓練の間にエポックの最初に呼び出されます。
on_epoch_end(self, epoch, logs=None)
訓練の間にエポックの最後に呼びされます。
logs 辞書の使用方法
logs 辞書はバッチかエポックの最後の loss 値と総てのメトリックを含みます。サンプルは loss と mean absolute error を含みます。
class LossAndErrorPrintingCallback(tf.keras.callbacks.Callback):
def on_train_batch_end(self, batch, logs=None):
print('For batch {}, loss is {:7.2f}.'.format(batch, logs['loss']))
def on_test_batch_end(self, batch, logs=None):
print('For batch {}, loss is {:7.2f}.'.format(batch, logs['loss']))
def on_epoch_end(self, epoch, logs=None):
print('The average loss for epoch {} is {:7.2f} and mean absolute error is {:7.2f}.'.format(epoch, logs['loss'], logs['mae']))
model = get_model()
_ = model.fit(x_train, y_train,
batch_size=64,
steps_per_epoch=5,
epochs=3,
verbose=0,
callbacks=[LossAndErrorPrintingCallback()])
For batch 0, loss is 27.82. For batch 1, loss is 924.20. For batch 2, loss is 25.03. For batch 3, loss is 8.72. For batch 4, loss is 7.15. The average loss for epoch 0 is 198.58 and mean absolute error is 8.32. For batch 0, loss is 6.34. For batch 1, loss is 5.79. For batch 2, loss is 5.41. For batch 3, loss is 5.14. For batch 4, loss is 4.95. The average loss for epoch 1 is 5.52 and mean absolute error is 1.94. For batch 0, loss is 4.80. For batch 1, loss is 4.69. For batch 2, loss is 4.61. For batch 3, loss is 4.53. For batch 4, loss is 4.46. The average loss for epoch 2 is 4.62 and mean absolute error is 1.74.
同様に、evaluate() 呼び出しで callback を提供できます。
_ = model.evaluate(x_test, y_test, batch_size=128, verbose=0, steps=20,
callbacks=[LossAndErrorPrintingCallback()])
For batch 0, loss is 4.16. For batch 1, loss is 4.16. For batch 2, loss is 4.16. For batch 3, loss is 4.16. For batch 4, loss is 4.16. For batch 5, loss is 4.16. For batch 6, loss is 4.16. For batch 7, loss is 4.16. For batch 8, loss is 4.16. For batch 9, loss is 4.16. For batch 10, loss is 4.16. For batch 11, loss is 4.16. For batch 12, loss is 4.16. For batch 13, loss is 4.16. For batch 14, loss is 4.16. For batch 15, loss is 4.16. For batch 16, loss is 4.16. For batch 17, loss is 4.16. For batch 18, loss is 4.16. For batch 19, loss is 4.16.
Keras callback アプリケーションのサンプル
次のセクションは貴方に単純な Callback アプリケーションを作成する案内をします。
最小損失における Early stopping
最初の例は Callback の作成を紹介します、これは損失の最小に達したとき属性 model.stop_training (boolean) を変えることにより Keras 訓練を停止します。オプションで、訓練がそれが最終的に停止する前に幾つのエポックを待つべきかを指定するための引数 patience をユーザは提供できます。
tf.keras.callbacks.EarlyStopping はより複雑で一般的な実装を提供します。
import numpy as np
class EarlyStoppingAtMinLoss(tf.keras.callbacks.Callback):
"""Stop training when the loss is at its min, i.e. the loss stops decreasing.
Arguments:
patience: Number of epochs to wait after min has been hit. After this
number of no improvement, training stops.
"""
def __init__(self, patience=0):
super(EarlyStoppingAtMinLoss, self).__init__()
self.patience = patience
# best_weights to store the weights at which the minimum loss occurs.
self.best_weights = None
def on_train_begin(self, logs=None):
# The number of epoch it has waited when loss is no longer minimum.
self.wait = 0
# The epoch the training stops at.
self.stopped_epoch = 0
# Initialize the best as infinity.
self.best = np.Inf
def on_epoch_end(self, epoch, logs=None):
current = logs.get('loss')
if np.less(current, self.best):
self.best = current
self.wait = 0
# Record the best weights if current results is better (less).
self.best_weights = self.model.get_weights()
else:
self.wait += 1
if self.wait >= self.patience:
self.stopped_epoch = epoch
self.model.stop_training = True
print('Restoring model weights from the end of the best epoch.')
self.model.set_weights(self.best_weights)
def on_train_end(self, logs=None):
if self.stopped_epoch > 0:
print('Epoch %05d: early stopping' % (self.stopped_epoch + 1))
model = get_model()
_ = model.fit(x_train, y_train,
batch_size=64,
steps_per_epoch=5,
epochs=30,
verbose=0,
callbacks=[LossAndErrorPrintingCallback(), EarlyStoppingAtMinLoss()])
For batch 0, loss is 27.10. For batch 1, loss is 923.16. For batch 2, loss is 24.37. For batch 3, loss is 8.33. For batch 4, loss is 6.95. The average loss for epoch 0 is 197.98 and mean absolute error is 8.29. For batch 0, loss is 6.27. For batch 1, loss is 5.80. For batch 2, loss is 5.47. For batch 3, loss is 5.23. For batch 4, loss is 5.05. The average loss for epoch 1 is 5.56 and mean absolute error is 1.96. For batch 0, loss is 4.91. For batch 1, loss is 4.79. For batch 2, loss is 4.70. For batch 3, loss is 4.62. For batch 4, loss is 4.55. The average loss for epoch 2 is 4.71 and mean absolute error is 1.77. For batch 0, loss is 4.48. For batch 1, loss is 4.42. For batch 2, loss is 4.36. For batch 3, loss is 4.31. For batch 4, loss is 4.26. The average loss for epoch 3 is 4.36 and mean absolute error is 1.68. For batch 0, loss is 4.21. For batch 1, loss is 4.17. For batch 2, loss is 4.13. For batch 3, loss is 4.13. For batch 4, loss is 4.35. The average loss for epoch 4 is 4.20 and mean absolute error is 1.63. For batch 0, loss is 6.24. For batch 1, loss is 22.54. For batch 2, loss is 133.97. For batch 3, loss is 249.51. For batch 4, loss is 108.04. The average loss for epoch 5 is 104.06 and mean absolute error is 8.31. Restoring model weights from the end of the best epoch. Epoch 00006: early stopping
学習率スケジューリング
モデル訓練で一般に成される一つのことはより多くのエポックが過ぎるにつれて学習率を変更することです。Keras バックエンドは get_value api を公開しています、これは値を設定するために使用できます。この例では、カスタム Callback が学習率を動的に変更するためにどのように使用できるかを示しています。
tf.keras.callbacks.LearningRateScheduler はより一般的な実装を提供します。
class LearningRateScheduler(tf.keras.callbacks.Callback):
"""Learning rate scheduler which sets the learning rate according to schedule.
Arguments:
schedule: a function that takes an epoch index
(integer, indexed from 0) and current learning rate
as inputs and returns a new learning rate as output (float).
"""
def __init__(self, schedule):
super(LearningRateScheduler, self).__init__()
self.schedule = schedule
def on_epoch_begin(self, epoch, logs=None):
if not hasattr(self.model.optimizer, 'lr'):
raise ValueError('Optimizer must have a "lr" attribute.')
# Get the current learning rate from model's optimizer.
lr = float(tf.keras.backend.get_value(self.model.optimizer.lr))
# Call schedule function to get the scheduled learning rate.
scheduled_lr = self.schedule(epoch, lr)
# Set the value back to the optimizer before this epoch starts
tf.keras.backend.set_value(self.model.optimizer.lr, scheduled_lr)
print('\nEpoch %05d: Learning rate is %6.4f.' % (epoch, scheduled_lr))
LR_SCHEDULE = [
# (epoch to start, learning rate) tuples
(3, 0.05), (6, 0.01), (9, 0.005), (12, 0.001)
]
def lr_schedule(epoch, lr):
"""Helper function to retrieve the scheduled learning rate based on epoch."""
if epoch < LR_SCHEDULE[0][0] or epoch > LR_SCHEDULE[-1][0]:
return lr
for i in range(len(LR_SCHEDULE)):
if epoch == LR_SCHEDULE[i][0]:
return LR_SCHEDULE[i][1]
return lr
model = get_model()
_ = model.fit(x_train, y_train,
batch_size=64,
steps_per_epoch=5,
epochs=15,
verbose=0,
callbacks=[LossAndErrorPrintingCallback(), LearningRateScheduler(lr_schedule)])
Epoch 00000: Learning rate is 0.1000. For batch 0, loss is 30.28. For batch 1, loss is 905.91. For batch 2, loss is 26.87. For batch 3, loss is 8.73. For batch 4, loss is 7.09. The average loss for epoch 0 is 195.78 and mean absolute error is 8.35. Epoch 00001: Learning rate is 0.1000. For batch 0, loss is 6.30. For batch 1, loss is 5.76. For batch 2, loss is 5.39. For batch 3, loss is 5.13. For batch 4, loss is 4.93. The average loss for epoch 1 is 5.50 and mean absolute error is 1.95. Epoch 00002: Learning rate is 0.1000. For batch 0, loss is 4.79. For batch 1, loss is 4.68. For batch 2, loss is 4.59. For batch 3, loss is 4.52. For batch 4, loss is 4.45. The average loss for epoch 2 is 4.61 and mean absolute error is 1.74. Epoch 00003: Learning rate is 0.0500. For batch 0, loss is 4.39. For batch 1, loss is 4.36. For batch 2, loss is 4.34. For batch 3, loss is 4.31. For batch 4, loss is 4.28. The average loss for epoch 3 is 4.34 and mean absolute error is 1.67. Epoch 00004: Learning rate is 0.0500. For batch 0, loss is 4.26. For batch 1, loss is 4.23. For batch 2, loss is 4.21. For batch 3, loss is 4.18. For batch 4, loss is 4.16. The average loss for epoch 4 is 4.21 and mean absolute error is 1.63. Epoch 00005: Learning rate is 0.0500. For batch 0, loss is 4.13. For batch 1, loss is 4.11. For batch 2, loss is 4.09. For batch 3, loss is 4.07. For batch 4, loss is 4.05. The average loss for epoch 5 is 4.09 and mean absolute error is 1.60. Epoch 00006: Learning rate is 0.0100. For batch 0, loss is 4.02. For batch 1, loss is 4.02. For batch 2, loss is 4.02. For batch 3, loss is 4.01. For batch 4, loss is 4.01. The average loss for epoch 6 is 4.02 and mean absolute error is 1.58. Epoch 00007: Learning rate is 0.0100. For batch 0, loss is 4.00. For batch 1, loss is 4.00. For batch 2, loss is 3.99. For batch 3, loss is 3.99. For batch 4, loss is 3.98. The average loss for epoch 7 is 3.99 and mean absolute error is 1.57. Epoch 00008: Learning rate is 0.0100. For batch 0, loss is 3.98. For batch 1, loss is 3.97. For batch 2, loss is 3.97. For batch 3, loss is 3.96. For batch 4, loss is 3.95. The average loss for epoch 8 is 3.97 and mean absolute error is 1.57. Epoch 00009: Learning rate is 0.0050. For batch 0, loss is 3.95. For batch 1, loss is 3.94. For batch 2, loss is 3.94. For batch 3, loss is 3.94. For batch 4, loss is 3.93. The average loss for epoch 9 is 3.94 and mean absolute error is 1.56. Epoch 00010: Learning rate is 0.0050. For batch 0, loss is 3.93. For batch 1, loss is 3.93. For batch 2, loss is 3.92. For batch 3, loss is 3.92. For batch 4, loss is 3.91. The average loss for epoch 10 is 3.92 and mean absolute error is 1.55. Epoch 00011: Learning rate is 0.0050. For batch 0, loss is 3.91. For batch 1, loss is 3.91. For batch 2, loss is 3.90. For batch 3, loss is 3.90. For batch 4, loss is 3.89. The average loss for epoch 11 is 3.90 and mean absolute error is 1.55. Epoch 00012: Learning rate is 0.0010. For batch 0, loss is 3.89. For batch 1, loss is 3.89. For batch 2, loss is 3.89. For batch 3, loss is 3.88. For batch 4, loss is 3.88. The average loss for epoch 12 is 3.89 and mean absolute error is 1.54. Epoch 00013: Learning rate is 0.0010. For batch 0, loss is 3.88. For batch 1, loss is 3.88. For batch 2, loss is 3.88. For batch 3, loss is 3.88. For batch 4, loss is 3.88. The average loss for epoch 13 is 3.88 and mean absolute error is 1.54. Epoch 00014: Learning rate is 0.0010. For batch 0, loss is 3.88. For batch 1, loss is 3.87. For batch 2, loss is 3.87. For batch 3, loss is 3.87. For batch 4, loss is 3.87. The average loss for epoch 14 is 3.87 and mean absolute error is 1.54.
標準的な Keras callback
api doc を訪ねる ことにより既存の Keras callback を確実に調べてください。アプリケーションは CSV へのロギング、モデルのセーブ、TensorBoard 上の可視化等々を含みます。
以上