TensorFlow 2.0 : ガイド : Keras :- Keras カスタム callback (翻訳/解説)

翻訳 : (株)クラスキャット セールスインフォメーション
作成日時 : 12/20/2019

* 本ページは、TensorFlow org サイトの Guide – Keras の以下のページを翻訳した上で
適宜、補足説明したものです：

• Keras custom callbacks

* サンプルコードの動作確認はしておりますが、必要な場合には適宜、追加改変しています。
* ご自由にリンクを張って頂いてかまいませんが、sales-info@classcat.com までご一報いただけると嬉しいです。

 

 

ガイド : Keras :- Keras カスタム callback

カスタム callback は Keras モデルを読み/変更することを含む、訓練、評価や推論の間の Keras モデルの動作をカスタマイズするためのパワフルなツールです。サンプルは訓練進捗と結果が TensorBoard でエクスポートされて可視化できる tf.keras.callbacks.TensorBoard や、モデルが訓練の間に自動的にセーブされる tf.keras.callbacks.ModelCheckpoint 等を含みます。このガイドでは、Keras callback が何か、それはいつ呼び出されるか、それは何ができるか、そして貴方自身のものをどのように構築できるかを学習します。ガイドの終わりに向けて、貴方のカスタム callback を始めるために 2, 3 の単純な callback アプリケーションを作成するデモがあります。

 

Setup

from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow as tf

 

Keras callback へのイントロダクション

Keras では、Callback は特定の機能を提供するためにサブクラス化されることを意図した python クラスで、(batch/epoch start と end を含む) 訓練、テストと予測の様々な段階で呼び出されるメソッドのセットを持ちます。callback は訓練の間モデルの内部状態と統計上のビューを得るために有用です。tf.keras.Model.fit(), tf.keras.Model.evaluate() と tf.keras.Model.predict() メソッドのいずれかに (キーワード引数 callbacks として) callback のリストを渡すことができます。そして callback のメソッドが訓練/評価/推論の異なるステージで呼び出されます。

始めるために、tensorflow をインポートして単純な Sequential Keras モデルを定義しましょう :

# Define the Keras model to add callbacks to
def get_model():
  model = tf.keras.Sequential()
  model.add(tf.keras.layers.Dense(1, activation = 'linear', input_dim = 784))
  model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.1), loss='mean_squared_error', metrics=['mae'])
  return model

それから訓練とテストのために Keras datasets API から MNIST データをロードします :

# Load example MNIST data and pre-process it
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 784).astype('float32') / 255
x_test = x_test.reshape(10000, 784).astype('float32') / 255

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step

今、データの総てのバッチの開始と終わりを追跡するために単純なカスタム callback を定義します。それらの呼び出しの間、それは現在のバッチのインデックスをプリントします。

import datetime

class MyCustomCallback(tf.keras.callbacks.Callback):

  def on_train_batch_begin(self, batch, logs=None):
    print('Training: batch {} begins at {}'.format(batch, datetime.datetime.now().time()))

  def on_train_batch_end(self, batch, logs=None):
    print('Training: batch {} ends at {}'.format(batch, datetime.datetime.now().time()))

  def on_test_batch_begin(self, batch, logs=None):
    print('Evaluating: batch {} begins at {}'.format(batch, datetime.datetime.now().time()))

  def on_test_batch_end(self, batch, logs=None):
    print('Evaluating: batch {} ends at {}'.format(batch, datetime.datetime.now().time()))

tf.keras.Model.fit() のような model メソッドに callback を提供することはメソッドがそれらのステージで呼び出されることを保証します :

model = get_model()
_ = model.fit(x_train, y_train,
          batch_size=64,
          epochs=1,
          steps_per_epoch=5,
          verbose=0,
          callbacks=[MyCustomCallback()])

Training: batch 0 begins at 02:22:16.589465
Training: batch 0 ends at 02:22:17.305300
Training: batch 1 begins at 02:22:17.305645
Training: batch 1 ends at 02:22:17.308415
Training: batch 2 begins at 02:22:17.308644
Training: batch 2 ends at 02:22:17.310919
Training: batch 3 begins at 02:22:17.311119
Training: batch 3 ends at 02:22:17.313492
Training: batch 4 begins at 02:22:17.313708
Training: batch 4 ends at 02:22:17.316000

 

callback を取る Model メソッド

ユーザは次の tf.keras.Model メソッドに callback のリストを供給できます :

fit(), fit_generator()

固定数のエポックのためにモデルを訓練する (データセットに渡る iteration、または Python generator によるバッチ毎に yield されたデータ)。

evaluate(), evaluate_generator()

与えられたデータかデータ generator のためのモデルを評価します。評価からの損失とメトリック値を出力します。

predict(), predict_generator()

入力データやデータ generator のための出力予測を生成します。

_ = model.evaluate(x_test, y_test, batch_size=128, verbose=0, steps=5,
          callbacks=[MyCustomCallback()])

Evaluating: batch 0 begins at 02:22:17.371868
Evaluating: batch 0 ends at 02:22:17.426746
Evaluating: batch 1 begins at 02:22:17.427181
Evaluating: batch 1 ends at 02:22:17.429044
Evaluating: batch 2 begins at 02:22:17.429259
Evaluating: batch 2 ends at 02:22:17.430987
Evaluating: batch 3 begins at 02:22:17.431363
Evaluating: batch 3 ends at 02:22:17.433296
Evaluating: batch 4 begins at 02:22:17.433541
Evaluating: batch 4 ends at 02:22:17.435420

 

callback メソッドの概要

訓練/テスト/予測のための一般的なメソッド

訓練、テストと予測のために、次のメソッドが override されるために提供されます。

on_(train|test|predict)_begin(self, logs=None)

fit/evaluate/predict の最初に呼び出されます。

on_(train|test|predict)_end(self, logs=None)

fit/evaluate/predict の最後に呼び出されます。

on_(train|test|predict)_batch_begin(self, batch, logs=None)

訓練/テスト/予測の間にバッチを処理する直前に呼び出されます。このメソッド内で、logs は batch と size を利用可能なキーとする辞書で、現在のバッチ数とバッチサイズを表わします。

on_(train|test|predict)_batch_end(self, batch, logs=None)

バッチの訓練/テスト/予測の終わりに呼び出されます。このメソッド内で、logs はステートフルなメトリクス結果を含む辞書です。

 

訓練 (用の) 特定メソッド

加えて、訓練のために、次が提供されます。

on_epoch_begin(self, epoch, logs=None)

訓練の間にエポックの最初に呼び出されます。

on_epoch_end(self, epoch, logs=None)

訓練の間にエポックの最後に呼びされます。

 

logs 辞書の使用方法

logs 辞書はバッチかエポックの最後の loss 値と総てのメトリックを含みます。サンプルは loss と mean absolute error を含みます。

class LossAndErrorPrintingCallback(tf.keras.callbacks.Callback):

  def on_train_batch_end(self, batch, logs=None):
    print('For batch {}, loss is {:7.2f}.'.format(batch, logs['loss']))

  def on_test_batch_end(self, batch, logs=None):
    print('For batch {}, loss is {:7.2f}.'.format(batch, logs['loss']))

  def on_epoch_end(self, epoch, logs=None):
    print('The average loss for epoch {} is {:7.2f} and mean absolute error is {:7.2f}.'.format(epoch, logs['loss'], logs['mae']))

model = get_model()
_ = model.fit(x_train, y_train,
          batch_size=64,
          steps_per_epoch=5,
          epochs=3,
          verbose=0,
          callbacks=[LossAndErrorPrintingCallback()])

For batch 0, loss is   28.35.
For batch 1, loss is 1127.19.
For batch 2, loss is   19.46.
For batch 3, loss is   11.65.
For batch 4, loss is    6.62.
The average loss for epoch 0 is  238.65 and mean absolute error is    8.81.
For batch 0, loss is    6.15.
For batch 1, loss is    5.54.
For batch 2, loss is    6.50.
For batch 3, loss is    5.95.
For batch 4, loss is    4.40.
The average loss for epoch 1 is    5.71 and mean absolute error is    1.99.
For batch 0, loss is    4.47.
For batch 1, loss is    3.14.
For batch 2, loss is    4.38.
For batch 3, loss is    5.48.
For batch 4, loss is    7.15.
The average loss for epoch 2 is    4.92 and mean absolute error is    1.81.

同様に、evaluate() 呼び出しで callback を提供できます。

_ = model.evaluate(x_test, y_test, batch_size=128, verbose=0, steps=20,
          callbacks=[LossAndErrorPrintingCallback()])

For batch 0, loss is    7.47.
For batch 1, loss is    5.97.
For batch 2, loss is    7.07.
For batch 3, loss is    6.74.
For batch 4, loss is    7.96.
For batch 5, loss is    6.84.
For batch 6, loss is    7.16.
For batch 7, loss is    6.78.
For batch 8, loss is    6.86.
For batch 9, loss is    8.03.
For batch 10, loss is    7.86.
For batch 11, loss is    8.37.
For batch 12, loss is    8.22.
For batch 13, loss is    8.79.
For batch 14, loss is    8.06.
For batch 15, loss is    6.77.
For batch 16, loss is    8.84.
For batch 17, loss is    9.73.
For batch 18, loss is    8.76.
For batch 19, loss is    6.63.

 

Keras callback アプリケーションのサンプル

次のセクションは貴方に単純な Callback アプリケーションを作成する案内をします。

 

最小損失における Early stopping

最初の例は Callback の作成を紹介します、これは損失の最小に達したとき属性 model.stop_training (boolean) を変えることにより Keras 訓練を停止します。オプションで、訓練がそれが最終的に停止する前に幾つのエポックを待つべきかを指定するための引数 patience をユーザは提供できます。

tf.keras.callbacks.EarlyStopping はより完全で一般的な実装を提供します。

import numpy as np

class EarlyStoppingAtMinLoss(tf.keras.callbacks.Callback):
  """Stop training when the loss is at its min, i.e. the loss stops decreasing.

  Arguments:
      patience: Number of epochs to wait after min has been hit. After this
      number of no improvement, training stops.
  """

  def __init__(self, patience=0):
    super(EarlyStoppingAtMinLoss, self).__init__()

    self.patience = patience

    # best_weights to store the weights at which the minimum loss occurs.
    self.best_weights = None

  def on_train_begin(self, logs=None):
    # The number of epoch it has waited when loss is no longer minimum.
    self.wait = 0
    # The epoch the training stops at.
    self.stopped_epoch = 0
    # Initialize the best as infinity.
    self.best = np.Inf

  def on_epoch_end(self, epoch, logs=None):
    current = logs.get('loss')
    if np.less(current, self.best):
      self.best = current
      self.wait = 0
      # Record the best weights if current results is better (less).
      self.best_weights = self.model.get_weights()
    else:
      self.wait += 1
      if self.wait >= self.patience:
        self.stopped_epoch = epoch
        self.model.stop_training = True
        print('Restoring model weights from the end of the best epoch.')
        self.model.set_weights(self.best_weights)

  def on_train_end(self, logs=None):
    if self.stopped_epoch > 0:
      print('Epoch %05d: early stopping' % (self.stopped_epoch + 1))

model = get_model()
_ = model.fit(x_train, y_train,
          batch_size=64,
          steps_per_epoch=5,
          epochs=30,
          verbose=0,
          callbacks=[LossAndErrorPrintingCallback(), EarlyStoppingAtMinLoss()])

For batch 0, loss is   26.52.
For batch 1, loss is  931.17.
For batch 2, loss is   27.66.
For batch 3, loss is   12.00.
For batch 4, loss is    8.24.
The average loss for epoch 0 is  201.12 and mean absolute error is    8.41.
For batch 0, loss is    4.61.
For batch 1, loss is    5.70.
For batch 2, loss is    5.74.
For batch 3, loss is    4.41.
For batch 4, loss is    4.12.
The average loss for epoch 1 is    4.92 and mean absolute error is    1.77.
For batch 0, loss is    6.30.
For batch 1, loss is    5.78.
For batch 2, loss is    4.86.
For batch 3, loss is    4.71.
For batch 4, loss is    6.14.
The average loss for epoch 2 is    5.56 and mean absolute error is    1.91.
Restoring model weights from the end of the best epoch.
Epoch 00003: early stopping

 

学習率スケジューリング

モデル訓練で一般に成される一つのことはより多くのエポックが過ぎるにつれて学習率を変更することです。Keras バックエンドは get_value API を公開しています、これは変数を設定するために使用できます。この例では、カスタム Callback が学習率を動的に変更するためにどのように使用できるかを示しています。

Note: これは単なるサンプル実装です、より一般的な実装については callbacks.LearningRateScheduler と keras.optimizers.schedules を見てください。

class LearningRateScheduler(tf.keras.callbacks.Callback):
  """Learning rate scheduler which sets the learning rate according to schedule.

  Arguments:
      schedule: a function that takes an epoch index
          (integer, indexed from 0) and current learning rate
          as inputs and returns a new learning rate as output (float).
  """

  def __init__(self, schedule):
    super(LearningRateScheduler, self).__init__()
    self.schedule = schedule

  def on_epoch_begin(self, epoch, logs=None):
    if not hasattr(self.model.optimizer, 'lr'):
      raise ValueError('Optimizer must have a "lr" attribute.')
    # Get the current learning rate from model's optimizer.
    lr = float(tf.keras.backend.get_value(self.model.optimizer.lr))
    # Call schedule function to get the scheduled learning rate.
    scheduled_lr = self.schedule(epoch, lr)
    # Set the value back to the optimizer before this epoch starts
    tf.keras.backend.set_value(self.model.optimizer.lr, scheduled_lr)
    print('\nEpoch %05d: Learning rate is %6.4f.' % (epoch, scheduled_lr))

LR_SCHEDULE = [
    # (epoch to start, learning rate) tuples
    (3, 0.05), (6, 0.01), (9, 0.005), (12, 0.001)
]

def lr_schedule(epoch, lr):
  """Helper function to retrieve the scheduled learning rate based on epoch."""
  if epoch < LR_SCHEDULE[0][0] or epoch > LR_SCHEDULE[-1][0]:
    return lr
  for i in range(len(LR_SCHEDULE)):
    if epoch == LR_SCHEDULE[i][0]:
      return LR_SCHEDULE[i][1]
  return lr

model = get_model()
_ = model.fit(x_train, y_train,
          batch_size=64,
          steps_per_epoch=5,
          epochs=15,
          verbose=0,
          callbacks=[LossAndErrorPrintingCallback(), LearningRateScheduler(lr_schedule)])

Epoch 00000: Learning rate is 0.1000.
For batch 0, loss is   25.77.
For batch 1, loss is 1110.67.
For batch 2, loss is   19.85.
For batch 3, loss is    6.15.
For batch 4, loss is   10.22.
The average loss for epoch 0 is  234.53 and mean absolute error is    8.76.

Epoch 00001: Learning rate is 0.1000.
For batch 0, loss is    5.19.
For batch 1, loss is    6.96.
For batch 2, loss is    5.57.
For batch 3, loss is    4.37.
For batch 4, loss is    4.64.
The average loss for epoch 1 is    5.35 and mean absolute error is    1.88.

Epoch 00002: Learning rate is 0.1000.
For batch 0, loss is    6.18.
For batch 1, loss is    4.76.
For batch 2, loss is    4.18.
For batch 3, loss is    6.45.
For batch 4, loss is    5.20.
The average loss for epoch 2 is    5.36 and mean absolute error is    1.85.

Epoch 00003: Learning rate is 0.0500.
For batch 0, loss is    4.51.
For batch 1, loss is    3.69.
For batch 2, loss is    5.51.
For batch 3, loss is    5.06.
For batch 4, loss is    4.62.
The average loss for epoch 3 is    4.68 and mean absolute error is    1.74.

Epoch 00004: Learning rate is 0.0500.
For batch 0, loss is    5.08.
For batch 1, loss is    3.74.
For batch 2, loss is    7.16.
For batch 3, loss is    3.96.
For batch 4, loss is    3.88.
The average loss for epoch 4 is    4.76 and mean absolute error is    1.77.

Epoch 00005: Learning rate is 0.0500.
For batch 0, loss is    4.24.
For batch 1, loss is    5.68.
For batch 2, loss is    5.60.
For batch 3, loss is    5.19.
For batch 4, loss is    5.92.
The average loss for epoch 5 is    5.33 and mean absolute error is    1.82.

Epoch 00006: Learning rate is 0.0100.
For batch 0, loss is    8.06.
For batch 1, loss is    6.06.
For batch 2, loss is    5.00.
For batch 3, loss is    4.51.
For batch 4, loss is    4.58.
The average loss for epoch 6 is    5.64 and mean absolute error is    1.87.

Epoch 00007: Learning rate is 0.0100.
For batch 0, loss is    2.86.
For batch 1, loss is    4.01.
For batch 2, loss is    3.29.
For batch 3, loss is    4.14.
For batch 4, loss is    2.62.
The average loss for epoch 7 is    3.38 and mean absolute error is    1.44.

Epoch 00008: Learning rate is 0.0100.
For batch 0, loss is    4.35.
For batch 1, loss is    3.27.
For batch 2, loss is    4.16.
For batch 3, loss is    3.73.
For batch 4, loss is    3.89.
The average loss for epoch 8 is    3.88 and mean absolute error is    1.53.

Epoch 00009: Learning rate is 0.0050.
For batch 0, loss is    3.33.
For batch 1, loss is    4.63.
For batch 2, loss is    4.94.
For batch 3, loss is    4.90.
For batch 4, loss is    3.78.
The average loss for epoch 9 is    4.32 and mean absolute error is    1.67.

Epoch 00010: Learning rate is 0.0050.
For batch 0, loss is    3.38.
For batch 1, loss is    3.47.
For batch 2, loss is    5.72.
For batch 3, loss is    5.28.
For batch 4, loss is    3.75.
The average loss for epoch 10 is    4.32 and mean absolute error is    1.65.

Epoch 00011: Learning rate is 0.0050.
For batch 0, loss is    3.89.
For batch 1, loss is    3.62.
For batch 2, loss is    5.97.
For batch 3, loss is    5.10.
For batch 4, loss is    3.76.
The average loss for epoch 11 is    4.47 and mean absolute error is    1.64.

Epoch 00012: Learning rate is 0.0010.
For batch 0, loss is    4.98.
For batch 1, loss is    3.61.
For batch 2, loss is    2.90.
For batch 3, loss is    3.83.
For batch 4, loss is    3.35.
The average loss for epoch 12 is    3.74 and mean absolute error is    1.56.

Epoch 00013: Learning rate is 0.0010.
For batch 0, loss is    4.66.
For batch 1, loss is    3.20.
For batch 2, loss is    3.53.
For batch 3, loss is    3.40.
For batch 4, loss is    3.66.
The average loss for epoch 13 is    3.69 and mean absolute error is    1.53.

Epoch 00014: Learning rate is 0.0010.
For batch 0, loss is    2.49.
For batch 1, loss is    3.71.
For batch 2, loss is    4.35.
For batch 3, loss is    3.80.
For batch 4, loss is    5.67.
The average loss for epoch 14 is    4.01 and mean absolute error is    1.55.

 

標準的な Keras callback

API doc を訪ねる ことにより既存の Keras callback を確実に調べてください。アプリケーションは CSV へのロギング、モデルのセーブ、TensorBoard 上の可視化等々それ以上を含みます。

以上