Keras 2 : examples : Transformer による動画分類 (翻訳/解説)

翻訳 : (株)クラスキャット セールスインフォメーション
作成日時 : 12/26/2021 (keras 2.7.0)

* 本ページは、Keras の以下のドキュメントを翻訳した上で適宜、補足説明したものです：

• Code examples : Computer Vision : Video Classification with Transformers (Author: Sayak Paul)

* サンプルコードの動作確認はしておりますが、必要な場合には適宜、追加改変しています。
* ご自由にリンクを張って頂いてかまいませんが、sales-info@classcat.com までご一報いただけると嬉しいです。

 

 

 

Keras 2 : examples : Transformer による動画分類

Description: ハイブリッド transformer で動画分類器を訓練する。

 
このサンプルは CNN-RNN アーキテクチャによる動画分類 サンプルへのフォローアップです。今回は、動画を分類するために Transformer ベースのモデル ( Vaswani et al. ) を使用していきます。Transformers (with コード) へのイントロダクションが必要な場合には この本の章 を追うことができます。このサンプルを読んだ後、CNN 特徴マップ上で動作する動画分類のためのハイブリッド Transformer ベースのモデルを開発方法を知ることができます。

このサンプルは TensorFlow 2.5 またはそれ以上と、TensorFlow Dos を必要とします、これは次のコマンドを使用してインストールできます :

!pip install -q git+https://github.com/tensorflow/docs

 

データ・コレクション

このサンプルに 先行するサンプル で行なわれたように、有名なベンチマーク・データセットの UCF101 データセット のサブサンプリングされたバージョンを使用していきます。より大きなサブサンプルやデータセット全体で操作したい場合には、このノートブック を参照してください。

!wget -q https://git.io/JGc31 -O ucf101_top5.tar.gz
!tar xf ucf101_top5.tar.gz

 

セットアップ

from tensorflow_docs.vis import embed
from tensorflow.keras import layers
from tensorflow import keras

import matplotlib.pyplot as plt
import tensorflow as tf
import pandas as pd
import numpy as np
import imageio
import cv2
import os

 

ハイパーパラメータの定義

MAX_SEQ_LENGTH = 20
NUM_FEATURES = 1024
IMG_SIZE = 128

EPOCHS = 5

 

データの準備

このサンプルでは殆ど同じデータ準備ステップに従っていきます、以下の変更を除いてです :

• 計算を高速化するために画像サイズを 224×224 の代わりに 128×128 に縮小します。
  

• 事前訓練済みの InceptionV3 ネットワークを使用する代わりに、特徴抽出のために事前訓練済みの DenseNet121 を使用します。
  

• 短い動画は長さ MAX_SEQ_LENGTH まで直接パディングします。

最初に、DataFrame にロードしましょう。

train_df = pd.read_csv("train.csv")
test_df = pd.read_csv("test.csv")

print(f"Total videos for training: {len(train_df)}")
print(f"Total videos for testing: {len(test_df)}")

center_crop_layer = layers.CenterCrop(IMG_SIZE, IMG_SIZE)


def crop_center(frame):
    cropped = center_crop_layer(frame[None, ...])
    cropped = cropped.numpy().squeeze()
    return cropped


# Following method is modified from this tutorial:
# https://www.tensorflow.org/hub/tutorials/action_recognition_with_tf_hub
def load_video(path, max_frames=0):
    cap = cv2.VideoCapture(path)
    frames = []
    try:
        while True:
            ret, frame = cap.read()
            if not ret:
                break
            frame = crop_center(frame)
            frame = frame[:, :, [2, 1, 0]]
            frames.append(frame)

            if len(frames) == max_frames:
                break
    finally:
        cap.release()
    return np.array(frames)


def build_feature_extractor():
    feature_extractor = keras.applications.DenseNet121(
        weights="imagenet",
        include_top=False,
        pooling="avg",
        input_shape=(IMG_SIZE, IMG_SIZE, 3),
    )
    preprocess_input = keras.applications.densenet.preprocess_input

    inputs = keras.Input((IMG_SIZE, IMG_SIZE, 3))
    preprocessed = preprocess_input(inputs)

    outputs = feature_extractor(preprocessed)
    return keras.Model(inputs, outputs, name="feature_extractor")


feature_extractor = build_feature_extractor()


# Label preprocessing with StringLookup.
label_processor = keras.layers.StringLookup(
    num_oov_indices=0, vocabulary=np.unique(train_df["tag"]), mask_token=None
)
print(label_processor.get_vocabulary())


def prepare_all_videos(df, root_dir):
    num_samples = len(df)
    video_paths = df["video_name"].values.tolist()
    labels = df["tag"].values
    labels = label_processor(labels[..., None]).numpy()

    # `frame_features` are what we will feed to our sequence model.
    frame_features = np.zeros(
        shape=(num_samples, MAX_SEQ_LENGTH, NUM_FEATURES), dtype="float32"
    )

    # For each video.
    for idx, path in enumerate(video_paths):
        # Gather all its frames and add a batch dimension.
        frames = load_video(os.path.join(root_dir, path))

        # Pad shorter videos.
        if len(frames) < MAX_SEQ_LENGTH:
            diff = MAX_SEQ_LENGTH - len(frames)
            padding = np.zeros((diff, IMG_SIZE, IMG_SIZE, 3))
            frames = np.concatenate(frames, padding)

        frames = frames[None, ...]

        # Initialize placeholder to store the features of the current video.
        temp_frame_features = np.zeros(
            shape=(1, MAX_SEQ_LENGTH, NUM_FEATURES), dtype="float32"
        )

        # Extract features from the frames of the current video.
        for i, batch in enumerate(frames):
            video_length = batch.shape[0]
            length = min(MAX_SEQ_LENGTH, video_length)
            for j in range(length):
                if np.mean(batch[j, :]) > 0.0:
                    temp_frame_features[i, j, :] = feature_extractor.predict(
                        batch[None, j, :]
                    )

                else:
                    temp_frame_features[i, j, :] = 0.0

        frame_features[idx,] = temp_frame_features.squeeze()

    return frame_features, labels

Total videos for training: 594
Total videos for testing: 224

2021-09-14 13:26:28.169035: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-14 13:26:28.169629: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-09-14 13:26:28.169696: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2021-09-14 13:26:28.169746: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2021-09-14 13:26:28.179403: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2021-09-14 13:26:28.179462: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2021-09-14 13:26:28.180051: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-09-14 13:26:28.180325: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/densenet/densenet121_weights_tf_dim_ordering_tf_kernels_notop.h5
29089792/29084464 [==============================] - 1s 0us/step
29097984/29084464 [==============================] - 1s 0us/step
['CricketShot', 'PlayingCello', 'Punch', 'ShavingBeard', 'TennisSwing']

train_df と test_df に対する prepare_all_videos() の呼び出しは終了まで ~20 分かかります。そのため、時間節約のため、ここでは既に前処理された NumPy 配列をダウンロードします。

!wget -q https://git.io/JZmf4 -O top5_data_prepared.tar.gz
!tar xf top5_data_prepared.tar.gz

train_data, train_labels = np.load("train_data.npy"), np.load("train_labels.npy")
test_data, test_labels = np.load("test_data.npy"), np.load("test_labels.npy")

print(f"Frame features in train set: {train_data.shape}")

Frame features in train set: (594, 20, 1024)

 

Transformer ベース・モデルの構築

Deep Learning with Python (Second ed.) by François Chollet の この本の章 で共有されるコードの上に構築していきます。

まず、Transformer の基本ブロックを形成する自己注意層は順序不可知です。動画はフレームの順序付けられたシークエンスですから、私達の Transformer モデルは順序情報を考慮する必要があります。これを 位置エンコーディング を通して行ないます。動画内に存在するフレームの位置を単純に Embedding 層 によって埋め込みます。そして事前計算された CNN 特徴マップにこれらの位置埋め込みを追加します。

class PositionalEmbedding(layers.Layer):
    def __init__(self, sequence_length, output_dim, **kwargs):
        super().__init__(**kwargs)
        self.position_embeddings = layers.Embedding(
            input_dim=sequence_length, output_dim=output_dim
        )
        self.sequence_length = sequence_length
        self.output_dim = output_dim

    def call(self, inputs):
        # The inputs are of shape: `(batch_size, frames, num_features)`
        length = tf.shape(inputs)[1]
        positions = tf.range(start=0, limit=length, delta=1)
        embedded_positions = self.position_embeddings(positions)
        return inputs + embedded_positions

    def compute_mask(self, inputs, mask=None):
        mask = tf.reduce_any(tf.cast(inputs, "bool"), axis=-1)
        return mask

そして、Transformer のためのサブクラス化された層を作成できます。

class TransformerEncoder(layers.Layer):
    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):
        super().__init__(**kwargs)
        self.embed_dim = embed_dim
        self.dense_dim = dense_dim
        self.num_heads = num_heads
        self.attention = layers.MultiHeadAttention(
            num_heads=num_heads, key_dim=embed_dim, dropout=0.3
        )
        self.dense_proj = keras.Sequential(
            [layers.Dense(dense_dim, activation=tf.nn.gelu), layers.Dense(embed_dim),]
        )
        self.layernorm_1 = layers.LayerNormalization()
        self.layernorm_2 = layers.LayerNormalization()

    def call(self, inputs, mask=None):
        if mask is not None:
            mask = mask[:, tf.newaxis, :]

        attention_output = self.attention(inputs, inputs, attention_mask=mask)
        proj_input = self.layernorm_1(inputs + attention_output)
        proj_output = self.dense_proj(proj_input)
        return self.layernorm_2(proj_input + proj_output)

 

訓練のためのユティリティ関数

def get_compiled_model():
    sequence_length = MAX_SEQ_LENGTH
    embed_dim = NUM_FEATURES
    dense_dim = 4
    num_heads = 1
    classes = len(label_processor.get_vocabulary())

    inputs = keras.Input(shape=(None, None))
    x = PositionalEmbedding(
        sequence_length, embed_dim, name="frame_position_embedding"
    )(inputs)
    x = TransformerEncoder(embed_dim, dense_dim, num_heads, name="transformer_layer")(x)
    x = layers.GlobalMaxPooling1D()(x)
    x = layers.Dropout(0.5)(x)
    outputs = layers.Dense(classes, activation="softmax")(x)
    model = keras.Model(inputs, outputs)

    model.compile(
        optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
    )
    return model


def run_experiment():
    filepath = "/tmp/video_classifier"
    checkpoint = keras.callbacks.ModelCheckpoint(
        filepath, save_weights_only=True, save_best_only=True, verbose=1
    )

    model = get_compiled_model()
    history = model.fit(
        train_data,
        train_labels,
        validation_split=0.15,
        epochs=EPOCHS,
        callbacks=[checkpoint],
    )

    model.load_weights(filepath)
    _, accuracy = model.evaluate(test_data, test_labels)
    print(f"Test accuracy: {round(accuracy * 100, 2)}%")

    return model

 

モデル訓練と推論

trained_model = run_experiment()

2021-09-14 13:27:55.649167: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)

Epoch 1/5
16/16 [==============================] - 2s 69ms/step - loss: 1.7206 - accuracy: 0.6548 - val_loss: 1.6100 - val_accuracy: 0.2889
Epoch 00001: val_loss improved from inf to 1.61001, saving model to /tmp/video_classifier
Epoch 2/5
16/16 [==============================] - 1s 58ms/step - loss: 0.1306 - accuracy: 0.9524 - val_loss: 1.9321 - val_accuracy: 0.4111
Epoch 00002: val_loss did not improve from 1.61001
Epoch 3/5
16/16 [==============================] - 1s 58ms/step - loss: 0.0704 - accuracy: 0.9742 - val_loss: 0.7381 - val_accuracy: 0.7556
Epoch 00003: val_loss improved from 1.61001 to 0.73814, saving model to /tmp/video_classifier
Epoch 4/5
16/16 [==============================] - 1s 56ms/step - loss: 0.0208 - accuracy: 0.9901 - val_loss: 0.8953 - val_accuracy: 0.7778
Epoch 00004: val_loss did not improve from 0.73814
Epoch 5/5
16/16 [==============================] - 1s 56ms/step - loss: 0.0076 - accuracy: 0.9980 - val_loss: 1.5643 - val_accuracy: 0.7111
Epoch 00005: val_loss did not improve from 0.73814
7/7 [==============================] - 0s 20ms/step - loss: 0.5903 - accuracy: 0.8750
Test accuracy: 87.5%

Epoch 1/5
 9/16 [===============>..............] - ETA: 0s - loss: 2.8924 - accuracy: 0.5382  
Epoch 00001: val_loss improved from inf to 0.18087, saving model to /tmp/video_classifier
16/16 [==============================] - 5s 36ms/step - loss: 2.0517 - accuracy: 0.6071 - val_loss: 0.1809 - val_accuracy: 0.9556
Epoch 2/5
 9/16 [===============>..............] - ETA: 0s - loss: 0.3547 - accuracy: 0.8924
Epoch 00002: val_loss did not improve from 0.18087
16/16 [==============================] - 0s 8ms/step - loss: 0.2953 - accuracy: 0.9067 - val_loss: 0.3006 - val_accuracy: 0.9000
Epoch 3/5
 9/16 [===============>..............] - ETA: 0s - loss: 0.1535 - accuracy: 0.9514
Epoch 00003: val_loss improved from 0.18087 to 0.11083, saving model to /tmp/video_classifier
16/16 [==============================] - 0s 18ms/step - loss: 0.1097 - accuracy: 0.9583 - val_loss: 0.1108 - val_accuracy: 0.9667
Epoch 4/5
 9/16 [===============>..............] - ETA: 0s - loss: 0.0107 - accuracy: 1.0000
Epoch 00004: val_loss did not improve from 0.11083
16/16 [==============================] - 0s 9ms/step - loss: 0.0102 - accuracy: 0.9980 - val_loss: 3.7833 - val_accuracy: 0.4333
Epoch 5/5
 9/16 [===============>..............] - ETA: 0s - loss: 0.0411 - accuracy: 0.9896
Epoch 00005: val_loss did not improve from 0.11083
16/16 [==============================] - 0s 8ms/step - loss: 0.0247 - accuracy: 0.9940 - val_loss: 1.2085 - val_accuracy: 0.7000
7/7 [==============================] - 0s 4ms/step - loss: 0.5305 - accuracy: 0.8795
Test accuracy: 87.95%
CPU times: user 4.62 s, sys: 1.05 s, total: 5.67 s
Wall time: 6.49 s

Note : このモデルは ~423 万のパラメータを持ち、これはこのサンプルの前編で使用したシークエンスモデル (99918 パラメータ) よりも遥かに多いです。この種類の Transformer モデルはより大きなデータセットとより長い事前訓練スケジュールにより最善に機能します。

def prepare_single_video(frames):
    frame_features = np.zeros(shape=(1, MAX_SEQ_LENGTH, NUM_FEATURES), dtype="float32")

    # Pad shorter videos.
    if len(frames) < MAX_SEQ_LENGTH:
        diff = MAX_SEQ_LENGTH - len(frames)
        padding = np.zeros((diff, IMG_SIZE, IMG_SIZE, 3))
        frames = np.concatenate(frames, padding)

    frames = frames[None, ...]

    # Extract features from the frames of the current video.
    for i, batch in enumerate(frames):
        video_length = batch.shape[0]
        length = min(MAX_SEQ_LENGTH, video_length)
        for j in range(length):
            if np.mean(batch[j, :]) > 0.0:
                frame_features[i, j, :] = feature_extractor.predict(batch[None, j, :])
            else:
                frame_features[i, j, :] = 0.0

    return frame_features


def predict_action(path):
    class_vocab = label_processor.get_vocabulary()

    frames = load_video(os.path.join("test", path))
    frame_features = prepare_single_video(frames)
    probabilities = trained_model.predict(frame_features)[0]

    for i in np.argsort(probabilities)[::-1]:
        print(f"  {class_vocab[i]}: {probabilities[i] * 100:5.2f}%")
    return frames


# This utility is for visualization.
# Referenced from:
# https://www.tensorflow.org/hub/tutorials/action_recognition_with_tf_hub
def to_gif(images):
    converted_images = images.astype(np.uint8)
    imageio.mimsave("animation.gif", converted_images, fps=10)
    return embed.embed_file("animation.gif")


test_video = np.random.choice(test_df["video_name"].values.tolist())
print(f"Test video path: {test_video}")
test_frames = predict_action(test_video)
to_gif(test_frames[:MAX_SEQ_LENGTH])

Test video path: v_TennisSwing_g05_c06.avi
  TennisSwing: 98.90%
  CricketShot:  1.10%
  Punch:  0.00%
  ShavingBeard:  0.00%
  PlayingCello:  0.00%

Test video path: v_CricketShot_g01_c04.avi
  CricketShot: 100.00%
  TennisSwing:  0.00%
  ShavingBeard:  0.00%
  PlayingCello:  0.00%
  Punch:  0.00%

The performance of our model is far from optimal, because it was trained on a small dataset.

以上