Keras 2 : examples : 単眼深度推定 (翻訳/解説)
翻訳 : (株)クラスキャット セールスインフォメーション
作成日時 : 11/18/2021 (keras 2.7.0)
* 本ページは、Keras の以下のドキュメントを翻訳した上で適宜、補足説明したものです：
- Code examples : Computer Vision : Monocular depth estimation (Author: Victor Basu)
* サンプルコードの動作確認はしておりますが、必要な場合には適宜、追加改変しています。
* ご自由にリンクを張って頂いてかまいませんが、sales-info@classcat.com までご一報いただけると嬉しいです。
クラスキャット 人工知能 研究開発支援サービス ★ 無料 Web セミナー開催中 ★
◆ クラスキャットは人工知能・テレワークに関する各種サービスを提供しております。お気軽にご相談ください :
- 人工知能研究開発支援
- 人工知能研修サービス(経営者層向けオンサイト研修)
- テクニカルコンサルティングサービス
- 実証実験(プロトタイプ構築)
- アプリケーションへの実装
- 人工知能研修サービス
- PoC(概念実証)を失敗させないための支援
- テレワーク & オンライン授業を支援
◆ 人工知能とビジネスをテーマに WEB セミナーを定期的に開催しています。スケジュール。
- お住まいの地域に関係なく Web ブラウザからご参加頂けます。事前登録 が必要ですのでご注意ください。
- ウェビナー運用には弊社製品「ClassCat® Webinar」を利用しています。
◆ お問合せ : 本件に関するお問い合わせ先は下記までお願いいたします。
- 株式会社クラスキャット セールス・マーケティング本部 セールス・インフォメーション
- E-Mail：sales-info@classcat.com ; WebSite: www.classcat.com ; Facebook
Keras 2 : examples : 単眼深度推定
イントロダクション
深度推定は 2D 画像からのシーン・ジオメトリの推論への重要なステップです。単眼 (= monocular) 深度推定の目的は、入力として単一の RGB 画像が与えられたとき、各ピクセルの深度値を予測したり深度情報を推論することです。このサンプルは convnet と単純な損失関数で深度推定モデルを構築するアプローチを示します。
セットアップ
import os
import sys
import tensorflow as tf
from tensorflow.keras import layers
import pandas as pd
import numpy as np
import cv2
import matplotlib.pyplot as plt
tf.random.set_seed(123)
データセットのダウンロード
このチュートリアルのためにはデータセット DIODE : A Dense Indoor and Outdoor Depth Dataset を使用していきます。けれども、私達のモデルのためには訓練と評価サブセットを生成する、検証セットを利用します。元のデータセットの訓練セットではなく検証セットを使用する理由は、訓練セットが 81GB のデータから成るためです、これは 2.6GB に過ぎない検証セットに比べてダウンロードすることは困難です。利用できる別のデータセットは NYU-v2 と KITTI です。
annotation_folder = "/dataset/"
if not os.path.exists(os.path.abspath(".") + annotation_folder):
annotation_zip = tf.keras.utils.get_file(
"val.tar.gz",
cache_subdir=os.path.abspath("."),
origin="http://diode-dataset.s3.amazonaws.com/val.tar.gz",
extract=True,
)
Downloading data from http://diode-dataset.s3.amazonaws.com/val.tar.gz 2774630400/2774625282 [==============================] - 90s 0us/step 2774638592/2774625282 [==============================] - 90s 0us/step
!ls val -lF
!ls val -lF
!ls -lF val/indoors
total 12 drwxrwxr-x 3 1177 2002 4096 Aug 1 2019 scene_00019/ drwxrwxr-x 6 1177 2002 4096 Jul 25 2019 scene_00020/ drwxrwxr-x 7 1177 2002 4096 Jul 25 2019 scene_00021/
!ls -lF val/outdoor
total 12 drwxrwxr-x 7 1177 2002 4096 Jul 25 2019 scene_00022/ drwxrwxr-x 5 1177 2002 4096 Jul 25 2019 scene_00023/ drwxrwxr-x 4 1177 2002 4096 Jul 25 2019 scene_00024/
!ls -lF val/indoors/scene_00019/scan_00183 | head
total 579852 -rwxrwxr-x 1 1177 2002 3145856 Jul 30 2019 00019_00183_indoors_000_010_depth_mask.npy* -rwxrwxr-x 1 1177 2002 3145856 Jul 29 2019 00019_00183_indoors_000_010_depth.npy* -rwxrwxr-x 1 1177 2002 1172425 Jul 29 2019 00019_00183_indoors_000_010.png* -rwxrwxr-x 1 1177 2002 3145856 Jul 30 2019 00019_00183_indoors_000_040_depth_mask.npy* -rwxrwxr-x 1 1177 2002 3145856 Jul 29 2019 00019_00183_indoors_000_040_depth.npy* -rwxrwxr-x 1 1177 2002 1102623 Jul 29 2019 00019_00183_indoors_000_040.png* -rwxrwxr-x 1 1177 2002 3145856 Jul 30 2019 00019_00183_indoors_010_000_depth_mask.npy* -rwxrwxr-x 1 1177 2002 3145856 Jul 29 2019 00019_00183_indoors_010_000_depth.npy* -rwxrwxr-x 1 1177 2002 1215526 Jul 29 2019 00019_00183_indoors_010_000.png*
データセットの準備
深度推定モデルを訓練するために indoor 画像だけを使用します。
path = "val/indoors"
filelist = []
for root, dirs, files in os.walk(path):
for file in files:
filelist.append(os.path.join(root, file))
filelist.sort()
data = {
"image": [x for x in filelist if x.endswith(".png")],
"depth": [x for x in filelist if x.endswith("_depth.npy")],
"mask": [x for x in filelist if x.endswith("_depth_mask.npy")],
}
df = pd.DataFrame(data)
df = df.sample(frac=1, random_state=42)
df.head()
image depth mask 234 val/indoors/scene_00021/scan_00188/00021_00188... val/indoors/scene_00021/scan_00188/00021_00188... val/indoors/scene_00021/scan_00188/00021_00188... 110 val/indoors/scene_00020/scan_00185/00020_00185... val/indoors/scene_00020/scan_00185/00020_00185... val/indoors/scene_00020/scan_00185/00020_00185... 248 val/indoors/scene_00021/scan_00189/00021_00189... val/indoors/scene_00021/scan_00189/00021_00189... val/indoors/scene_00021/scan_00189/00021_00189... 9 val/indoors/scene_00019/scan_00183/00019_00183... val/indoors/scene_00019/scan_00183/00019_00183... val/indoors/scene_00019/scan_00183/00019_00183... 93 val/indoors/scene_00020/scan_00184/00020_00184... val/indoors/scene_00020/scan_00184/00020_00184... val/indoors/scene_00020/scan_00184/00020_00184...
!ls val/indoors/scene_00019/scan_00183 | head
00019_00183_indoors_000_010_depth_mask.npy 00019_00183_indoors_000_010_depth.npy 00019_00183_indoors_000_010.png 00019_00183_indoors_000_040_depth_mask.npy 00019_00183_indoors_000_040_depth.npy 00019_00183_indoors_000_040.png 00019_00183_indoors_010_000_depth_mask.npy 00019_00183_indoors_010_000_depth.npy 00019_00183_indoors_010_000.png 00019_00183_indoors_010_020_depth_mask.npy
ハイパーパラメータの準備
HEIGHT = 256
WIDTH = 256
LR = 0.0002
EPOCHS = 30
BATCH_SIZE = 32
データパイプラインの構築
- パイプラインは RGB 画像、そして深度と深度マスクファイルのパスを含むデータフレームを取ります。
- RGB 画像を読みリサイズします。
- 深度と深度マスクファイルを読み、深度マップ画像を生成するためにそれらを処理してリサイズします。
- バッチのために RGB 画像と深度マップ画像を返します。
class DataGenerator(tf.keras.utils.Sequence):
def __init__(self, data, batch_size=6, dim=(768, 1024), n_channels=3, shuffle=True):
"""
Initialization
"""
self.data = data
self.indices = self.data.index.tolist()
self.dim = dim
self.n_channels = n_channels
self.batch_size = batch_size
self.shuffle = shuffle
self.min_depth = 0.1
self.on_epoch_end()
def __len__(self):
return int(np.ceil(len(self.data) / self.batch_size))
def __getitem__(self, index):
if (index + 1) * self.batch_size > len(self.indices):
self.batch_size = len(self.indices) - index * self.batch_size
# Generate one batch of data
# Generate indices of the batch
index = self.indices[index * self.batch_size : (index + 1) * self.batch_size]
# Find list of IDs
batch = [self.indices[k] for k in index]
x, y = self.data_generation(batch)
return x, y
def on_epoch_end(self):
"""
Updates indexes after each epoch
"""
self.index = np.arange(len(self.indices))
if self.shuffle == True:
np.random.shuffle(self.index)
def load(self, image_path, depth_map, mask):
"""Load input and target image."""
image_ = cv2.imread(image_path)
image_ = cv2.cvtColor(image_, cv2.COLOR_BGR2RGB)
image_ = cv2.resize(image_, self.dim)
image_ = tf.image.convert_image_dtype(image_, tf.float32)
depth_map = np.load(depth_map).squeeze()
mask = np.load(mask)
mask = mask > 0
max_depth = min(300, np.percentile(depth_map, 99))
depth_map = np.clip(depth_map, self.min_depth, max_depth)
depth_map = np.log(depth_map, where=mask)
depth_map = np.ma.masked_where(~mask, depth_map)
depth_map = np.clip(depth_map, 0.1, np.log(max_depth))
depth_map = cv2.resize(depth_map, self.dim)
depth_map = np.expand_dims(depth_map, axis=2)
depth_map = tf.image.convert_image_dtype(depth_map, tf.float32)
return image_, depth_map
def data_generation(self, batch):
x = np.empty((self.batch_size, *self.dim, self.n_channels))
y = np.empty((self.batch_size, *self.dim, 1))
for i, batch_id in enumerate(batch):
x[i,], y[i,] = self.load(
self.data["image"][batch_id],
self.data["depth"][batch_id],
self.data["mask"][batch_id],
)
return x, y
サンプルの可視化
def visualize_depth_map(samples, test=False, model=None):
input, target = samples
cmap = plt.cm.jet
cmap.set_bad(color="black")
if test:
pred = model.predict(input)
fig, ax = plt.subplots(6, 3, figsize=(50, 50))
for i in range(6):
ax[i, 0].imshow((input[i].squeeze()))
ax[i, 1].imshow((target[i].squeeze()), cmap=cmap)
ax[i, 2].imshow((pred[i].squeeze()), cmap=cmap)
else:
fig, ax = plt.subplots(6, 2, figsize=(50, 50))
for i in range(6):
ax[i, 0].imshow((input[i].squeeze()))
ax[i, 1].imshow((target[i].squeeze()), cmap=cmap)
visualize_samples = next(
iter(DataGenerator(data=df, batch_size=6, dim=(HEIGHT, WIDTH)))
)
visualize_depth_map(visualize_samples)
3D ポイントクラウドの可視化
depth_vis = np.flipud(visualize_samples[1][1].squeeze()) # target
img_vis = np.flipud(visualize_samples[0][1].squeeze()) # input
fig = plt.figure(figsize=(15, 10))
ax = plt.axes(projection="3d")
STEP = 3
for x in range(0, img_vis.shape[0], STEP):
for y in range(0, img_vis.shape[1], STEP):
ax.scatter(
[depth_vis[x, y]] * 3,
[y] * 3,
[x] * 3,
c=tuple(img_vis[x, y, :3] / 255),
s=3,
)
ax.view_init(45, 135)
モデルの構築
- 基本モデルは U-Net に由来します。
- 追加のスキップ接続は downscaling ブロックで実装されます。
class DownscaleBlock(layers.Layer):
def __init__(
self, filters, kernel_size=(3, 3), padding="same", strides=1, **kwargs
):
super().__init__(**kwargs)
self.convA = layers.Conv2D(filters, kernel_size, strides, padding)
self.convB = layers.Conv2D(filters, kernel_size, strides, padding)
self.reluA = layers.LeakyReLU(alpha=0.2)
self.reluB = layers.LeakyReLU(alpha=0.2)
self.bn2a = tf.keras.layers.BatchNormalization()
self.bn2b = tf.keras.layers.BatchNormalization()
self.pool = layers.MaxPool2D((2, 2), (2, 2))
def call(self, input_tensor):
d = self.convA(input_tensor)
x = self.bn2a(d)
x = self.reluA(x)
x = self.convB(x)
x = self.bn2b(x)
x = self.reluB(x)
x += d
p = self.pool(x)
return x, p
class UpscaleBlock(layers.Layer):
def __init__(
self, filters, kernel_size=(3, 3), padding="same", strides=1, **kwargs
):
super().__init__(**kwargs)
self.us = layers.UpSampling2D((2, 2))
self.convA = layers.Conv2D(filters, kernel_size, strides, padding)
self.convB = layers.Conv2D(filters, kernel_size, strides, padding)
self.reluA = layers.LeakyReLU(alpha=0.2)
self.reluB = layers.LeakyReLU(alpha=0.2)
self.bn2a = tf.keras.layers.BatchNormalization()
self.bn2b = tf.keras.layers.BatchNormalization()
self.conc = layers.Concatenate()
def call(self, x, skip):
x = self.us(x)
concat = self.conc([x, skip])
x = self.convA(concat)
x = self.bn2a(x)
x = self.reluA(x)
x = self.convB(x)
x = self.bn2b(x)
x = self.reluB(x)
return x
class BottleNeckBlock(layers.Layer):
def __init__(
self, filters, kernel_size=(3, 3), padding="same", strides=1, **kwargs
):
super().__init__(**kwargs)
self.convA = layers.Conv2D(filters, kernel_size, strides, padding)
self.convB = layers.Conv2D(filters, kernel_size, strides, padding)
self.reluA = layers.LeakyReLU(alpha=0.2)
self.reluB = layers.LeakyReLU(alpha=0.2)
def call(self, x):
x = self.convA(x)
x = self.reluA(x)
x = self.convB(x)
x = self.reluB(x)
return x
損失の定義
私達のモデルでは 3 つの損失を最適化します。1. 構造的類似性インデックス (SSIM, Structural similarity index), 2. L1-損失 or 私達のケースでは Point-wise 深度値, 3. 深度 smoothness 損失。
3 つの損失関数から、SSIM がモデル性能の向上に最も寄与します。
class DepthEstimationModel(tf.keras.Model):
def __init__(self):
super().__init__()
self.ssim_loss_weight = 0.85
self.l1_loss_weight = 0.1
self.edge_loss_weight = 0.9
self.loss_metric = tf.keras.metrics.Mean(name="loss")
f = [16, 32, 64, 128, 256]
self.downscale_blocks = [
DownscaleBlock(f[0]),
DownscaleBlock(f[1]),
DownscaleBlock(f[2]),
DownscaleBlock(f[3]),
]
self.bottle_neck_block = BottleNeckBlock(f[4])
self.upscale_blocks = [
UpscaleBlock(f[3]),
UpscaleBlock(f[2]),
UpscaleBlock(f[1]),
UpscaleBlock(f[0]),
]
self.conv_layer = layers.Conv2D(1, (1, 1), padding="same", activation="tanh")
def calculate_loss(self, target, pred):
# Edges
dy_true, dx_true = tf.image.image_gradients(target)
dy_pred, dx_pred = tf.image.image_gradients(pred)
weights_x = tf.exp(tf.reduce_mean(tf.abs(dx_true)))
weights_y = tf.exp(tf.reduce_mean(tf.abs(dy_true)))
# Depth smoothness
smoothness_x = dx_pred * weights_x
smoothness_y = dy_pred * weights_y
depth_smoothness_loss = tf.reduce_mean(abs(smoothness_x)) + tf.reduce_mean(
abs(smoothness_y)
)
# Structural similarity (SSIM) index
ssim_loss = tf.reduce_mean(
1
- tf.image.ssim(
target, pred, max_val=WIDTH, filter_size=7, k1=0.01 ** 2, k2=0.03 ** 2
)
)
# Point-wise depth
l1_loss = tf.reduce_mean(tf.abs(target - pred))
loss = (
(self.ssim_loss_weight * ssim_loss)
+ (self.l1_loss_weight * l1_loss)
+ (self.edge_loss_weight * depth_smoothness_loss)
)
return loss
@property
def metrics(self):
return [self.loss_metric]
def train_step(self, batch_data):
input, target = batch_data
with tf.GradientTape() as tape:
pred = self(input, training=True)
loss = self.calculate_loss(target, pred)
gradients = tape.gradient(loss, self.trainable_variables)
self.optimizer.apply_gradients(zip(gradients, self.trainable_variables))
self.loss_metric.update_state(loss)
return {
"loss": self.loss_metric.result(),
}
def test_step(self, batch_data):
input, target = batch_data
pred = self(input, training=False)
loss = self.calculate_loss(target, pred)
self.loss_metric.update_state(loss)
return {
"loss": self.loss_metric.result(),
}
def call(self, x):
c1, p1 = self.downscale_blocks[0](x)
c2, p2 = self.downscale_blocks[1](p1)
c3, p3 = self.downscale_blocks[2](p2)
c4, p4 = self.downscale_blocks[3](p3)
bn = self.bottle_neck_block(p4)
u1 = self.upscale_blocks[0](bn, c4)
u2 = self.upscale_blocks[1](u1, c3)
u3 = self.upscale_blocks[2](u2, c2)
u4 = self.upscale_blocks[3](u3, c1)
return self.conv_layer(u4)
モデル訓練
optimizer = tf.keras.optimizers.Adam(
learning_rate=LR,
amsgrad=False,
)
model = DepthEstimationModel()
# Define the loss function
cross_entropy = tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True, reduction="none"
)
# Compile the model
model.compile(optimizer, loss=cross_entropy)
train_loader = DataGenerator(
data=df[:260].reset_index(drop="true"), batch_size=BATCH_SIZE, dim=(HEIGHT, WIDTH)
)
validation_loader = DataGenerator(
data=df[260:].reset_index(drop="true"), batch_size=BATCH_SIZE, dim=(HEIGHT, WIDTH)
)
model.fit(
train_loader,
epochs=EPOCHS,
validation_data=validation_loader,
)
Epoch 1/30 9/9 [==============================] - 18s 1s/step - loss: 1.1543 - val_loss: 1.4281 Epoch 2/30 9/9 [==============================] - 3s 390ms/step - loss: 0.8727 - val_loss: 1.0686 Epoch 3/30 9/9 [==============================] - 4s 428ms/step - loss: 0.6659 - val_loss: 0.7884 Epoch 4/30 9/9 [==============================] - 3s 334ms/step - loss: 0.6462 - val_loss: 0.6198 Epoch 5/30 9/9 [==============================] - 3s 355ms/step - loss: 0.5689 - val_loss: 0.6207 Epoch 6/30 9/9 [==============================] - 3s 361ms/step - loss: 0.5067 - val_loss: 0.4876 Epoch 7/30 9/9 [==============================] - 3s 357ms/step - loss: 0.4680 - val_loss: 0.4698 Epoch 8/30 9/9 [==============================] - 3s 325ms/step - loss: 0.4622 - val_loss: 0.7249 Epoch 9/30 9/9 [==============================] - 3s 393ms/step - loss: 0.4215 - val_loss: 0.3826 Epoch 10/30 9/9 [==============================] - 3s 337ms/step - loss: 0.3788 - val_loss: 0.3289 Epoch 11/30 9/9 [==============================] - 3s 345ms/step - loss: 0.3347 - val_loss: 0.3032 Epoch 12/30 9/9 [==============================] - 3s 327ms/step - loss: 0.3488 - val_loss: 0.2631 Epoch 13/30 9/9 [==============================] - 3s 326ms/step - loss: 0.3315 - val_loss: 0.2383 Epoch 14/30 9/9 [==============================] - 3s 331ms/step - loss: 0.3349 - val_loss: 0.2379 Epoch 15/30 9/9 [==============================] - 3s 333ms/step - loss: 0.3394 - val_loss: 0.2151 Epoch 16/30 9/9 [==============================] - 3s 337ms/step - loss: 0.3073 - val_loss: 0.2243 Epoch 17/30 9/9 [==============================] - 3s 355ms/step - loss: 0.3951 - val_loss: 0.2627 Epoch 18/30 9/9 [==============================] - 3s 335ms/step - loss: 0.3657 - val_loss: 0.2175 Epoch 19/30 9/9 [==============================] - 3s 321ms/step - loss: 0.3404 - val_loss: 0.2073 Epoch 20/30 9/9 [==============================] - 3s 320ms/step - loss: 0.3549 - val_loss: 0.1972 Epoch 21/30 9/9 [==============================] - 3s 317ms/step - loss: 0.2802 - val_loss: 0.1936 Epoch 22/30 9/9 [==============================] - 3s 316ms/step - loss: 0.2632 - val_loss: 0.1893 Epoch 23/30 9/9 [==============================] - 3s 318ms/step - loss: 0.2862 - val_loss: 0.1807 Epoch 24/30 9/9 [==============================] - 3s 328ms/step - loss: 0.3083 - val_loss: 0.1923 Epoch 25/30 9/9 [==============================] - 3s 312ms/step - loss: 0.3666 - val_loss: 0.1795 Epoch 26/30 9/9 [==============================] - 3s 316ms/step - loss: 0.2928 - val_loss: 0.1753 Epoch 27/30 9/9 [==============================] - 3s 325ms/step - loss: 0.2945 - val_loss: 0.1790 Epoch 28/30 9/9 [==============================] - 3s 325ms/step - loss: 0.2642 - val_loss: 0.1775 Epoch 29/30 9/9 [==============================] - 3s 333ms/step - loss: 0.2546 - val_loss: 0.1810 Epoch 30/30 9/9 [==============================] - 3s 315ms/step - loss: 0.2650 - val_loss: 0.1795 <keras.callbacks.History at 0x7f5151799fd0>
(訳注: 実験結果)
Epoch 1/30 9/9 [==============================] - 18s 1s/step - loss: 1.1154 - val_loss: 1.4483 Epoch 2/30 9/9 [==============================] - 2s 260ms/step - loss: 0.8266 - val_loss: 1.0560 Epoch 3/30 9/9 [==============================] - 2s 258ms/step - loss: 0.6899 - val_loss: 0.9659 Epoch 4/30 9/9 [==============================] - 2s 262ms/step - loss: 0.5891 - val_loss: 1.1010 Epoch 5/30 9/9 [==============================] - 2s 252ms/step - loss: 0.5427 - val_loss: 0.7874 Epoch 6/30 9/9 [==============================] - 2s 268ms/step - loss: 0.5103 - val_loss: 0.9982 Epoch 7/30 9/9 [==============================] - 2s 246ms/step - loss: 0.3996 - val_loss: 0.4821 Epoch 8/30 9/9 [==============================] - 2s 245ms/step - loss: 0.4218 - val_loss: 0.3737 Epoch 9/30 9/9 [==============================] - 2s 247ms/step - loss: 0.4108 - val_loss: 0.3250 Epoch 10/30 9/9 [==============================] - 2s 249ms/step - loss: 0.3721 - val_loss: 0.2965 Epoch 11/30 9/9 [==============================] - 2s 255ms/step - loss: 0.4406 - val_loss: 0.2646 Epoch 12/30 9/9 [==============================] - 2s 255ms/step - loss: 0.3440 - val_loss: 0.2697 Epoch 13/30 9/9 [==============================] - 2s 244ms/step - loss: 0.3888 - val_loss: 0.3238 Epoch 14/30 9/9 [==============================] - 2s 250ms/step - loss: 0.3977 - val_loss: 0.2332 Epoch 15/30 9/9 [==============================] - 2s 257ms/step - loss: 0.3888 - val_loss: 0.2240 Epoch 16/30 9/9 [==============================] - 2s 257ms/step - loss: 0.3830 - val_loss: 0.2145 Epoch 17/30 9/9 [==============================] - 2s 246ms/step - loss: 0.4022 - val_loss: 0.2082 Epoch 18/30 9/9 [==============================] - 2s 254ms/step - loss: 0.3392 - val_loss: 0.2097 Epoch 19/30 9/9 [==============================] - 2s 248ms/step - loss: 0.3324 - val_loss: 0.2043 Epoch 20/30 9/9 [==============================] - 2s 257ms/step - loss: 0.3304 - val_loss: 0.2075 Epoch 21/30 9/9 [==============================] - 2s 249ms/step - loss: 0.3221 - val_loss: 0.2130 Epoch 22/30 9/9 [==============================] - 2s 247ms/step - loss: 0.3610 - val_loss: 0.2223 Epoch 23/30 9/9 [==============================] - 2s 248ms/step - loss: 0.3028 - val_loss: 0.2148 Epoch 24/30 9/9 [==============================] - 2s 249ms/step - loss: 0.3205 - val_loss: 0.2067 Epoch 25/30 9/9 [==============================] - 2s 244ms/step - loss: 0.2951 - val_loss: 0.1932 Epoch 26/30 9/9 [==============================] - 2s 250ms/step - loss: 0.2801 - val_loss: 0.1929 Epoch 27/30 9/9 [==============================] - 2s 245ms/step - loss: 0.3169 - val_loss: 0.1945 Epoch 28/30 9/9 [==============================] - 2s 250ms/step - loss: 0.2757 - val_loss: 0.1858 Epoch 29/30 9/9 [==============================] - 2s 245ms/step - loss: 0.3135 - val_loss: 0.1831 Epoch 30/30 9/9 [==============================] - 2s 248ms/step - loss: 0.2638 - val_loss: 0.1819 CPU times: user 1min 36s, sys: 7.92 s, total: 1min 44s Wall time: 1min 47s
モデル出力の可視化
検証セットに渡るモデル出力を可視化します。最初の画像は RGB 画像で、2 番目の画像は正解深度値マップ画像で、そして 3 番目のものは予測された深度値マップ画像です。
test_loader = next(
iter(
DataGenerator(
data=df[265:].reset_index(drop="true"), batch_size=6, dim=(HEIGHT, WIDTH)
)
)
)
visualize_depth_map(test_loader, test=True, model=model)
test_loader = next(
iter(
DataGenerator(
data=df[300:].reset_index(drop="true"), batch_size=6, dim=(HEIGHT, WIDTH)
)
)
)
visualize_depth_map(test_loader, test=True, model=model)
可能な改良
このモデルを、U-Net のエンコーディング部を事前訓練済み DenseNet や ResNet で置き換えることにより改良できます。損失関数はこの問題を解くのに重要な役割を果たします。この損失関数の調整は大幅な改良を生成する可能性があります。
リファレンス
以下の論文は深度値推定のための可能なアプローチを掘り下げます。
- Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos
- Digging Into Self-Supervised Monocular Depth Estimation
- Deeper Depth Prediction with Fully Convolutional Residual Networks
You can also find helpful implementations in the papers with code depth estimation task.
以上