train-tui came out of training AI models on Apple’s MPS and AMD’s ROCm. The problem was that the
training monitors I found were web-based or had lots of dependencies, so I built my own. The main
things I needed to keep in check were VRAM and the loss, and it does exactly what I need. It’s
written in C, so it’s lightweight, and it only reads the log and the GPU stats, so it adds no
complications to the training itself. It only runs on Linux, though, and reads AMD and NVIDIA GPUs.
For quick and easy training runs it was really a blessing.
██ train-tui hat_l_face_x4 pid 3703603 profile basicsr
iter 184300/300000 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 61.4%
epoch 43 lr 1.000e-04 t/it 1.89s (data 0.004s)
eta 2 days, 12:44:33
────────────────────────────────────────────────────────────────────────────────────────────────
● [TRAINING] cpu 64°C nvme 51°C 2 GPUs
● gpu0 RTX 5090 load ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 99% vram ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░ 28.4G/31.8G
core 71°C 548/575 W fan 61% this run
● gpu1 RTX 5090 load ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 100% vram ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░ 28.4G/31.8G
core 79°C 575/575 W fan 78% this run
────────────────────────────────────────────────────────────────────────────────────────────────
metrics l_g_pix 0.005051 l_g_percep 1.4949 l_g_gan 0.055676 l_d_real 0.45404
l_d_fake 0.41163 out_d_r 1.207 out_d_f -1.3434
────────────────────────────────────────────────────────────────────────────────────────────────
validation last @ 180000 psnr 28.6538 (best 28.8585 @ 165000)
ssim 0.8275 (best 0.8325 @ 165000) (every 5000)
checkpoint net_g_180000.pth iter 180000 07:40 (2h 15m ago) every 5000
────────────────────────────────────────────────────────────────────────────────────────────────
proc alive log ok last write 09:56:05 (0s ago)
q quit r refresh (auto 1000ms)