HAT Face Finetune

The story

I trained a face-specialised 4x super-resolution checkpoint as an optional companion model for LocalSR, starting from the official ImageNet-pretrained HAT-L weights (XPixel Group, 20.8M parameters). Stock HAT-L handles general photographs well, but old, compressed, or soft portraits leave room for improvement when the degradation looks nothing like a clean bicubic downscale.

The first attempt was a warning. An earlier HAT-S run trained from iteration zero with GAN and VGG perceptual losses at learning rate 1e-4 and no clean-image replay. It invented plausibility instead of structure: clean-face PSNR dropped by around 5 dB (36.25 dB down to 31.1 dB), and edge correlation fell from 0.868 to 0.729.

For the HAT-L runs I threw out the GAN and perceptual terms completely and trained strictly with L1 loss, gradient clipping (1.0), EMA (0.999), AMP, and gradient checkpointing so everything would fit on a single 16 GiB ROCm GPU (an RX 9060 XT). Learning rates dropped by an order of magnitude (1e-5, followed by 2.5e-6 in the final pass), and every batch mixed 15% undamaged replay images alongside high-order degradations (Poisson noise and double JPEG compression) drawn from a research mix of 60% FFHQ, 20% identity- and clip-balanced EFHQ, and 20% general high-resolution imagery. Three unattended runs stretched across roughly three weeks (250k iterations at 192 px batch 1, 250k at 256 px batch 4, and 299k low-LR refinement).

Evaluating 14 candidate checkpoints, soups, and weight blends against 512-image clean, mild, and hard recovery buckets showed the same underlying tension in gentler form. The fully fine-tuned weights gained around +1.0 dB on mildly degraded portraits, yet still cost 1.9 dB on undamaged pictures. Instead of retraining from scratch, I interpolated weights back toward the stock HAT-L base and picked alpha = 0.25 (75% stock HAT-L, 25% face-tuned):

  • Clean (undamaged): 32.78 dB to 32.58 dB (-0.19 dB unrounded)
  • Mild degradation: 28.69 dB to 29.13 dB (+0.44 dB)
  • Hard degradation: 25.72 dB to 25.97 dB (+0.25 dB)

A lighter blend (alpha = 0.1, -0.03 dB clean / +0.19 dB mild) sat close behind; anything at 0.5 or above gave away too much clean-image fidelity to be safe as a default companion.

Inside LocalSR, hat_l_x4_face_task4.pth only runs as an opt-in companion when stock HAT-L is chosen with face-aware detection enabled. Neither the training repository nor the weights are distributed publicly: because the training mix includes FFHQ and EFHQ (research-only dataset terms with unresolved commercial redistribution rights), LocalSR refuses to auto-download the file and verifies any user-supplied import strictly by SHA-256.