Evaluating the impact of preprocessing on sign language translation

Abstract

Sign Language Translation (SLT) requires learning spatio-temporal visual representations and aligning them with textual outputs under limited data conditions. This dissertation evaluates the impact of preprocessing on SLT performance through a four-phase experimental pipeline: feature extraction, translation-level hyperparameter optimisation, decoder-side pretraining, and cross-dataset transfer. Feature extractors are first systematically evaluated and fine-tuned to identify effective visual representations. Translation performance is then optimised using a fixed transformer model, with visual features held fixed to isolate the effect of translation-level hyperparameters. Decoder-side pretraining is assessed in-domain, followed by cross-dataset adaptation from RWTH-PHOENIXWeather 2014T to South African Sign Language to evaluate transfer under low-resource conditions. Results show that visual representation quality is the primary driver of SLT performance, with consistent differences across extractor architectures. Feature extractor fine-tuning and translation-level optimisation yield complementary improvements. In contrast, decoder pre-training provides no consistent gains when strong representations and optimised configurations are already in place. Cross-dataset transfer improves performance and training stability to a limited extent only, and overall results remain constrained by data scarcity. These findings provide a clearer understanding of the challenges in low-resource SLT, particularly for South African Sign Language (SASL), highlighting the central role of visual representation quality and the extent to which data scarcity constrains achievable performance.