Unveiling the Secret Linearity of Transformers: Further Advance Model Efficiency and Performance
In a new paper Your Transformer is Secretly Linear, a research team uncovers a near-perfect linear relationship in transformations between sequential layers and introduces a novel distillation tech...
Source: syncedreview.com
In a new paper Your Transformer is Secretly Linear, a research team uncovers a near-perfect linear relationship in transformations between sequential layers and introduces a novel distillation technique that approximates certain layers linearly while preserving model performance.