同济大学同济大学 数字经济研究中心网络金融安全(国家级)协同创新中心

Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning

aLLM4TS 模型框架
aLLM4TS 模型框架

内容概括

aLLM4TS 将大语言模型用于时间序列表示学习,把时间序列预测重新表述为自监督的多 patch 预测任务。训练分两个阶段:先在多种时间序列数据集上以预测下一个 patch 为目标进行因果持续预训练,使大语言模型适应时间序列数据;再在目标场景中针对多 patch 预测进行微调。模型采用逐 patch 解码层,直接将各个 patch 映射为时间序列,区别于以往依赖序列级解码的方法。论文在多项下游任务上评估了所学表示的效果与可迁移性。

原文摘要

In this study, we present aLLM4TS, an innovative framework that adapts Large Language Models (LLMs) for time-series representation learning. Central to our approach is that we reconceive time-series forecasting as a self-supervised, multi-patch prediction task, which, compared to traditional mask-and-reconstruction methods, captures temporal dynamics in patch representations more effectively. Our strategy encompasses two-stage training: (i). a causal continual pre-training phase on various time-series datasets, anchored on next patch prediction, effectively syncing LLM capabilities with the intricacies of time-series data; (ii). fine-tuning for multi-patch prediction in the targeted time-series context. A distinctive element of our framework is the patch-wise decoding layer, which departs from previous methods reliant on sequence-level decoding. Such a design directly transposes individual patches into temporal sequences, thereby significantly bolstering the model’s proficiency in mastering temporal patch-based representations. aLLM4TS demonstrates superior performance in several downstream tasks, proving its effectiveness in deriving temporal representations with enhanced transferability and marking a pivotal advancement in the adaptation of LLMs for time-series analysis.

相关资源

来源:ICML 2024 · 原文摘要

返回论文发表