VideoLLM: Modeling Video Sequence with Large

VideoLLM:Modeling Video Sequence with Large Language Models Guo Chen1,2, Yin-Dong Zheng1, Jiahao Wang1, Jilan Xu2,3, Yifei Huang2, Junting Pan4 Yi Wang2, Yali Wang2, Yu Qiao2, Tong Lu1, Limin Wang1,2 1 Nanjing University, 2 OpenGVLab, Shanghai AI Laboratory, 3 Fudan University 4 The Chinese University of Hong Kong https://github.com/cg1177/VideoLLM Abstract With the exponential growth of video data, there is an urgent need for automated technology to analyze and comprehend video content. However, existing video understanding models are often task-specific and lack a comprehensive capability of handling diverse tasks. The success of large language models (LLMs) like GPT has demonstrated their impressive abilities in sequence causal reasoning. Building upon this insight, we propose a novel framework calledVideoLLMthat leverages the sequence reasoning capabilities of pre-trained LLMs from natural language processing (NLP) for video sequence understanding. VideoLLM incorporates a carefully designed Modality Encoder and Semantic Translator, which convert inputs from various modalities into a unified token sequence. This token sequence is then fed into a decoder-only LLM. Subsequently, with the aid of a simple task head, our VideoLLM yields an effective unified framework for different kinds of video understanding tasks. To evaluate the efficacy of VideoLLM, we conduct extensive experiments using multiple LLMs and fine-tuning methods. We evaluate our VideoLLMon eight tasks sourced from four different datasets. The experimental results demonstrate that the understanding and reasoning capabilities of LLMs can be effectively transferred to video understanding tasks. 1 Introduction The advent of phenomenon-level language applications, such as ChatGPT [59], has showcased LLMs’ [61; 62; 7; 58; 64; 101; 75; 83] remarkable zero-shot capability in effectively addressing multiple NLP or vision-centric tasks. The remarkable sequence modeling and reasoning capabilities that these large language models exhibited can be traced back to their acquisition through rigorous pre-training with substantial parameters on large-scale corpora. Despite the amazing achievements in processing language sequences, understanding video sequences that record the real world’s objective laws and can be regarded as long image sequences is far from the level of present LLM. Video sequence understanding involves various real-world applications, such as surveillance systems [37], autonomous vehicles [70], robotics [66], and wearable devices [71]. Simply put, it involves AI systems in the real-time processing of visual information streams, reasoning them in the context of long-term time series, and then providing responses. The vanilla paradigm for video sequence understanding tasks relies on task-specific designs [93; 103; 11; 98; 51; 100; 39] to encode or decode video sequences, thereby achieving a promising performance but brings additional tailored cost. Compared with natural language, there is no scalable video sequence model that can be seamlessly adapted to different video sequence tasks. This is primarily attributed to the challenges associated with large-scale video self-supervision, which arise from the expensive nature of temporal-intensive visual annotation, as well as the time-consuming process of acquiring and processing extensive video arXiv:2305.13292v2 [cs.CV] 23 May 2023

Figure 1: Overview of our motivation and method. (a) LLM taking words as input is pretrained on large-scale nature language composed of word sequences. (b) VideoLLMencodes video stream to token sequences and applies large-scale pre-trained LLMs to video sequence reasoning tasks. data. As a result, there is a pressing demand for an efficient method that can offer fundamental modeling capabilities for tasks involving video sequence understanding. In this work, we present a novel paradigm called VideoLLM, as shown in Figure 1, which aligns video and language sequences and harnesses LLMs’ reasoning and understanding capabilities. This paradigm enables videos to engage in reasoning about real-world events through the medium of language. Specifically, it is composed of three core components: (1) a temporal-wise unitization method to encode unit-wise data stream, (2) an appended semantic translator to transfer visual semantics to language semantics, and (3) a decoder-only LLM as a generalist video sequence reasoner for various video sequence understanding tasks. The design allows sequence tasks with different modalities (e.g. visual and text) to be seamlessly integrated, as we verified in the experiments visualonly tasks such as temporal action detection and action anticipation, etc., and visual-language tasks such as temporal grounding and highlight detection, etc. The unit-wise encoding and decoder-only reasoning enable the system to run with minimal delay, greatly meeting real-time or interactive systems’ experience requirements. In contrast to the long-term temporal post-fusion approach proposed in [3], our method emphasizes learning short-term visual token representations for effectively integrating frozen LLMs. This adaptation is conducted within a well-pretrained LLM with robust sequence processing and causal reasoning abilities. Consequently, long-term video modeling can be disregarded, effectively simplifying the complexity of the system design. Compared to recent API-based or ensemble-based visual understanding applications [12; 97; 68; 54; 45], we offer an end-to-end system-level approach for video understanding by bridging visual models and LLMs, enhancing the overall efficiency of the long-term video sequence understanding pipeline. Moreover, our method achieves maximal decoupling between short-term and long-term visual modeling, enabling the flexible adoption of heterogeneous short-term visual encoding techniques while rapidly incorporating state-of-the-art LLMs. Our contributions can be succinctly summarized as follows: (1) We present VideoLLM, a novel framework that harnesses the sequence reasoning capabilities of pre-trained LLMs to tackle video sequence understanding tasks through the medium of language. By aligning videos with language, VideoLLMenables simultaneous reasoning about language logic and the evolution of real-world states through unified modeling. (2) We reexamine the characteristics and challenges associated with various video sequence understanding tasks and develop a novel, plug-and-play adaptation scheme to adapt off-the-shelf visual encoders and advanced LLMs effectively. This scheme is built upon a unified adaptation principle, eliminating the need for task-specific customization. (3) We conduct extensive experiments across four datasets, encompassing eight video sequence understanding tasks. These tasks encompass diverse settings, including data accessibility (causal or non-causal), perceptual objectives (memory or anticipation), prediction granularity (segment-level or frame-level), and modalities (vision-only or vision-language). The experiments employ a range of LLMs, such as GPT-2, T5, and OPT. Comparative analyses against task-specific tailored models demonstrate that our VideoLLMachieves state-of-the-art or comparable performance on these tasks, employing comparable or fewer trainable parameters. These results effectively establish LLM as an 2

effective video reasoner, while validating the efficacy of our proposed VideoLLMframework for multiple video sequence understanding tasks. 2 Related Work 2.1 Video Sequence Understanding Video Sequence Understanding tasks can be categorized into two types based on the granularity of predictions: timestamp-level tasks and segment-level tasks. Timestamp-level tasks aim to predict closed-set properties at each time step or filter suitable time steps based on textual conditions. For example, [25; 87; 93; 21; 98] implement online action detection or action segmentation tasks to predict the category of each time step in a video stream. Similarly, [103; 26; 24; 67] implement action anticipation tasks to predict the action category that occurs after a certain time gap. Additionally, methods such as [39; 52] achieve text-based highlight detection. Segment-level tasks involve predicting segment boundaries in a video sequence based on closed-set categories or open text. Related tasks include moment query [50; 94; 102; 96; 99] and natural language query [100; 65; 92]. The model proposed in this paper is tested on multiple video sequence understanding tasks to verify the language models’ capability to reason about videos from different perspectives. 2.2 Vision Models Vision Models, including image and video models, have recently been developed rapidly, mainly focusing on representing short-term vision information. Vision models are divided into convolution, transformer, and hybrid networks. Convolution models learn spatial [32; 28; 90; 56; 95; 84] or spacetime [82; 9; 23; 77; 76; 81; 57] visual representations by aggregating neighborhood information using 2D or 3D convolution operators. With the great success of the transformer [78] in the NLP field, the visual transformer has also been continuously developed. The visual transformer models space [18; 55; 86; 74; 5; 20] or space-time [19; 22; 6; 3; 73; 80] through an attention mechanism. Due to the data-hungry problem caused by the lack of inductive bias in the transformer network, a hybrid network [85; 46; 2; 47; 88] combining attention mechanism and convolution operator is proposed to improve performance. 2.3 Large Language Models Large Language Models have emerged in recent years in natural language processing. These models usually contain billions to hundreds of billions of parameters and are trained on large text corpora [61; 62; 89; 64; 30; 13; 75]. The core architecture of the model is based on the Transformer [78] while the objective functions range from masked language modeling [17; 53; 35], generative language modeling [61; 62; 7] and permuted language modeling [14]. Among these works, the generativebased language models showed promising results [62; 7] on a wide range of natural language understanding benchmarks. Beginning with the representative work GPT-3 [7], a series of works [69; 63; 30; 101; 13; 75] scaled up the model and pre-training data and demonstrated strong few-shot and zero-shot performance. Despite the promising results on natural language tasks, the capability of the models are still less explored in multimodal domain. In this paper, we attempt to discover the long-range modeling capacity of LLMs in improving video understanding. 2.4 Multimodal Models Multimodal Models aim to learn joint vision and language representation for multimodal downstream tasks. The dominant works are VLP models trained end-to-end on large-scale image/video-text pairs [60; 34; 44; 40; 79; 4; 49]. To relieve the high computation resources, modulated visionlanguage models adopted frozen unimodal or multimodal pre-trained encoders with learnable modules [43; 42; 1]. These models leveraged strong representation ability of large language models for alignment or generation tasks. BLIP-2 [42] trained a lightweight Transformer to compress the visual tokens and built a bridge between vision output and language input. Flamingo [1] injected visual features into LLM by adding intermediate cross-attention Transformer layers. 3

3 Preliminary 3.1 Large Language Model Model #Param #Tokens GPT-2 [62] 1.5B 10B GPT-3 [7] 175B 499B T5 [64] 11B 156B OPT [101] 175B 180B PaLM [13] 540B 780B LaMDA [72] 137B 1.56T LLaMA [75] 65B 1.4T Table 1: Parameter and training scale of LLMs. The current Language Model can be mainly sorted into encoder-decoder and decoder-only structures. The encoderdecoder uses bidirectional Masked Language Modeling to restore corrupted tokens in a document for textual representation learning, such as BERT [17] and T5 [64]. Alternatively, the decoder-only (GPT family [61], OPT [101]) uses unidirectional Language Modeling to directly maximize the likelihood of the sequence under the forward autoregressive factorization. These two training mechanisms grant the language model powerful language sequence modeling and reasoning capabilities. Model parameters and data size of Language models are continuous growth. Table 1 lists the model parameter amount and pre-training token size. These models usually adopt different network structures, training strategies, and corpora. We will explore various LLMs’ performance, advantages, and drawbacks as video sequence reasoners. 3.2 Tasks VideoLLMis verified on 8 video understanding tasks across 4 datasets in Table 2. Online Action Detection, Action Segmentation, andTemporal Action Detectionfocus on detecting and recognizing actions and their temporal boundaries. Online Captioninggenerates textual descriptions of video content, while Highlight Detectionidentifies exciting parts and generates summaries. Action AnticipationandLong-term Anticipationpredict future actions and content in advance, respectively. Moment Queryquickly retrieves specific segments or events in a video. Nature Language Query localize a temporal segment through a textual question. Task Datasets Metric Online Action Detection EK100 [15] Recall Top-5 Action Segmentation Breakfast [38] F1; Edit distance Online Captioning Ego4D-Narration [27] METEOR; ROUGE-L Action Anticipation EK100 [15] Recall Top-5 Long-term Anticipation Ego4D-LTA [27] Edit distance Moment Query Ego4D-MQ [27] mAP@IoU Nature Language Query Ego4D-NLQ [27] Rank@1, Rank@5 Highlight Detection QVHighlights [39] mAP Table 2: Statistics of datasets in our experiments. 4 VideoLLM VideoLLMis a novel online video reasoning system that aims to apply large-scale pre-trained Large Language Models to video sequence understanding tasks through parameter-efficient transfer learning. It directly borrows the sequence modeling ability of LLM to video sequence reasoning, allowing vision to flow in a natural time sequence in the form of language. This section will overview the VideoLLM architecture, as shown in Figure 2. Specifically, VideoLLMcomprises several components: Modality Encoder, Semantic Translator, decoder-only Reasoner, and simple task heads. In this framework, each short video clip is tokenized using corresponding audio and video encoders and then sequentially processed by the LLM. It is important to note that our unified LLM naturally integrates textual conditions into the framework. Furthermore, our framework allows for the easy integration of various human prompts, commands, human-computer interaction techniques, and parameter-efficient fine-tuning techniques to improve model performance and efficiency. 4.1 Modality Encoder We adopt a temporal-wise unitization method to process unit-wise visual (or audio and other modality) information for utilizing LLMs to understand video streams comprehensively. We naturally consider 4

Figure 2: VideoLLMleverages LLM as its core to handle video and text sequences seamlessly. In detail, all input video frames are converted into a visual encoding sequence using a short-term visual encoder. On the other hand, the text condition is transformed into a textual sequence using a text encoder or a text tokenizer. Subsequently, the semantic translator aligns the visual and text encoding, thus feeding the two sequences to LLM for seamless sequence reasoning. Finally, the output generated by LLM can be applied to various video understanding tasks. integrating natural language modeling with LLMs for unified processing to achieve multimodal understanding. Vision. To encode a video sequence of F frames x ∈ RF×H×W×C where H, W, and C are the height, width, and the number of channels of each frame, we use a short-term visual encoder fv, which can be a well-established image encoder or a short-term video encoder. Given Fs presenting the number of frames in a short-term clip, all frames are divided into Nv = F Fs space-time visual unit, and each unit is encoded by fv independently. Hence, fv outputs a sequence of space-time visual units xv =fv(x) ∈R Nv× F s s t × H s h× W sw×d ={x1 v, x 2 v, ..., x Nv v }, where d is the representation dimension andst, sh andsw are the strides of space-time dimensions within fv. Text. We support two encoding approaches when presented with a textual input y containing narration or a question. The first approach involves tokenizing y into yt ∈ RNt×d, where d represents the output dimension of the tokenizer. The other is to process further yt using language encoders ft, such as BERT [17], T5 [64], or CLIP [60], to extract textual features denoted as ye. Subsequently, either yt or ye can be employed as input for the video sequence reasoner to implement the control based on text condition. 4.2 Semantic Translator The language model is essentially a blind who can receive language input and learn various knowledge, but it has no vision and cannot directly perceive the visual world. Therefore, we need to translate the visual semantics into language representations that the language model can interpret. Similar to Perceiver [33], Flamingo [1], and BLIP-2 [42], we adopt an appended sub-network to transfer the semantic space. In this paper, for efficiency, we adopt a simpler design that freezes the visual encoder and transfers the final visual feature into the language world. In detail, given xv ∈ R Nv× F s s t × H s h× W sw×dv, we first pool each visual unit of xv to the temporal token. Hence, we obtain a video sequence representation xt ∈ RNv×dv. We use one linear projector ϕ to learn translation from the visual to language semantics to attain translated semantics sv =ϕ(xt) ∈RNv×d, where dis the hidden dimension of the used LLM. 4.3 Decoder-only Reasoner As detailed in Table 2, our objective is to enable our VideoLLMto accommodate a broad range of video sequence understanding tasks. However, the disparate constraints inherent to these tasks, 5

Figure 3: Schematic diagram of LLMs adapting to 4 types of tasks. (a) “Seen Tokens” denote the data units accepted by the AI system and encoded by the encoder. Predicting the attributes of the latest seen token or near-term unseen token can be seen as an online reasoning task. (b) “Unssen tokens” are data units that have not yet arrived, and predicting their attributes or when they appear in the future usually belongs to future prediction tasks. (c) Given a text condition or a closed category set, retrieving “moments” from a past sequence of seen tokens, also known as memory, is a memory retrieval task. (d) A similar task for memory called dense prediction predicts attributes of each seen token or highlights tokens that match the condition. including their respective inputs and outputs, are a potential obstacle to achieving this goal. To better understand the multifaceted nature of these tasks, we have classified them into four categories, which may exhibit some overlap, as illustrated in Figure 3. This section will discuss efficiently adapting LLMs to address different video understanding tasks. We employ LLM with a decoder-only structure, denoted as M, as the key component of our video sequence reasoner, informed by three critical considerations. First, compelling evidence indicates that decoder-only LLMs are particularly adept at handling causal reasoning tasks for language sequences. Second, the most advanced and high-performing large language models in the current landscape are predominantly decoder-only and are subject to continuous optimization by the research community. Third, a real-world video processor should ideally be designed around a unidirectional visual data flow to maximize performance. This design philosophy aligns seamlessly with the underlying structure of decoder-only language models. Subsequently, we provide a succinct overview of our adaptation method. Online Reasoning. Online Reasoning primarily focuses on real-time prediction of the category or caption for the most recently attended data unit, which in this paper refers to a new short-term video clip. Given a playing video stream and working memorym={s−t+1 v , s−t+2 v , ..., s i v, ..., s 0 v}, where t is the number of seen tokens in memory and s0 v is the latest translated token. In the training phase, mwill be fed into Mto construct a causal sequence c ={c−t+1, c−t+2, ..., ci, ..., c0} for parallel training. We use two linear layers to predict the category of each token si v and its next token si+1 v . Thanks to the causal structure of decoder-only LLM, we do not need to calculate the context of the entire sequence when accepting a novel token in the inference phase, compared with a bidirectional encoder. We only make s0 v cross-attend to the historical context to calculate new states c0. Additionally, we use each ci as the hidden states for online captioning and input into an extra generative language model Mg (e.g., GPT-2 [62]) for autoregressive text generation. Future Prediction. Given a sequence of seen tokens m = {s−t+1 v , s−t+2 v , ..., s i v, ..., s 0 v} as the working memory, model need predict the next Nf tokens or events. In this case, we still utilize the causal structure, supervising each seen token to learn future representations. For predicting different Nf future states, we use Nf normalization layers to separate Nf anticipation presentations a ={a1, a2, ..., ai, ..., aNf}. Memory Retrieval. Memory Retrieval often is an offline task to detect event segments in a closed category set or by a text condition. In our online system, however, the task can evaluate the model’s understanding of segment-level transitions and evolutions in the video. Given a sequence of seen tokens m= {s−t+1 v , s−t+2 v , ..., s i v, ..., s 0 v} as the working memory, to get the context of the whole video, we use the last token s0 v to predict segments in the memory. Another alternative is to concatenate a learnable token sq v or <EOT> at the end of the mto learn the memory summary. To predict at most Nmpossible segments with category-closed in memory, similar to future prediction, we use Nmnormalization layers to separate Nmsegment-level memory presentations ms = {m1 s, m2 s, ..., mi s, ..., mNm s }. Then we adopt two linear layers to predict the category and boundary of each segment. The segments are matched with ground truth through Hungarian matching algorithm [8] for supervision. For memory retrieval based on text condition, we concatenate text presentation yt or ye at the end of mand feed them into Mtogether. Hence, Mcan generate the causal sequence conditioned on text for retrieving matched moments. 6

Dense Prediction. Dense Prediction can be likened to an offline reasoning task where the goal is to predict the category of each token or identify highlight tokens based on textual conditions. In this work, we treat dense prediction as an online task, which serves as a simplified implementation of online action segmentation or highlight detection. Our system uses decoder-only LLM as the default video reasoner and handles online prediction and text conditions like the aforementioned tasks. However, it is worth exploring whether a bidirectional reasoner can provide performance improvements for memory-related tasks. Therefore, we also consider a bidirectional encoder as a potential candidate for our video reasoner, which we evaluate in subsequent experiments. In summary, our experimental objective is to assess the intrinsic capability of Min understanding video sequences. To accomplish this, we propose three fundamental adaptation principles, which have been adhered to by the aforementioned methods. Firstly, we exclusively supervise tasks by relying on the final output of M, instead of employing multi-stage supervision as demonstrated in the works of [51] and [98]. Secondly, we refrain from incorporating prior operators, such as convolution layers, intoM. Lastly, we employ linear layers for each task to transform the hidden states generated byMinto task results, thereby eschewing the utilization of intricate task-specific heads. 4.4 Model Training The training process of VideoLLMinvolves three fine-tuning methods for training the model. Basic Tuning. When working with a frozen language model, the optimization of VideoLLM primarily focuses on fine-tuning the semantic translation and output layers. In this scenario, the model’s performance completely relies on the capabilities of the LLM after semantic translation. Partial Tuning. The partial tuning method involves optimizing specific parts of the LLM in addition to the basic tuning. We adopt three settings for partial tuning: optimizing all bias parameters, optimizing the first block, and optimizing the last block. PEFT Tuning. The widely popular and effective parameter-efficient fine-tuning (PEFT) techniques in NLP, such as LoRA [31], Prompt Tuning [41], and Prefix Tuning [48], have also been applied to optimize VideoLLM. 5 Experiments 5.1 Experimental Setup Dataset and Tasks. In order to thoroughly assess the capabilities of LLMs in video understanding, we performed experiments on four datasets, covering a total of eight tasks. The details of these tasks and datasets are presented in Table 2. The tasks were categorized into four types, as illustrated in Figure 3: Online Reasoning, Future Prediction, Memory Retrieval, and Dense Prediction. This diverse set of tasks allows for comprehensive evaluations from various perspectives, including data accessibility (causal or non-causal), perceptual objectives (memory or anticipation), and prediction granularity (segment-level or frame-level), modalities (vision-only or vision-language). Evaluation and Metrics. Our model evaluation is conducted in accordance with previous studies [25; 103; 21; 29; 27; 39; 10]. Specifically, we measure the accuracy of online action detection and action anticipation tasks using class-mean recall@5(%) following the established standard protocol [15]. To assess the performance of our model in the action segmentation task, we report the framewise accuracy (Acc), segmental edit distance (ED), and the segmental F1 score at overlapping thresholds of 25% denoted as F1@25. For the Long-term anticipation task, we submit our results to the EvalAI platform to evaluate the test set. Consistent with the approach employed in [27], we evaluate the mean Average Precision (mAP) under multiple temporal Intersection over Union (tIoU) thresholds, specifically {0.1; 0.2; 0.3; 0.4; 0.5}, for the Moment Query task. In addition, we report the recall@k, where k = 1, and the IoU=m metric, where m ={0.3, 0.5}, for the Nature Language Query task. Implementation Details. To ensure fairness and facilitate meaningful comparisons within the research community, we employ various visual encoders [9; 60; 23; 91; 73; 104; 88] that have been pretrained on different datasets [16; 36; 60; 27; 15] to extract visual features. This approach helps establish alignment with existing community settings and ensures equitable evaluations. Note that, the same modality encoder could share semantic translator. In this work, using different encoders and semantic translators for aligning community settings is a special case. In particular, we adopt the 7

Figure 4: We conducted a performance comparison of different base-level Language Models with basic tuning across various tasks. We compared the performance of GPT-2 [62], T5 Decoder [64], and OPT [101]. For each task, we select representative metrics to facilitate the comparison. For LTA task, we report the results on val set. fundamental settings proposed in [93; 103] for the Online Action Detection and Action Anticipation tasks. We leverage the settings introduced in [98] for the Action Segmentation task. The Online Captioning task follows the settings outlined in [104]. Similarly, we adhere to the settings specified in [27] for the Long-term Anticipation, Moment Query and Nature Language Query task. The Highlight Detection task builds upon the settings presented in [39]. 5.2 Main Results and Analysis Which language model performs better? Figure 4 presents the comparison results between three base-level LMs, GPT-2 [62], T5 Decoder [64], and OPT [101]. The results are obtained through the basic tuning method. We select representative metrics for each task for intuitive comparison. From the results, we can see that different language models have different performances on different video sequence understanding tasks. Both GPT-2 and OPT are better than T5 decoder in future prediction tasks (see AA and LTA in the figure). On the contrary, OPT is significantly better than GPT-2 and T5 decoder in OAD task. For Moment Retrieval tasks, we find that GPT-2 can still gain dominance (see MQ and NLQ in the figure). It is worth noting that T5 Decoder has a great advantage over GPT-2 and OPT in dense prediction tasks (see AS and HD in the figure). For online captioning, GPT-2 attains the best performance, compared with T5 Decoder and OPT. We suppose that using GPT-2 as video sequence reasoner Mbetter aligns the text generator Mg (also GPT-2) we used from [104]. In general, the structure and training strategy of the language model will result in different processing capabilities for video sequences and exhibit different adept abilities. In fact, when we calculated their average scores based on the results, we found that GPT-2 and T5 decoder were basically on par, and OPT was slightly worse than GPT-2 and T5 decoder. Model Action Top-5 Recall +Trainable Param (M) Basic 20.1 0 LoRA (r=1/2/4/8) 19.5/19.7/19.8/19.6 0.04/0.07/0.15/0.30 Prompt (r=1/2/4/8) 20.3/20.6/20.7/20.8 0.00/0.00/0.00/0.00 Prefix (r=1/2/4/8) 20.8/20.6/21.4/21.1 0.02/0.04/0.07/0.15 Partial (bias/F/L/FL) 20.5/20.6/20.5/20.8 0.1/7.09/7.09/14.18 Table 3: Impact of different tuning methods using GPT2 on OAD task. r denote the hyperparameter of the three PEFT tuning methods. “F” and “L” represent the first block and the last block in LMs. Which Tuning method performs better? To evaluate the influence of various tuning methods on performance, we opt OAD as the experimental object. It is a causal dense prediction task, providing a more realistic representation of performance alterations. Table 3 presents the Action Top-5 Recall achieved through the utilization of various tuning methods, along with the corresponding increase in trainable parameters compared to the basic tuning approach. We employ r as a uniform representation of the hyperparameter for the three PEFT tuning methods, and carry out experiments using r = 1/2/4/8. As depicted in the table, employing LoRA with different r results in a decline in performance. Conversely, the other tuning methods exhibit performance improvements of at least 0.2 points in the Action Top-5 Recall metric. Although fine-tuning the first or last block can yield performance gains, it also entails a significantly larger number of trainable parameters compared to the other methods. Remarkably, when employing prefix tuning with r = 4, the model achieves the best outcome, attaining an Action Top-5 Recall of 21.4, surpassing the basic tuning method by 1.3 points. Comparison to the state-of-the-art methods. Table 4 presents the evaluation results for seven video sequence understanding tasks. It is important to note that the OC task is not included in this analysis due to the lack of comparable sequence-level methods. To thoroughly assess the effectiveness of VideoLLM, we conduct a comparative analysis with other cutting-edge methods that are specifically tailored to individual tasks. The reported results for VideoLLMrepresent the most favorable performance achieved from numerous combinations. To evaluate the OAD task, we 8

Model Trainable Param OAD AA AS LTA MQ NLQ HD O U T O U T F1@25 ED Acc V↓ N↓ A↓ mAP R1@0.3 R1@0.5 mAP non-language-model-based method LSTR‡ [93] 27.19M 22.6 18.7 20.7 - - - - - - - - - - - - - Testra‡ [103] 27.70M 23.2 19.0 20.9 15.5 12.4 11.9 - - - - - - - - - - ASFormer† [98] 1.13M - - - - - - 27.3 16.2 31.0 - - - - - - - SS-TCN† [21] 0.80M - - - - - - 20.2 14.3 56.5 - - - - - - - MS-TCN [21] 0.80M - - - - - - 52.9 61.4 65.1 - - - - - - - Ego4D [27] 30.33M - - - - - - - - - 71.7 73.6 92.5 - - - - VSGN [51] 3.80M - - - - - - - - - - - - 6.03 - - - InternVideo [10] 6.72M - - - - - - - - - - - - - 14.4 9.60 - Moment DETR [39] 2.56M - - - - - - - - - - - - - - - 36.5 language-model-based method VideoLLM 2 - 15M 23.4 20.2 21.6 15.4 12.6 12.0 55.3 63.4 65.7 72.1 72.5 92.1 6.09 15.5 10.1 37.7 Table 4: Comparison with the state-of-the-art models on 7 video sequence understanding tasks. For OAD and AA tasks, we evaluate Overall, Unseen and Tail Action Top-5 Recall. We follow [27] to evaluate the LTA task with edit distance (ED) of Verb, Noun andAction on the test set, and other tasks are evaluated onvalidation set. We compare performance through the average mAP of tIoU thresholds between 0.1 and 0.5. ‡ denote the results we reproduced. † denotes the results that we align the method with our adaption principle. reproduce the existing state-of-the-art methods [93; 103] and adopt the same evaluation metrics [15] as the AA task. Notably, we ensure a fair comparison by excluding the data augmentation techniques employed by Testra [103]. Our model demonstrates higher or comparable performance in both the OAD and AA tasks. Particularly, our approach achieves a higher Unseen Action Top-5 Recall, highlighting the ability of utilizing LLMs to ensure and potentially enhance generalization in unseen scenarios. For the AS task, our model outperforms the state-of-the-art method MS-TCN [21] in terms of F1@25, edit distance, and accuracy. It is worth emphasizing that our adaptation principle solely relies on the sequence modeling capability of the LMs themself, without introducing any local prior operator or multi-stage refinement. This observation emphasizes that a language sequence-trained model can serve as a robust initialization for video sequence modeling. We also apply our adaptation principles to MS-TCN [21] and ASFormer [98], with the corresponding results presented in the table. In the table, SS-TCN† refers to the deep network with a single-stage supervision mentioned in the MS-TCN paper. These results demonstrate a significant inferiority to our single-stage adaptation. Furthermore, we compare VideoLLMagainst state-of-the-art or baseline methods on multiple subtasks, namely LTA, MQ, and NLQ, of Ego4D [27]. The evaluation conducted on the LTAv2 test set, using the EvalAI platform, shows that our model outperforms the official baseline methods. Moreover, under the constraints of the adaptation principle, our model exhibits a slight performance superiority over VSGN [102], which employs an anchor-based prior setting for the MQ task. In the realm of visual-language tasks, our models exhibit substantial superiority over existing state-of-the-art methods [10; 39]. This finding underscores the impressive performance of language models once the vision-to-language semantic translation is accomplished. Furthermore, in addition to the performance comparisons, we also compare the trainable parameters with these methods. The table reveals that our method necessitates approximately 2M to 15M learnable parameters across multiple tasks, with most of these parameters primarily utilized in semantic translator and task head. This substantiates the parameter efficiency of our proposed framework. In summary, these results convincingly demonstrate the adaptability of our proposed framework across a diverse range of video sequence understanding tasks, each with its own unique settings. 0 500 1,000 1,500 2,000 2,500 20 21 22 23 Parameters (M) Recall Top5 Action GPT-2 T5 Decoder OPT Figure 5: Performance of GPT-2, T5 Decoder and OPT with different number of total parameters. Scale of LLM. We also assess the scalability of utilizing LLMs as video sequence reasoners for our approach, through experimental evaluations conducted on the OAD task. Figure 5 displays the Action Top-5 Recall achieved by employing LLMs with varying scales of total parameters. In these experiments, we scale up three decoder-only LLMs, namely GPT-2, T5 Decoder, and OPT, and solely fine-tune two projectors using the basic tuning method. This ensures a comprehensive evaluation of the intrinsic capabilities possessed by these LLMs. As depicted in the figure, when utilizing language models with parameter sizes less than 2B, compelling evidence suggests that larger models yield more substantial improvements in video sequence reasoning. Among the three models, it is worth noting that OPT-1.3B 9

yields the most favorable results, achieving a remarkable 23.4 Action Top-5 Recall. Furthermore, when considering the overall performance improvement trend observed during the scaling-up process, it becomes evident that OPT outperforms T5 Decoder, which, in turn, surpasses GPT-2. However, for larger LLMs, their performance begins to decline. One plausible explanation for this phenomenon is that the dimension-expansion projector causes the model to overfit, as the dimension of the extracted feature sequence is typically less than 2048. In conclusion, these experiments effectively demonstrate the scalability of our method to LLMs, highlighting their potential for adapting video sequence reasoning tasks. Model #Param O U T OPT [101] 6.7B 22.1 19.9 21.6 T5 Decoder [64] 6.5B 19.8 20.2 21.1 LLaMA [75] 7B 21.8 20.1 21.1 Table 5: Performance of larger and advanced language models, i.e OPT-6.7B, T5-Decoder-6.5B and LLaMA7B on OAD task. Advanced LLM. We further scale up OPT and T5 decoder to 6.7B and utilize the latest 7B LLaMA [75] model. The performance of T5 and OPT, as depicted in Table 5, continues to align with the declining trend observed in Figure 5. Notably, the performance of LLaMA closely approximates that of OPT. Encoder vs. Decoder. We conducted experiments to compare the performance of bidirectional and unidirectional sequence reasoners on three tasks: AS, HD , and NLQ. For the bidirectional sequence reasoner, we employed the T5 [64] encoder, while the unidirectional sequence reasoner utilized the T5 decoder. A comprehensive comparison of all task metrics is presented in Table 6. Task Metric Decoder Encoder AS F1@25 25.3 51.1 ED 34.0 55.7 Acc 44.0 60.7 HD mAP 37.4 37.7 HiT@1 61.0 61.6 NLQ Rank1@0.3 6.5 7.4 Rank1@0.5 3.6 3.5 Rank1@Mean 5.1 5.5 Table 6: Impact of encoder and decoder as video sequence reasoner on AS, HD and NLQ tasks. Here encoder and decoder are T5 [64]. As evident from the table, the bidirectional reasoner consistently outperformed the unidirectional reasoner in most cases. This discrepancy is particularly prominent in AS tasks, where the bidirectional reasoner exhibited a significantly higher level of performance compared to its unidirectional counterpart. This may be attributed to the importance of bidirectional attention in confirming temporal correlations and pre-post-action relationships within a complete event during action segmentation. In the case of visual-language tasks, HD and NLQ, the bidirectional reasoner also showcased a slight advantage over the unidirectional reasoner. However, it is worth noting that the Rank1@0.3 obtained by the OPT on the NLQ task, as depicted in Figure 4, is comparable to that achieved by the T5 Encoder (7.3 vs 7.4). This suggests that the decoder-only unidirectional reasoner holds the potential to achieve performance on par with the bidirectional reasoner. 6 Conclusion and Future Work In this paper, we propose a novel video understanding framework calledVideoLLM, which transfers the sequence causal reasoning abilities of large language models (LLMs) from natural language processing to video understanding. The VideoLLMframework comprises a well-designed Modality Encoder and a Semantic Translator, which convert inputs from different modalities into a unified token sequence. This sequence is then fed into a decoder-only reasoner realized by the large-scale language pretrained and parameter-frozen LLM, which possesses the ability to decode and output meaningful high-level semantics. With the help of simple task heads, the output of the LLM corresponds to various specific video understanding tasks. Extensive experiments were conducted on eight tasks from four different datasets using multiple LLMs and fine-tuning methods to evaluate the effectiveness of VideoLLM. The experimental results demonstrate that LLMs’ comprehension and reasoning abilities can be effectively applied to video understanding tasks. In our future work, we will further explore the potential of LLM. Building upon time series reasoning, we aim to incorporate serialized information about the appearance of video frames, enabling LLM to achieve a more comprehensive video understanding across the entire spatiotemporal dimension. 10

References [1] Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716–23736. [2] Ali, A.; Touvron, H.; Caron, M.; Bojanowski, P.; Douze, M.; Joulin, A.; Laptev, I.; Neverova, N.; Synnaeve, G.; Verbeek, J.; et al. 2021. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34: 20014–20027. [3] Arnab, A.; Dehghani, M.; Heigold, G.; Sun, C.; Lucic, M.; and Schmid, C. 2021. ViViT: A Video Vision Transformer. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 6816–6826. IEEE. [4] Bain, M.; Nagrani, A.; Varol, G.; and Zisserman, A. 2021. Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval. In2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 1708–1718. IEEE. [5] Bao, H.; Dong, L.; Piao, S.; and Wei, F. 2022. BEiT: BERT Pre-Training of Image Transformers. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [6] Bertasius, G.; Wang, H.; and Torresani, L. 2021. Is Space-Time Attention All You Need for Video Understanding? In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, 813–824. PMLR. [7] Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020. Language Models are Few-Shot Learners. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. [8] Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020. End-to-End Object Detection with Transformers. In Vedaldi, A.; Bischof, H.; Brox, T.; and Frahm, J., eds., ECCV, volume 12346 of Lecture Notes in Computer Science, 213–229. [9] Carreira, J.; and Zisserman, A. 2017. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In CVPR, 4724–4733. [10] Chen, G.; Xing, S.; Chen, Z.; Wang, Y.; Li, K.; Li, Y.; Liu, Y.; Wang, J.; Zheng, Y.-D.; Huang, B.; et al. 2022. InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges. arXiv preprint arXiv:2211.09529. [11] Chen, G.; Zheng, Y.; Wang, L.; and Lu, T. 2021. DCAN: Improving Temporal Action Detection via Dual Context Aggregation. CoRR, abs/2112.03612. [12] Chen, J.; Guo, H.; Yi, K.; Li, B.; and Elhoseiny, M. 2022. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18030–18040. [13] Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; Schuh, P.; Shi, K.; Tsvyashchenko, S.; Maynez, J.; Rao, A.; Barnes, P.; Tay, Y.; Shazeer, N.; Prabhakaran, V.; Reif, E.; Du, N.; Hutchinson, B.; Pope, R.; Bradbury, J.; Austin, J.; Isard, M.; Gur-Ari, G.; Yin, P.; Duke, T.; Levskaya, A.; Ghemawat, S.; Dev, S.; Michalewski, H.; Garcia, X.; Misra, V.; Robinson, K.; Fedus, L.; Zhou, D.; Ippolito, D.; Luan, D.; Lim, H.; Zoph, B.; Spiridonov, A.; Sepassi, R.; Dohan, D.; Agrawal, S.; Omernick, M.; Dai, A. M.; Pillai, T. S.; Pellat, M.; Lewkowycz, A.; Moreira, E.; Child, R.; Polozov, O.; Lee, K.; Zhou, Z.; Wang, X.; Saeta, B.; Diaz, M.; Firat, O.; Catasta, 11

M.; Wei, J.; Meier-Hellstern, K.; Eck, D.; Dean, J.; Petrov, S.; and Fiedel, N. 2022. PaLM: Scaling Language Modeling with Pathways. CoRR, abs/2204.02311. [14] Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J. G.; Le, Q. V.; and Salakhutdinov, R. 2019. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Korhonen, A.; Traum, D. R.; and Màrquez, L., eds., Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, 2978–2988. Association for Computational Linguistics. [15] Damen, D.; Doughty, H.; Farinella, G. M.; Furnari, A.; Kazakos, E.; Ma, J.; Moltisanti, D.; Munro, J.; Perrett, T.; Price, W.; and Wray, M. 2022. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. Int. J. Comput. Vis., 130(1): 33–55. [16] Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A largescale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, 248–255. Ieee. [17] Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Burstein, J.; Doran, C.; and Solorio, T., eds., Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), 4171–4186. Association for Computational Linguistics. [18] Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [19] Fan, H.; Xiong, B.; Mangalam, K.; Li, Y.; Yan, Z.; Malik, J.; and Feichtenhofer, C. 2021. Multiscale Vision Transformers. In2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 6804–6815. IEEE. [20] Fang, Y.; Wang, W.; Xie, B.; Sun, Q.; Wu, L.; Wang, X.; Huang, T.; Wang, X.; and Cao, Y. 2022. EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. CoRR, abs/2211.07636. [21] Farha, Y. A.; and Gall, J. 2019. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3575–3584. [22] Feichtenhofer, C.; Fan, H.; Li, Y.; and He, K. 2022. Masked Autoencoders As Spatiotemporal Learners. In NeurIPS. [23] Feichtenhofer, C.; Fan, H.; Malik, J.; and He, K. 2019. SlowFast Networks for Video Recognition. InICCV, 6201–6210. [24] Furnari, A.; and Farinella, G. M. 2019. What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention. In International Conference on Computer Vision (ICCV). [25] Geest, R. D.; Gavves, E.; Ghodrati, A.; Li, Z.; Snoek, C.; and Tuytelaars, T. 2016. Online Action Detection. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V, volume 9909 of Lecture Notes in Computer Science, 269–284. Springer. [26] Girdhar, R.; and Grauman, K. 2021. Anticipative Video Transformer. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 13485–13495. IEEE. 12

[27] Grauman, K.; Westbury, A.; Byrne, E.; Chavis, Z.; Furnari, A.; Girdhar, R.; Hamburger, J.; Jiang, H.; Liu, M.; Liu, X.; et al. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18995–19012. [28] He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, 770–778. IEEE Computer Society. [29] Heilbron, F. C.; Escorcia, V.; Ghanem, B.; and Niebles, J. C. 2015. ActivityNet: A large-scale video benchmark for human activity understanding. InCVPR, 961–970. [30] Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; de Las Casas, D.; Hendricks, L. A.; Welbl, J.; Clark, A.; Hennigan, T.; Noland, E.; Millican, K.; van den Driessche, G.; Damoc, B.; Guy, A.; Osindero, S.; Simonyan, K.; Elsen, E.; Rae, J. W.; Vinyals, O.; and Sifre, L. 2022. Training Compute-Optimal Large Language Models. CoRR, abs/2203.15556. [31] Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [32] Ioffe, S.; and Szegedy, C. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Bach, F. R.; and Blei, D. M., eds., Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, 448–456. JMLR.org. [33] Jaegle, A.; Gimeno, F.; Brock, A.; Vinyals, O.; Zisserman, A.; and Carreira, J. 2021. Perceiver: General perception with iterative attention. In International conference on machine learning, 4651–4664. PMLR. [34] Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.; Parekh, Z.; Pham, H.; Le, Q. V.; Sung, Y.; Li, Z.; and Duerig, T. 2021. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, 4904–4916. PMLR. [35] Joshi, M.; Chen, D.; Liu, Y.; Weld, D. S.; Zettlemoyer, L.; and Levy, O. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Trans. Assoc. Comput. Linguistics, 8: 64–77. [36] Kay, W.; Carreira, J.; Simonyan, K.; Zhang, B.; Hillier, C.; Vijayanarasimhan, S.; Viola, F.; Green, T.; Back, T.; Natsev, P.; Suleyman, M.; and Zisserman, A. 2017. The Kinetics Human Action Video Dataset. CoRR, abs/1705.06950. [37] Khan, M. A.; Javed, K.; Khan, S. A.; Saba, T.; Habib, U.; Khan, J. A.; and Abbasi, A. A. 2020. Human action recognition using fusion of multiview and deep features: an application to video surveillance. Multimedia tools and applications, 1–27. [38] Kuehne, H.; Arslan, A.; and Serre, T. 2014. The language of actions: Recovering the syntax and semantics of goal-directed human activities. InProceedings of the IEEE conference on computer vision and pattern recognition, 780–787. [39] Lei, J.; Berg, T. L.; and Bansal, M. 2021. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34: 11846– 11858. [40] Lei, J.; Li, L.; Zhou, L.; Gan, Z.; Berg, T. L.; Bansal, M.; and Liu, J. 2021. Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 7331–7341. Computer Vision Foundation / IEEE. 13

Made with FlippingBook

RkJQdWJsaXNoZXIy NzQwMjQ=