Human-inspired Perspectives: A Survey on AI Long-term Memory

According to Atkinson et al. [6], the sensory register is responsible for receiving and temporarily storing information from sensory systems, allowing information like visual stimuli to be retained in a highly precise form for a short period. They used Sperling et al.’s 1960 visual exposure experiment [62] to illustrate the existence of the sensory register. In Sperling et al.’s experiment, subjects were briefly exposed to a series of printed letters and then asked to recite them after exposure. They discovered that for exposures lasting between 15 to 500 milliseconds, subjects could correctly report an average of slightly over four letters. When the stimuli contained four or fewer letters, subjects could report them with almost 100% accuracy. This finding demonstrated that individuals can retain brief visual stimuli through a mechanism that reflects the function of the sensory register. Information stored in the sensory register rapidly decays and either vanishes or is transferred to short-term memory within a very short time, providing raw material for subsequent memory processing [6]. This underscores the sensory register’s critical role in the initial stages of information processing and memory formation.

In 1968, Atkinson et al. [6] summarized the concept of “short-term storage” based on previous research findings. These findings included the work of Peterson et al. [63], who observed variations in participants’ ability to recall consonant letters¹¹ 1 Consonant letters are letters of the alphabet that are not vowels (a, e, i, o, u). at different time intervals, and Milner et al. [64, 65, 66], who discovered that patients who had their bilateral hippocampal region removed could recall short-term information like normal individuals, but could not retain new long-term information. Atkinson et al. indicated that short-term storage temporarily holds information, and if this information is not maintained through control processes such as rehearsal, it will decay and vanish within a short period, approximately 30 seconds, as noted by Milner et al. [66].

However, the short-term storage theory proposed by Atkinson et al. has limitations in explaining complex human information processing, particularly regarding how information is processed and stored within the short-term store. They did not systematically introduce the division and details of information processing in the short-term store. Addressing these limitations, Baddeley et al. [60] proposed the working memory theory in 1974, suggesting that working memory is used for the temporary storage and manipulation of information. For example, when performing mental arithmetic, one must not only store numbers temporarily but also apply arithmetic operations to them sequentially. Another example is that recalling numerical information from an oral narrative and completing a reasoning task both require storing and manipulating the information within working memory. Similarly, taking language comprehension to exemplify working memory, individuals must hold words and phrases in memory while simultaneously integrating them into larger syntactic structures and deriving meaning. The manipulations in these examples are essential for tasks like problem-solving, decision-making, and reasoning, which rely on the dynamic handling of information within working memory [60]. From 1974 to 2000, a series of works by Baddeley et al. [60, 67, 68] expanded the working memory theory, dividing it into four components: the Central Executive, Phonological Loop, Visuospatial Sketchpad, and Episodic Buffer. We explain these components briefly while summarizing them in Fig. 2:

• $\bullet$

  Central Executive. Baddeley et al. [60, 67] highlighted that the central executive is a critical component of working memory, designed to control and supervise the operation of various modules within working memory, playing a pivotal role in resource allocation similar to that of a CPU (Central Processing Unit) in a computer. When individuals need to process complex transient information, such as temporarily remembering a long sequence of phone numbers, the central executive directs the individual to concentrate more on this task.

• $\bullet$

  Phonological Loop. Baddeley et al. [67] further categorized the phonological loop as a type of working memory, aiming to represent humans’ transient memory for verbal information. The phonemic loop refers to the looping of phonemes in the brain, like an “inner ear” that holds information in a speech-based form for a short time. It is responsible for storing words we hear and silently repeat. The phonemic buffer temporarily stores phonological information, such as phrases, sentences, and numbers. This buffer acts like an “inner voice”, enabling us to repeat information in a loop to maintain and refresh it, preventing it from fading. A demonstration of the phonological loop in action is the process of memorizing a string of phone number by reciting it repeatedly (and silently) until we are able to jot it down in a notebook. The phonological loop plays a crucial role in memorizing linguistic information, such as new words [69].

• $\bullet$

  Visuospatial Sketchpad. According to Baddeley et al. [67], the visuospatial sketchpad temporarily stores and processes visual and spatial information. It integrates visual and spatial information from perception with information stored in long-term memory. This helps in recognizing objects, understanding scenes, and performing tasks that require spatial awareness, such as driving, reading maps, engaging in sports, and visualizing and manipulating objects in mind. In 1994, the theory proposed by Kosslyn et al. [70] provided an enhanced understanding of visuospatial sketchpad. This theory delineates that visuospatial sketchpad can be divided it into three steps: image generation, image maintenance, and image rotation. This process is analogous to solving geometry problems that involve rotating shapes. First, an individual creates an initial mental image of the shape. Next, they maintain the image in their mind. Finally, they imagine how the shape would rotate to match the target image. These researches highlight the critical role of the visuospatial sketchpad, demonstrating its importance in cognitive tasks that require mental manipulation of visual information.

• $\bullet$

  Episodic Buffer. Baddeley et al. [68] introduced the episodic buffer as a component of working memory. Episodic buffer facilitates the transfer of information between working memory and long-term memory. Specifically, the episodic buffer enables the movement of significant information from working memory to long-term memory and allows fragments of long-term memory that are relevant to the current context to be retrieved and placed into the episodic buffer. This process supports the formation of new long-term memory and the effective use of long-term memory.

In 1968, Atkinson et al. [6] suggested that the short-term store includes a mechanism for transferring information to the long-term store. Unlike the sensory register and short-term store, information in the long-term store does not rapidly decay or vanish. This information is relatively permanent, although it can be modified by other information or become temporarily inaccessible. According to Atkinson et al., the information in long-term store, referred to as “traces”, does not strictly conform to the “all-or-none” characteristic. Instead, “traces” can exist in intermediate states such as “decay” and “interference”. “Decay” refers to the weakening of long-term store “traces”, such as information that deteriorates over time due to a lack of review, for example, certain words in a long sentence. “Interference” describes the mutual influence between “traces”, such as when learning word A followed by a similar word B leads to confusion during recall. The term "long-term store" used by Atkinson et al. refers specifically to memory as a physical or functional analogy for information storage. In a broader context, this concept is referred to as “long-term memory”.

In 1972, Tulving et al. [24] highlighted the complex nature of long-term memory by introducing two distinct types: episodic memory and semantic memory. Episodic memory relates to personal past events and experiences, such as an experience of mountain climbing or a recent vacation. In contrast, semantic memory relates to general information, such as “Paris is the capital of France” or understanding the concept of gravity. Furthermore, in 1985, Tulving et al. [26] classified memory involving the acquisition of perceptual feedback and motor skills as procedural memory [71, 72, 73]. An example of this is learning to ride a bicycle, which requires acquiring motor skills through balance feedback. Based on these existing findings, Tulving et al. [26] assumed three memory systems: procedural memory, semantic memory, and episodic memory.

Graf et al. [74] introduced an alternative method of memory classification based on whether the memory can be consciously recalled, categorizing memory into explicit and implicit types. Implicit memory operates without conscious recollection; for example, maintaining balance on a bicycle after learning to ride involves implicit memory, as it does not require active thought. In contrast, explicit memory necessitates conscious recollection, such as when recalling a specific experience or piece of knowledge while writing an article. According to the classification proposed by Tulving et al. [26], episodic memory and semantic memory are types of explicit memory [24], whereas procedural memory falls under implicit memory [75].

Having established the categorization of long-term memory types, we now discuss the specifics of episodic, semantic, and procedural memory and show them briefly in Fig. 2:

• $\bullet$

  Episodic Memory. Tulving et al. [24] defined episodic memory in 1972 as a type of memory specific to individuals, related to time and space, involving personal experiences, events, and contexts. In 1973, Tulving et al. [25] noted that input stimuli such as sound and visual information are processed for episodic memory storage through specific encoding, which combines perceived information with specific contexts and experiences to form memory trace (discussed in the leading paragraph of Sec. 3.1.3) that aids in subsequent retrieval. For successful recall of this trace, similar input stimuli need to be provided. For example, adults often recall childhood experiences upon seeing childhood photos, which stimulates the recall of relevant long-term memory traces. Furthermore, episodic memory also facilitates learning from past experiences and applying this knowledge to future decisions, thereby enhancing an individual’s ability to adapt to their environment [76]. The hippocampus in the brain plays a vital role in learning episodic memory by encoding them through a process that separates similar personal experiences and stores them distinctly, thereby minimizing interference between different episodic memory [77, 78, 79].

• $\bullet$

  Semantic Memory. According to Tulving et al.’s perspective [24] in 1972, semantic memory encompasses an individual’s knowledge about words and other linguistic symbols, including the relationships between them, and the rules, formulas, and algorithms for manipulating them. It functions to receive, retain, and convey semantic information about words and concepts. Unlike episodic memory, which is tied to personal experiences, semantic memory deals with general information. An example of semantic memory is the long-term memory formed by learning knowledge in a specific field or by extracting general rules from past experiences. The management of semantic memory involves multiple brain regions, such as the left prefrontal cortex (LPC) being involved in the retrieval of semantic information, and the temporal lobes storing specific information related to objects [80, 81, 82].

• $\bullet$

  Procedural Memory. Tulving et al. [26] describe procedural memory as the memory developed from learning motor skills through feedback and perceptual abilities. This type of memory is intrinsically linked to the immediate temporal and spatial contexts and does not depend on information beyond the current stimuli. This allows for instantaneous reactions based on direct perception. For example, learning to ride a bicycle involves the sensation of losing balance, which triggers discomfort and gradually leads to the development of balancing skills. Once these skills are fully acquired, individuals can effortlessly maintain balance while riding, demonstrating the role of procedural memory in learning motor skills. Brain regions such as the supplementary motor area (SMA), primary motor cortex (M1), prefrontal cortex, and cerebellum play crucial roles in skill consolidation for the formation of procedural memory [83].

Based on the perspective of Tulving et al. [25], memory storage is achieved by processing input stimuli through specific encoding (detailed in the episodic part of Sec. 3.1.3). The hippocampus and cortex of the brain are involved in the encoding of memory, with the hippocampus encoding unique episodic information through its pattern separation properties and the cortex encoding more generalized information through its distributed representations [77, 78, 96, 79]. Sensory register (Sec. 3.1.1), short-term store (Sec. 3.1.2), and long-term store (Sec. 3.1.3) have distinct methods for storing and encoding information at different stages of the memory process [29]. The long-term store, in particular, encodes memory through four distinct encoding strategies: “meaning”, “association”, “repetition”, and “organization”, as discussed below:

• $\bullet$

  Meaning refers to encoding information that holds significant importance, such as events that deeply impact individuals.

• $\bullet$

  Association involves pairing related objects with the encoded information, like remembering a “bird” based on characteristics such as flying.

• $\bullet$

  Repetition indicates the likelihood of information being transferred to the Long-term Store, as repeated exposure to scenes enhances memory retention.

• $\bullet$

  Organization suggests that the storage location in the Long-term Store is influenced by the content itself, such as recalling “hamburger” in the brain’s “food section”.

In summary, it is evident that memory storage, especially for long-term memory, is a relatively complex process. It involves processing perception, which we call “encoding”, to produce information stored in the brain. The quality of memory storage largely determines the subsequent quality of memory retrieval and the rate of memory forgetting.

According to Tulving et al. [25], memory retrieval involves a conscious search and identification of information stored in memory. They emphasized that this process is not merely the activation of learned associations or the recall of stored traces. Instead, it involves complex interactions between stored information and specific features of the current retrieval context. Providing cues similar to the original stimuli that formed the memory can enhance retrieval effectiveness. An example of this is individuals visiting a place they have previously stayed at, which is likely to evoke the events they experienced there. The hippocampus of brain is involved in the complex process of retrieving specific memory based on cues [77, 79].

During the 1970s, one of the most prominent theories of memory retrieval, coinciding with Tulving et al.’s research, was the Generation-Recognition Theory [29, 97, 25]. This theory outlines the stages of memory retrieval, dividing them into the generation stage and the recognition stage. In the generation stage, individuals attempt to implicitly generate possible responses from memory, such as words, images, or other information associated with given cues. The recognition stage involves evaluating the responses generated in the generation stage to determine whether they meet specific acceptance criteria. In the recognition stage, the generated responses are matched against stored memory traces to see if they align with the specific information that was originally learned. Recognition requires not only identifying whether the response is familiar, but also confirming its context and correctness. For instance, it involves determining whether a recalled fact was encountered in a specific learning context or whether an image accurately represents an event experienced previously. It is concluded that the effectiveness of retrieval is enhanced if the process provides cues similar to those used during encoding [25, 98].

Memory forgetting is commonly perceived as the erasure of memory from storage. However, Atkinson et al. [29] proposed in 1969 that memory forgetting and related phenomena are often not due to the disappearance of information from long-term storage, but rather failures in the retrieval process. As individuals are exposed to increasing amounts of external information, their long-term memory grows correspondingly vast. The expanding scope of memory retrieval can cause interference between similar memory, leading to retrieval failures. This interference leads to the phenomenon known as “memory forgetting”, which is a form of passive forgetting. On the other hand, some studies [99, 100, 101, 102] suggested that in addition to passive forgetting, humans can also achieve active regulation of the memory system through a mechanism that suppresses redundant information to facilitate the storage and retrieval of key memory. This type of forgetting mechanism is termed “active forgetting”.

The systematic study of memory forgetting dates back to the late 19th century. In 1885, Ebbinghaus et al. [87] demonstrated that 38 repetitions over three days had a similar effect to 68 repetitions in one day. In 1897, Jost’s Law  [88] indicated that the probability of forgetting distant memory is higher than that of recent memory. Later, in 1957, Underwood et al. [103] explained a phenomenon where previously learned or concurrently learned information affects the forgetting of specific memory, which is termed “interference”. These studies elucidate factors that can be utilized to reduce the probability of memory forgetting, including long intervals, high recency, and low interference, providing insights into effective strategies for retaining key information. For instance, leveraging the three strategies, individuals can improve word test results by reviewing words over extended intervals instead of repeating them in a short period (long intervals), focusing on word review as tests approach (high recency), and using specific markers to distinguish easily confused words (low interference). Moreover, there are active mechanisms in the brain to alleviate memory forgetting. Some studies suggest that the hippocampus engages in memory replay during certain phases, such as sleep, to consolidate memory and prevent forgetting. For example, place cells in the hippocampus spontaneously replay past trajectories to strengthen spatial memory [104]. Additionally, coordinated activity between the hippocampus and the visual cortex supports the replay of episodic memory, further aiding memory consolidation [92].

Studies that employ specific functions to describe memory forgetting curves are also crucial to our understanding of memory patterns. In 1970, Wickelgren et al. [105] demonstrated through a letter memory experiment that the strength of memory traces in short-term memory (working memory) decays with increasing delay and/or interference, following an exponential form. Similarly, in 1985, White et al. [106] found that the memory forgetting curve in pigeons’ discriminative ability to sample stimuli under different delay conditions could be well described by a simple negative exponential function. This function has two parameters: one for the initial discriminative ability and another for the rate at which this ability decreases with increasing delay intervals. However, in 1997, Wixted et al. [89] conducted a series of experiments showing that forgetting curves are better described by a power function than by an exponential function, regardless of whether arithmetic or geometric averaging is used. Nevertheless, Anderson et al. [90] argued that the arithmetic averaging of exponential memory curves might produce an artifactual power memory curve. These findings highlight the complexity of accurately modeling memory forgetting curves and suggest that further research is needed to resolve these discrepancies.

In summary, memory forgetting can be caused by interference between memory that leads to retrieval failure; active forgetting of redundant information can facilitate the storage and retrieval of critical memory; long intervals, high recency, and low interference reduce the probability of memory forgetting; the memory replay mechanism of the hippocampus contributes to the alleviation of memory forgetting; and the quantitative modeling of memory forgetting helps researchers understand the patterns of memory.

Non-parametric memory can be stored in various media, such as databases, file systems, and computer memory. File systems are used by computer operating systems to efficiently store, organize, manage, and access data on disks [254, 255]. Computer memory, on the other hand, provides temporary storage space for a computer [256, 257]. Among these storage options, databases are particularly noteworthy for their scalability and maintenance efficiency, making them the preferred medium for accessing non-parametric memory. The databases provide diverse storage options for different types of non-parametric memory. For example, sparse features derived from user interactions can be stored in relational databases [243] for use in recommender systems [258], whereas vectorized document chunks can be stored in vector databases [239, 240] for RAG systems [252, 253]. The following sections illustrate how these two types of database are utilized for storing non-parametric memory.

• $\bullet$

  Relational database. Relational databases store data using a relational model [242, 243]. For example, consider a structured data “tiger” - “belongs to” - “feline”. In a relational database, this can be stored as a record in a table representing a collection of animals. These databases support encoding of structured non-parametric memory. Operations on data in relational databases can be performed using Structured Query Language (SQL) [259]. Several studies leveraged LLMs to generate SQL operations for storing, manipulating, and retrieving non-parametric memory in relational databases. For instance, Luo et al. [10] utilized LLMs to generate SELECT SQL operations, which are used to select entries that satisfy given conditions, to retrieve long-term memory from a relational database. Similarly, Hu et al. [241] used LLMs to convert the original queries to multi-step SQL commands to perform INSERT (insert a new entry), UPDATE (update an existing entry), SELECT, and DELETE (remove an existing entry) operations, which enables the flexible handling of non-parametric memory.

• $\bullet$

  Vector Database. Traditional relational databases require highly structured data and are not effective at data types like texts, images, and audios [260]. To address this limitation, vector databases have been developed to store vectors encoded from various types of data, making them more suitable for unified storage of these unstructured data types, and supports more efficient retrieval based on vector similarity [239, 240]. Non-parametric memory suitable for storage in vector databases can be derived from representation vectors of different modalities (e.g., text, images) obtained through contrastive learning [234, 235, 261], language representation vectors acquired via pre-training methods like masked word prediction and next sentence prediction [231, 262, 263], or key and value vectors from the attention mechanism in Transformer models [264, 265, 266, 198]. Moreover, compared to traditional indexing techniques in relational databases such as B-trees and hash tables, indexing techniques specialized for high-dimensional vectors can be adopted in vector databases, such as indexing of Locality-Sensitive Hashing (LSH) [224] and Hierarchical Navigable Small World (HNSW) graphs [227]. These indexing methods enable the efficient storage of non-parametric memory. Multiple studies have demonstrated the advantages of using vector databases to store non-parametric memory. For example, Zhang et al. [250] utilized vector databases to provide personalized long-term memory for LLMs, which enhances the personalization of AI assistants. Shen et al. [13] explored the use of vector databases to enhance the long video understanding capability of AI systems by encoding the natural language descriptions of video frames into vectors. This can be considered as long-term memory within video contexts.

In summary, relational databases can be used to store structured data in non-parametric memory [242, 243, 10, 241], while vector databases are suitable for storing non-parametric memory of various modalities [239, 250]. Considering the diversity of non-parametric memory, vector databases are often more appropriate for their storage.

Non-parametric memory functions as a collection of stored information, and its retrieval process is similar to the concept of information retrieval. The retrieval of non-parametric memory involves locating the most relevant $K$ memory fragments. This process consists of two main steps. It first calculates the relevance score between the query $q$ and each fragment $m$ in the non-parametric memory, and then ranks the fragments $m$ based on their relevance scores and selects the top-$K$ fragments $m$ with the highest scores to form the retrieval results [267, 268, 232].

Based on our review of existing work, we identify two primary retrieval methods suitable for non-parametric memory including sparse methods based on bag-of-words models and dense methods based on deep representation learning for vector encoding and relevance score calculation [269, 270, 232, 271]. Sparse retrieval, which requires both the query and the memory fragment to be in text format, involves methods such as BM25 [236, 237] and TF-IDF [238]. In contrast, dense retrieval matches queries with memory fragments in deep semantic representations format [230, 231] that cater to various modalities, including text, images, audio and code [233, 234, 235]. In this case, memory fragments are typically stored in a vector database to enable fast retrieval. Dense retrieval methods often utilize deep models, such as Transformer-based models [3], to encode the query $q$ and the memory fragment $m$ into vectors of equal dimension. The ranking score can be obtained by calculating the Euclidean distance [229] or by performing a dot product [232] between the query vector and the memory fragment vector.

As dense retrieval supports deep semantic matching across various modalities, it emerges as a more effective method for non-parametric memory. In applying dense retrieval to non-parametric memory storage, we identify three key challenges along with potential solutions found in the literature: (1) efficiently managing large-scale memory retrieval, (2) improving retrieval effectiveness for downstream tasks, and (3) augmenting query for memory retrieval.

• $\bullet$

  Handling of Large-Scale Memory Retrieval. Efficient retrieval from large-scale non-parametric memory, including the effective assessment of the matching scores between items and efficient sorting, poses a significant challenge. Approximate Nearest Neighbor Search (ANNS) [222, 223, 227, 228] is a commonly used method for efficiently retrieving large-scale non-parametric memory, effectively reducing computational complexity and finding results that are approximately accurate. ANNS can be efficiently realized through methods such as hashing functions that map similar vectors into the same bucket for focused searching [224], quantization techniques that decompose high-dimensional vectors into sub-vectors and utilize their centroids as matching objects for reducing computational complexity of matching score calculations [225, 226], and hierarchical approaches that employ layered sparse graphs [227] or clusters [228] to implement localized search. Furthermore, using graph structures to construct non-parametric memory is highly effective for processing retrieval tasks from large-scale non-parametric memory, particularly for queries involving multiple entities [180, 179]. For instance, consider the query “In which of Antoine’s books is the planet B612 mentioned?”. If there are only two relevant text fragments in text-based non-parametric memory, one stating “The Little Prince mentions the planet B612”, and the other noting “Antoine is the author of The Little Prince”, traditional retrieval might return all text fragments containing “Antoine” or “planet B612”, causing too much redundant information in the results. In contrast, if this text-based non-parametric memory is represented in a graph format where entities and their relationships are clearly depicted, the entity of “The Little Prince” will receive the highest score due to its direct connections with the entities “Antoine” and “planet B612” mentioned in the query, thereby ensuring the accuracy of the retrieval results. Additionally, the divide-and-conquer strategy [219, 220] offers an efficient approach for the sorting phase in retrieval of large-scale non-parametric memory. For example, Chen et al. [221] addressed the sorting of long list with relevant score by employing a recursive sorting function that can bisect the problem. They sort the entire list by recursively combining two separately sorted sub lists with a merge function.

• $\bullet$

  Retrieval Enhancement for Downstream Tasks. Non-Parametric Memory retrieval is often used to support downstream models in tasks such as question answering [7, 8, 9, 10]. If the retrieval model and the downstream model are trained independently, it leads to suboptimal performance of the overall system. Considering this aspect, optimizing the retrieval model using feedback from the downstream model can be adopted to enhance the performance of both the retrieval process and the downstream task. Within the purview of our investigation, we delineate three principal solutions. The first approach is joint training, where the retrieval model and the downstream task model are trained together. For instance, Lewis et al. [27] jointly trained the retrieval model and the downstream text generating model by minimizing negative marginal log-likelihood for each target associated with the given input. Lin et al. [218] proposed a loss function that integrates predictions from both models and a pseudo relevance score, which assesses whether the retrieved content contains the target answer, enabling joint training of the retrieval and question-answering models. The second approach involves using reinforcement learning [19, 217] to optimize the retrieval model. This method designs reward functions based on downstream performance, encouraging the retrieval model when the downstream task is well executed and discouraging it otherwise. The third approach allows the system to decide whether to invoke retrieval based on the context [215, 216, 217]. For example, the system developed by Wang et al. [216] assesses enhancement brought to problem solving by external knowledge, and triggers the retrieval component only when the retrieval of external knowledge is deemed necessary.

• $\bullet$

  Query Augmentation for Memory Retrieval. Query augmentation in information retrieval refers to techniques aimed at enhancing or modifying a user’s query to improve the quality of search results. The main objective of query augmentation is to reformulate the user’s initial query to better match the relevant documents in the database, effectively bridging the gap between the user’s query and the information they seek. The query augmentation method used in non-parametric memory retrieval is consistent with that in information retrieval. Recently, beyond traditional query augmentation methods [272, 273], LLM-based query augmentation techniques have gained attention. For example, Query2Doc [210] generates pseudo-documents with relevant information through LLMs, expanding the original query by incorporating more details. Step-Back Prompting [213] employs LLMs to identify high-level terms or topics that capture the core concepts of a query, generating queries at various levels of abstraction to enhance search outcomes. Similarly, Hypothetical Document Embeddings (HyDE) [209] convert the original query into a hypothetical document, retrieving information based on its encoded representation. Rewrite-Retrieve-Read [211] trains the rewriter model for query rewriting based on samples of rewritten queries generated by LLMs that are accurately predicted. Iterative Retrieval-Generation Synergy (ITER-RETGEN) [212] iteratively rewrites the original query by using generated or retrieved answers, thus improving query effectiveness. LLM-based techniques not limited to query augmentation also offer valuable concepts. For example, Reflectxion [188] enables LLMs to revise experiences based on environmental feedback, a mechanism that could be adapted for revising queries using similar feedback. Prompt Optimization with Textual Gradients (ProTeGi) [147] allows LLMs to optimize the original prompt by reflecting on the error between predicted and actual results, a process applicable to queries in retrieval contexts. Reinforcement learning offers another approach to query augmentation, optimizing models to enhance reward signals tied to task performance. For example, Self-Critical Sequence Training (SCST) [207] normalizes rewards during test-time inference by focusing on sequences (which can include queries) that exceed current system performance, encouraging improvements. Close Variant Generation (CLOVER) [208] utilizes diversity-driven reinforcement learning to optimize query rewriting, enhancing both the quality and diversity of query expansions.

In summary, the retrieval of non-parametric memory can be divided into sparse retrieval and dense retrieval. The former is based on the matching of textual fragments, while the latter is based on the matching of deep semantic representations. We identify three key challenges in the retrieval of non-parametric memory: achieving large-scale retrieval, enhancing retrieval for downstream tasks, and query augmentation. We discuss both existing and potential solutions to address these challenges.

The previous subsection mentioned that retrieving non-parametric memory involves two main steps: generating relevance scores and ranking to select the top-K fragments. When the retrieval model for generating relevance scores remains unchanged and the top-K value of selected fragments is constant, the probability of correctly selecting the target memory fragment decreases as the amount of information in non-parametric memory increases. This increase in information raises the likelihood of retrieval failure. According to the theory of human memory proposed by Atkinson et al. [29], the phenomenon of retrieval failure can be considered a form of forgetting, which is also a key challenge in non-parametric memory systems. On the other hand, we can draw on the mechanism of human active forgetting (discussed in Sec. 3.2.3) to actively lessen redundancy of non-parametric memory, thereby reducing storage space and improving retrieval efficiency. Implementing active forgetting of redundancy can alleviate the forgetting of critical information. We review the literature and identify several practical solutions for implementing active forgetting:

• $\bullet$

  Compression for Memory Forgetting. The goal of compression, especially the lossy compression, is to reduce the storage space or transmission bandwidth required to represent data, while ensuring that the data quality remains acceptable [274, 275, 276]. Some classic feature dimension reduction algorithms can be used to achieve data compression, such as Principal Component Analysis (PCA) [204], which is used to reduce the dimension of features and identify the most important components within a dataset. For instance, Sharifi et al. [206] used PCA to detect the less significant parts of features, effectively reducing storage redundancy by discarding unnecessary information. Another classic algorithm for data compression is t-Distributed Stochastic Neighbor Embedding (t-SNE) [205], which achieves the mapping of high-dimensional data to low-dimensional space by optimizing an objective function that measures whether similar points in high-dimensional space remain similar in low-dimensional space. In contrast, autoencoders are a newer approach to data compression [201, 203]. For example, Kingma et al. [201] proposed variational autoencoders that project high-dimensional data into a lower-dimensional latent space using an encoder, employ variational inference to approximate the posterior distribution of the data within the latent space. Improving autoencoders with vector quantization can more efficiently achieve data compression, such as the Vector-Quantized Generative Adversarial Network proposed by Esser et al. [202], which learns a discrete codebook to map continuous data latent representations to a finite number of code vectors. Data compression can also be achieved through a perception-driven approach. For example, Patel et al. [199] allocated more bits to key areas in images (such as faces and text) using a saliency mask, ensuring the visual quality of compressed images and achieving efficient image compression that aligns with human visual perception. Salehkalaibar et al. [200] explored perceptual compression techniques by comparing Perception Loss Function based on Joint Distribution (PLF-JD), which considers the joint distribution of video frames, and Perception Loss Function based on Framewise Marginal Distribution (PLF-FMD), focusing on the marginal distribution of individual frames. While PLF-JD excels at maintaining inter-frame coherence, it is more susceptible to error propagation. In contrast, PLF-FMD demonstrates superior capability in correcting errors across frames. Data compression can be applied in various applications, such as video processing, as demonstrated by Liu et al. in the literature [277] and Bidwe et al. in the literature [278], where deep learning-based video compression techniques can reduce superfluous information and enhance the storage and retrieval of video data. Additionally, using a fixed-dimensional memory matrix to store an ever-expanding data source can be regarded as a form of data compression, where this compressed memory matrix can be retrieved through attention mechanism-based computations, reducing the complexity of storage and retrieval. In this regard, Rae et al. [197] achieve the extension of fixed-dimensional memory matrix by applying linear algebra components for compression, while Munkhdalai et al. [198] update the memory matrix by overlaying the results of nonlinear key-value pair computations with retrieved memory content. These compression techniques can be used in active forgetting, where non-essential details are discarded to optimize storage and retrieval.

• $\bullet$

  Deduplication for Memory Forgetting. Data deduplication aims to reduce redundant information by identifying and eliminating duplicate data [194, 195].Kaur et al. [195] outlined three main types of data deduplication. The first type, text data deduplication, encompasses both file-level and sub-file-level methods. File-level deduplication, also known as single instance storage, removes duplicate files entirely. Sub-file-level deduplication, or block-level deduplication, further divides files into smaller segments and uses hash algorithms to identify and eliminate redundant blocks. The second type, multimedia data deduplication, focuses on removing redundant multimedia content by identifying duplicate images or key frames using visual feature similarity. The third type, cloud-based deduplication, identifies redundant data by comparing new uploads to existing content stored in the cloud. Data deduplication can be implemented through various techniques, such as hash functions, bloom filters and sparse indexing [194]. For instance, Periasamy et al. [196] achieved efficient data deduplication by computing the hash values of content and comparing these values to quickly identify and avoid storing duplicate data. Wei et al. [193] utilized a Bloom Filter Array to swiftly pre-check data fingerprints, leveraging its efficient capability to distinguish unique from potentially duplicate data, thereby reducing unnecessary accesses to the on-disk Hash Bucket Matrix that serves as the primary data store, and thus accelerating the deduplication process. Lillibridge et al. [192] presented sparse indexing, which achieves data compression by sampling small chunks within the data stream, indexing these chunks, and using the index information of these chunks to identify and merge larger data segments containing duplicate content. These deduplication techniques can be used in active forgetting, where duplicate data are discarded to optimize storage and retrieval.

In conclusion, active forgetting is effective in improving the storage and retrieval of critical information in memory systems. Active forgetting can be achieved through data compression and deduplication.

In AI models, particularly in many neural networks, the training process begins by computing the difference between ground-truth labels and model predictions, then updates the parameters using methods like gradient descent with backward propagation; during the inference process, these updated parameters are used to produce model predictions by forward propagation [284, 285, 286]. For instance, in image classification [246], neural networks update parameters using the images and their associated class labels in the training set. When the model is presented with unseen images, it uses these updated parameters during forward propagation to generate the predicted class for these images. In this process, AI models implicitly “memorize” useful information from the training data through parameter updating. Thus, the type of long-term memory that exists implicitly within AI models is referred to as “parametric memory”. Parametric memory can implicitly integrate learned information during the inference process, without requiring a mechanism explicitly designed for integrating diverse information, as is necessary in non-parametric memory driven systems [287, 288].

The storage of parametric memory poses several challenges. We discuss these challenges as follows:

• $\bullet$

  Filtering of Inappropriate Data. Parametric memory, which involves altering a model’s parameters, presents significant challenges in erasing relevant information compared to non-parametric memory. Thus, it is essential to filter out inappropriate data before constructing parametric memory. This inappropriate data may include text generated by AI models. Such text may contain more errors than the original training texts and may lead to the production of meaningless outputs or the omission of infrequently mentioned information in the dataset. Recursive use of model-generated texts as training data can precipitate model collapse [176, 177]. Another concern is poisoned data, which can drastically degrade model performance; for instance, experiments with the IMDB sentiment dataset have demonstrated that introducing just 3 percent poisoned data can escalate the test error from 12 percent to 23 percent [174, 175]. Lastly, privacy-related concerns arise when training data used for parametric memory can be inadvertently revealed through prompts resembling parts of the training data, leading to potential leaks of sensitive information [173]. Such data may be unsuitable for storage in parametric memory systems, especially in public services [170, 171, 172]. To mitigate these risks, employing keyword filtering and discriminative models may effectively prevent the inclusion of such inappropriate data [168, 169].

• $\bullet$

  Storing Long-tail and Time-sensitive Information. Language models have been shown to efficiently query and “implicitly” store factual knowledge akin to knowledge bases [249, 287, 288]. The factual knowledge stored in LLMs constitutes their parametric memory. However, it is challenging for language models to memorize long-tail information, which refers to the low-frequency data [167, 28]. This occurs in part because such information constitutes a small fraction of the pre-training dataset, making it difficult for models to store and recall it efficiently, even after extensive pre-training. Additionally, the static nature of parametric memory can lead to information lag, posing challenges in maintaining the relevance of the stored data as it may not reflect the most current information [27, 166]. To address these challenges, researchers have developed methods to mitigate the issues associated with parametric memory. Methods like improving the model capacity, increasing the frequency of exposure to examples, and enriching the content of prompt possess the potential to alleviate the difficulties of long-tail information storage and information lag [165]. However, it is suggested that storing time-sensitive and accuracy-critical information in non-parametric memory systems might be more effective [27, 28, 166].

• $\bullet$

  Increase of Storage Capacity. There is a series of papers on scaling laws for the storage efficiency of parametric memory in LLMs (as evidenced by task performance) and the computational resources required during training [163, 164]. Specifically, the information stored in parametric memory is positively correlated with the scale of model parameters $N$, the size of the training dataset $D$, and the amount of computation used during training $C$. Here $C$ required for model training, excluding the embedding process, can be approximated by $C\approx 6\text{NBS}$, where $B$ denotes the batch size, $S$ represents the number of training steps, and 6 signifies the operations involved in forward and backward passes [163]. Initially, as the training computation $C$ increases, the model’s performance improves significantly. However, after a certain threshold of $C$ is reached, the rate of performance enhancement begins to decelerate. This relationship can be mathematically formulated as follows:

  $$E=a\cdot C^{-b}+c.$$

  Here, $E$ represents the model’s error rate that reflects the storage capacity of parametric memory, $a$, $b$ and $c$ are all constants. This formula indicates a diminishing returns issue with parametric memory storage. The improvement in the parametric memory storage capacity brought by the unit computation amount $C$ becomes less and less noticeable as the training progresses. According to Zhai et al. [164], simultaneously scaling up the model parameter size $N$, training data size $D$, and computation $C$ can enhance the storage capability of parametric memory. However, this approach leads to escalating costs, making it increasingly unaffordable, particularly when computational resources are limited. On the other hand, some parameter-efficient fine-tuning [160, 161] techniques help reduce computational resource usage during incremental training while mitigating the loss of already stored parametric memory [131, 132]. For instance, LoRA [128] adds trainable rank decomposition matrices into Transformer model layers. QLoRA [162] extends LoRA by quantizing the pretrained model to 4-bit precision, further reducing computational resource usage. Furthermore, an effective strategy for enhancing long-term memory retention within the constraints of limited computational resources and model size can be realized by strategically distributing non-parametric and parametric memory types. This method involves stratifying the training data by allocating certain types of data, such as long-tail data, to non-parametric memory, while retaining the rest in parametric memory [159, 28].

In summary, parametric memory is stored in AI models’ parameters and implicitly integrates learned information. We can improve the reliability of the training set for building parametric memory by filtering out low-quality data, poisoned data, and data that are prone to privacy leakage. In comparison with non-parametric memory, parametric memory is not well-suited for storage of long-tail knowledge or time-sensitive information. Additionally, the storage capacity of parametric memory can be enhanced by simultaneously increasing the scale of model parameters, the size of the training set, and computational effort.

Parametric memory retrieval involves forward propagating queries through AI models to generate predictions, along with several associated challenges:

• $\bullet$

  Resolving Conflicts in Parametric Memory Retrieval. Recent research suggests that language models serve a function similar to that of traditional knowledge bases in retrieving information [248, 249, 287, 289]. For example, if the training data includes “Bert is a character on Sesame Street”, the model stores this information in its parametric memory. When asked “Bert is a character on which show?” the model performs retrieval from its parametric memory to answer, “Sesame Street”. However, Xu et al. [157] observed that retrieval from parametric memory can sometimes conflict with contextual information, such as non-parametric memory fragments or system inputs. For example, if the parametric memory asserts “Tomatoes are classified solely as fruits”, while either the system input or the retrieved non-parametric memory fragments contend that “Tomatoes are classified solely as vegetables”, such apparent contradictions can lead to the generation of hallucinated information. This issue is extremely difficult to manage, as it involves making judgments about the trustworthiness of various different information sources. Nonetheless, it is possible to address the issue by managing the adherence of language models to conflicting information sources. When prioritizing contextual information, context-aware decoding techniques can be employed to decrease dependence on parametric memory. This approach, detailed by Shi et al. [158], modifies the output probability distribution to accentuate differences when the model uses contextual information in generation versus when it does not. This adjustment encourages the model to rely more heavily on the provided context during inference, enhancing its accuracy and relevance. Conversely, if adherence is based on factual consistency, language models can be equipped with self-discrimination to evaluate and reconcile parametric and context information [155, 156].

• $\bullet$

  Resolving Hallucinations in Parametric Memory Retrieval. In addition to conflicts of parametric memory, hallucinations can also be caused by low-quality training data and poor model generalization abilities [151]. According to Zhang et al. [151], hallucinations refer to the phenomenon where the content retrieved and generated through parametric memory does not align with the source input provided by the user [290], previously generated content [291], or established facts [292]. Mitigating hallucinations can be achieved through a variety of strategies [151]. One strategy is ensuring the factuality in the pre-training corpus [152, 153, 154]. Another approach is constructing a reward model based on human feedback, and using reinforcement learning algorithms with a reward model that encourages the target model to acknowledge its uncertainty, especially in scenarios where correctness cannot be guaranteed [149, 150]. The previously mentioned methods for resolving conflicts can also be used to mitigate hallucinations. These mitigation methods are quite similar to the recognition stage of the Generation-Recognition Theory [29, 97, 25] described in Sec. 3.2.2, which judges the correctness of retrieved content in human memory retrieval.

• $\bullet$

  Enhancement of Query for Parametric Memory Retrieval. Tulving et al. [25, 98] highlight the importance of cues for memory retrieval in the human brain. Similarly, effective queries enhance AI model’s parametric memory retrieval, benefiting task performance. In image classification [246], the query is the image itself, so image denoising [146] and quality enhancement [145] improve the accuracy of parametric memory retrieval in this task. In content generation [231, 288, 148], the query for parametric memory retrieval is a prompt, which is a specific input or instruction given to an AI model to guide its output and serves as the starting point or the initial information that the model uses to generate the desired content. Therefore, prompt engineering [147, 148] can be adopted to refine these inputs to achieve better performance and higher quality results.

In summary, parametric memory retrieval can be achieved through forward propagation (within neural networks). We discuss three challenges in leveraging parametric memory: conflicts, hallucinations, and query enhancements. We review existing works to offer insights into the potential solutions to these challenges.

Unlike non-parametric memory, precisely managing the forgetting process of parametric memory through specific mechanisms is challenging. In multi-task learning with neural networks, updating parameters for new tasks may introduce errors into the parametric memory of previously learned tasks, potentially leading to catastrophic forgetting. Aleixo et al. [123] summarized four primary methods to mitigate catastrophic forgetting: rehearsal, distance-based methods, sub-networks, and dynamic networks. In addition to these methods, our findings indicate that curriculum learning can alleviate catastrophic forgetting. The following paragraphs explore these methods.

• $\bullet$

  Rehearsal Method. The rehearsal method, based on Aleixo et al.’s theory [123], involves obtaining representative samples of previous tasks for incremental training to mitigate catastrophic forgetting. These samples can either be sourced directly from the original data [137, 138, 139, 141, 142, 143] or generated synthetically [140, 144, 143, 112]. These samples are used in incremental training to maintain knowledge learned in the previous training process when training on new tasks. For instance, Hinton et al. [137] used fast weights to quickly deblur old memory stored in slow weights that have been blurred by the learning of new samples, based on a small subset of the old samples. Spens et al. [112] developed an autoassociative network that can encode sensory input and replay related episodic memory using a modern Hopfield network [280]. This autoassociative network assists in training the generative models through the process of replay. Roscow et al. [143] stored the interaction data comprising states, actions, time steps, and rewards in a memory buffer, and select samples from it to prevent catastrophic forgetting, either randomly or based on criteria such as temporal differences or error rates. They also discussed the use of generative models to produce data that match the parameter distribution of historical input data. This method can reduce the need for storage space when replaying training data from old tasks to the target model, as it does not require storing exact copies of training samples. The rehearsal method is similar to the mechanism of the hippocampus that alleviates forgetting through memory replay [92, 104].

• $\bullet$

  Distance-based Method. The distance-based method, based on Aleixo et al.’s theory [123], is suitable for multi-class continual learning tasks. Its objective is to minimize the distance between data points within the same class and to maximize the distance between data points of different classes, thereby maintaining the distinctiveness between classes and avoiding catastrophic forgetting of class attribution information during continual learning. For example, both Cheraghian et al. [133] and Ni et al. [136] used fixed semantic embeddings to represent class information, which ensured a degree of separability between data from different classes. Their approaches helped alleviate representation drift and mitigate catastrophic forgetting. Shi et al. [134] minimizes redundant information between features by reducing the correlation between data representations within the same class, thereby encouraging a more independent distribution of data points in the feature space. As a result, representations of the same class exhibit tighter clustering due to diminished correlation, while representations of different classes are more dispersed, occupying distinct regions of the feature space. This enhances the overall discriminability between classes. Song et al. [135] expanded the base classes using predefined transformations, such as rotations and color permutations, to create virtual classes. They provided data from these expanded classes for the model to learn class discrimination. This approach helped the newly learned classes reduce overlap with the feature space of existing classes, thereby mitigating the risk of catastrophic forgetting. The distance-based method resembles how humans construct long-term memory by building associations between pieces of information [29, 24].

• $\bullet$

  Sub-Networks Method. The sub-networks method, based on Aleixo et al.’s theory [123], mitigates catastrophic forgetting by leveraging the idea that different tasks rely on distinct sets of weight parameters within the same model. For example, task $A$’s accuracy may be highly dependent on weight parameter $a$ but less so on parameter $b$, while task $B$’s accuracy may be highly dependent on parameter $b$ but less so on parameter $a$. Therefore, during training for task $A$, we should focus on updating parameter $a$ and avoid changing parameter $b$. Similarly, during training for task $B$, we should focus on updating parameter $b$ and avoid changing parameter $a$. This approach of designating specific weight parameters for different tasks is known as the Sub-Networks Method. One method to implement sub-networks is through normalization techniques. Pham et al. [130] apply group normalization to feature channel groups prior to batch normalization, helping to balance knowledge transfer. It enhances adaptation to new tasks with significantly different data distributions while preserving the model’s performance on previously learned tasks. Another method involves Orthogonal Weights Modification. Zeng et al. [125] and Shen et al. [127] construct a matrix representing all previously trained input vectors when training on a new task. By utilizing an orthogonal projector, weight updates are confined to directions orthogonal to this matrix, thus preventing negative interference with earlier tasks. A third method employs adaptive parameters to control changes in model weights. Adel et al. [126] introduced an adaptive binary variable that determines whether the parameters of each neuron in a deep network should update during the learning of a new task. Xue et al. [129] addressed catastrophic forgetting by dynamically generating adaptive attention masks for each new task, targeting key parameters in the Vision Transformer (ViT). Some studies indicated that Low-Rank Adaptation (LoRA) alleviates catastrophic forgetting in incremental training specifically for Transformer models by integrating tunable adapter modules into certain layers, such as attention layers and feed-forward neural network (FFN) layers, while minimally affecting the original model weights [128, 131, 132]. The concept of sub-networks is analogous to the activity in specific regions of the hippocampus in the human brain, which is involved in the formation of new memory [91, 293, 94].

• $\bullet$

  Dynamic Networks Method. The dynamic networks method, discussed by Aleixo et al. [123], expands the parameters when learning new tasks to mitigate catastrophic forgetting. For instance, Roy et al. proposed Tree-CNN for handling the introduction of new classes [120]. When a new class is introduced and has a low association with existing child nodes, or the current child nodes are full, the Tree-CNN expands its parameters by adding the new class as a new child node to the original structure. Hu et al. [121] capture task-specific features by utilizing expert networks, ensuring that networks associated with earlier tasks remain unaltered. During inference, they interconnect the expert networks, enabling the exchange of feature information across tasks. Ye et al. [122] utilized the diversity of knowledge among experts to dynamically expand the expert network. Specifically, they create new experts based on the current data stream. If the minimum similarity between the new expert and existing experts, based on the output of the current data stream, is greater than or equal to a preset threshold, it indicates sufficient knowledge diversity, and the new expert is added to the ensemble. In addition to dynamically expanding experts based on new tasks, the expert network can also grow by learning task-specific routing for the experts. Yu et al. [124] proposed a method that integrates Mixture-of-Experts (MoE) adapters into a pre-trained CLIP model. These adapters function as experts within the model and can adapt to new tasks using an incremental activate-freeze strategy. In this approach, when new tasks are introduced, the model selectively activates or freezes experts based on the routers’ output distribution. This allows the model to dynamically expand and fine-tune itself to adapt to the needs of new tasks while preserving knowledge of previous tasks. The dynamic networks method mirrors the way the hippocampus in the human brain stores new memory through the formation of new neurons [93].

• $\bullet$

  Curriculum Learning Method. Curriculum Learning mimics the learning order in human education by training the model in a specific data sequence, which enhances the model’s adaptability to new tasks and improves its resistance to catastrophic forgetting [115, 117]. For example, Lisicki et al. [116] employs an adaptive staircase strategy for curriculum learning, gradually increasing problem difficulty and periodically reviewing simpler tasks during training to balance the acquisition of new knowledge with the retention of old knowledge, thereby alleviating catastrophic forgetting. Bhat et al. [118] leveraged features extracted by Convolutional Neural Networks (CNNs) [110] to measure class similarity and arranged the learning order from the most to the least similar, thereby linking new knowledge with prior knowledge to mitigate catastrophic forgetting. Saglietti et al. [119] drawing on the concept of synaptic consolidation from biology, apply L2 regularization during curriculum phase transitions to guide the current weights towards the direction of the weights learned in previous stages, thereby facilitating the retention of memory for previously learned content in new learning phases.

In summary, the tendency of parametric memory to experience catastrophic forgetting is similar to the human brain’s susceptibility to memory forgetting due to interference [103]. According to Aleixo et al. [123], the mitigation of catastrophic forgetting in parametric memory can be achieved through incremental training of previous tasks with rehearsal samples, maintaining class separability, partitioning task-specific parameter subsets, and dynamically expanding the models based on the learning needs of new tasks. Additionally, Curriculum Learning can also be utilized to alleviate catastrophic forgetting.