Difference between revisions of "AI research trends"

From GISAXS
Jump to: navigation, search
(→‎Missing Elements)
(→‎Continual Learning)
 
(12 intermediate revisions by the same user not shown)
Line 7: Line 7:
 
* 2024-12: [https://arxiv.org/abs/2412.09764 Memory Layers at Scale]
 
* 2024-12: [https://arxiv.org/abs/2412.09764 Memory Layers at Scale]
 
* 2025-10: [https://arxiv.org/abs/2510.15103 Continual Learning via Sparse Memory Finetuning]
 
* 2025-10: [https://arxiv.org/abs/2510.15103 Continual Learning via Sparse Memory Finetuning]
 +
* 2026-01: [https://developer.nvidia.com/blog/reimagining-llm-memory-using-context-as-training-data-unlocks-models-that-learn-at-test-time/ Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time] (Nvidia)
 +
* 2026-01: [https://arxiv.org/abs/2601.02151 Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting]
  
 
==Context Length==
 
==Context Length==
Line 36: Line 38:
 
* 2025-04-05: Meta [https://ai.meta.com/blog/llama-4-multimodal-intelligence/ Llama 4] 10M
 
* 2025-04-05: Meta [https://ai.meta.com/blog/llama-4-multimodal-intelligence/ Llama 4] 10M
 
* 2025-04-14: OpenAI [https://openai.com/index/gpt-4-1/ GPT-4.1] 1M
 
* 2025-04-14: OpenAI [https://openai.com/index/gpt-4-1/ GPT-4.1] 1M
 +
* 2025-12-04: Google [https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/ Titans/Miras] 10M
 +
* 2025-12-13: [https://arxiv.org/abs/2512.12167 Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings]
  
 
==Extended Context==
 
==Extended Context==
Line 43: Line 47:
 
* 2025-09: [https://arxiv.org/abs/2509.25140 ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory]
 
* 2025-09: [https://arxiv.org/abs/2509.25140 ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory]
 
* 2025-10: [https://arxiv.org/abs/2510.04618 Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models]
 
* 2025-10: [https://arxiv.org/abs/2510.04618 Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models]
 +
* 2025-12: [https://arxiv.org/abs/2512.24601 Recursive Language Models] (model searches/queries the full context)
 +
* 2026-01: [https://arxiv.org/abs/2601.02553 SimpleMem: Efficient Lifelong Memory for LLM Agents]
 +
* 2026-01: [https://arxiv.org/abs/2601.07190 Active Context Compression: Autonomous Memory Management in LLM Agents]
  
 
==Retrieval beyond RAG==
 
==Retrieval beyond RAG==
Line 55: Line 62:
 
==Working Memory==
 
==Working Memory==
 
* 2024-12: [https://www.arxiv.org/abs/2412.18069 Improving Factuality with Explicit Working Memory]
 
* 2024-12: [https://www.arxiv.org/abs/2412.18069 Improving Factuality with Explicit Working Memory]
 +
* 2026-01: [https://arxiv.org/abs/2601.03192 MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory]
  
 
==Long-Term Memory==
 
==Long-Term Memory==
 
* 2025-04: [https://arxiv.org/abs/2504.19413 Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory]
 
* 2025-04: [https://arxiv.org/abs/2504.19413 Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory]
 +
 +
* 2025-12: Google [https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/ Titans + Miras]
 +
** [https://arxiv.org/abs/2504.13173 It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization]
 +
** [https://arxiv.org/abs/2501.00663 Titans: Learning to Memorize at Test Time]
  
 
===Storage and Retrieval===
 
===Storage and Retrieval===
 
* 2025-09: [https://arxiv.org/abs/2509.04439 ArcMemo: Abstract Reasoning Composition with Lifelong LLM Memory]
 
* 2025-09: [https://arxiv.org/abs/2509.04439 ArcMemo: Abstract Reasoning Composition with Lifelong LLM Memory]
 +
* 2026-01: [https://www.arxiv.org/abs/2601.07372 Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models]
  
 
===Episodic Memory===
 
===Episodic Memory===
Line 67: Line 80:
  
 
==Continual Learning==
 
==Continual Learning==
 +
* 2022-02: [https://arxiv.org/abs/2202.00275 Architecture Matters in Continual Learning]
 
* 2025-10: [https://arxiv.org/abs/2510.15103 Continual Learning via Sparse Memory Finetuning]
 
* 2025-10: [https://arxiv.org/abs/2510.15103 Continual Learning via Sparse Memory Finetuning]
 
* 2025-11: [https://research.google/blog/introducing-nested-learning-a-new-ml-paradigm-for-continual-learning/ Introducing Nested Learning: A new ML paradigm for continual learning]
 
* 2025-11: [https://research.google/blog/introducing-nested-learning-a-new-ml-paradigm-for-continual-learning/ Introducing Nested Learning: A new ML paradigm for continual learning]
Line 164: Line 178:
 
** 2025-06: [https://x.com/karpathy/status/1937902205765607626 "Context Engineering" instead of "Prompt Engineering"]
 
** 2025-06: [https://x.com/karpathy/status/1937902205765607626 "Context Engineering" instead of "Prompt Engineering"]
 
** 2025-06: [https://x.com/karpathy/status/1938626382248149433 LLMs as "cognitive cores"]
 
** 2025-06: [https://x.com/karpathy/status/1938626382248149433 LLMs as "cognitive cores"]
 +
** 2025-11: [https://x.com/karpathy/status/1990116666194456651?s=20 Software 1.0 easily automates what you can specify. Software 2.0 easily automates what you can verify.]
  
 
=See Also=
 
=See Also=
 
* [[Increasing AI Intelligence]]
 
* [[Increasing AI Intelligence]]

Latest revision as of 10:40, 17 January 2026

System 2 Reasoning

See: Increasing AI Intelligence

Memory

LLM Weights Memory

Context Length

Extended Context

Context Remaking

Retrieval beyond RAG

See also: AI tools: Retrieval Augmented Generation (RAG)

Working Memory

Long-Term Memory

Storage and Retrieval

Episodic Memory

Continual Learning

Updating Weights at Inference-time

Parameters as Tokens

Internal Thought Representation Space

Visual Thinking

Neural (non-token) Latent Representation

Altered Transformer

Tokenization

Generation Order

Diffusion Language Models

Related: Image Synthesis via Autoregression/Diffusion

Sampling

Daydreaming, brainstorming, pre-generation

Pre-generation


Missing Elements

  • Memory
  • Continuous learning/update
  • Robust contextual model
  • Long-time-horizon coherence
  • Fluid intelligence
  • Agency
  • Modeling of self
  • Daydreaming

Memes

See Also

Retrieved from "http://gisaxs.com/index.php?title=AI_research_trends&oldid=8557"