Difference between revisions of "AI research trends"
KevinYager (talk | contribs) (→Context Length) |
KevinYager (talk | contribs) (→Neural (non-token) Latent Representation) |
||
| (One intermediate revision by the same user not shown) | |||
| Line 50: | Line 50: | ||
* 2025-01: [https://arxiv.org/abs/2501.06252 Transformer<sup>2</sup>: Self-adaptive LLMs] | * 2025-01: [https://arxiv.org/abs/2501.06252 Transformer<sup>2</sup>: Self-adaptive LLMs] | ||
| − | =Internal | + | =Internal Thought Representation Space= |
==Visual Thinking== | ==Visual Thinking== | ||
* 2025-01: [https://arxiv.org/abs/2501.05874 VideoRAG: Retrieval-Augmented Generation over Video Corpus] | * 2025-01: [https://arxiv.org/abs/2501.05874 VideoRAG: Retrieval-Augmented Generation over Video Corpus] | ||
| Line 63: | Line 63: | ||
* 2024-12: Google DeepMind: [https://arxiv.org/abs/2412.17747 Deliberation in Latent Space via Differentiable Cache Augmentation] | * 2024-12: Google DeepMind: [https://arxiv.org/abs/2412.17747 Deliberation in Latent Space via Differentiable Cache Augmentation] | ||
* 2024-12: [https://github.com/jerber/lang-jepa LANG-JEPA: Learning to Think in Latent Space] | * 2024-12: [https://github.com/jerber/lang-jepa LANG-JEPA: Learning to Think in Latent Space] | ||
| + | * 2025-01: [https://arxiv.org/abs/2501.19201 Efficient Reasoning with Hidden Thinking] ([https://github.com/shawnricecake/Heima code]) | ||
=See Also= | =See Also= | ||
* [[Increasing AI Intelligence]] | * [[Increasing AI Intelligence]] | ||
Latest revision as of 09:38, 3 February 2025
Contents
Novel Tokenization and/or Sampling
- 2024-04: Better & Faster Large Language Models via Multi-token Prediction
- 2024-10: entropix: Entropy Based Sampling and Parallel CoT Decoding
- 2024-10: softmax is not enough (for sharp out-of-distribution)
- 2024-12: I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token
System 2 Reasoning
See: Increasing AI Intelligence
Memory
LLM Weights Memory
- 2024-12: Memory Layers at Scale
Context Length
- 2020: Various ideas for scaling context window, including Longformer
- 2023-04-02: Discussion of ideas for how to scale context window
- 2023-05-11: Anthropic announces 100k window
- 2023-06-07: magic.dev claims 5M tokens coming soon
- 2023-07-05: Microsoft describes LongNet, with 1 billion token window
- 2023-07-11: Focused Transformer 256k
- 2023-11-06: GPT-4 turbo 128k
- 2023-11-22: Anthropic Claude 2.1 200k
- 2023-12-13: Mamba alternative
- 2024-01-04: LongLM to extend context window
- 2024-02-15: Gemini 1.5 1M tokens
- 2024-03-04: Anthropic Claude 3 200k
- 2024-03-08: Google claims Gemini 1.5 can scale to 10M
- 2024-04-10: Google preprint demonstrates infinite context length by using compressive memory
- 2024-04-12: Meta et al. demonstrate Megalodon that enables infinite context via a more efficient architecture
- 2024-04-14: Google presents TransformerFAM, which leverages a feedback loop so it attends to its own latent representations, acting as working memory and provides effectively infinite context
- 2025-01Jan-14: MiniMax-01 4M (paper)
- 2025-01Jan-27: Qwen2.5-1M (report)
Extended Context
Retrieval beyond RAG
See also: AI tools: Retrieval Augmented Generation (RAG)
- 2024-12: Let your LLM generate a few tokens and you will reduce the need for retrieval
- 2024-12: RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation
Working Memory
Episodic Memory
Updating Weights at Inference-time
- 2025-01: Transformer2: Self-adaptive LLMs
Internal Thought Representation Space
Visual Thinking
- 2025-01: VideoRAG: Retrieval-Augmented Generation over Video Corpus
- 2025-01: Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
Neural (non-token) Latent Representation
- 2024-11: Microsoft: DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving: LLMs invent their own inter-communication language
- 2024-12: Meta: Training Large Language Models to Reason in a Continuous Latent Space: feeding the latent representation directly back into the model, instead of tokenizing intermediate thoughts (Chain of Continuous Thought, a.k.a. Coconut)
- 2024-12: Meta: Large Concept Models: Language Modeling in a Sentence Representation Space: train a model that operates at a higher level of abstraction than typical word/token LLMs; model operates in a space of concept embeddings (more akin to full sentences than individual words)
- 2024-12: Meta: Byte Latent Transformer: Patches Scale Better Than Tokens: Instead of tokenization, dynamically convert input byte-stream into patches, yielding gains in compute efficiency, with minimal loss in performance
- 2024-12: Compressed Chain of Thought: Efficient Reasoning Through Dense Representations
- 2024-12: Google DeepMind: Deliberation in Latent Space via Differentiable Cache Augmentation
- 2024-12: LANG-JEPA: Learning to Think in Latent Space
- 2025-01: Efficient Reasoning with Hidden Thinking (code)
