Difference between revisions of "AI research trends"
KevinYager (talk | contribs) (→Context Length) |
KevinYager (talk | contribs) (→Diffusion Language Models) |
||
| (10 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
=System 2 Reasoning= | =System 2 Reasoning= | ||
See: [[Increasing AI Intelligence]] | See: [[Increasing AI Intelligence]] | ||
| Line 38: | Line 32: | ||
* 2025-02-19: [https://github.com/MoonshotAI/MoBA MoBA: Mixture of Block Attention for Long-Context LLMs] | * 2025-02-19: [https://github.com/MoonshotAI/MoBA MoBA: Mixture of Block Attention for Long-Context LLMs] | ||
* 2025-02-27: [https://arxiv.org/abs/2502.20082 LongRoPE2: Near-Lossless LLM Context Window Scaling] ([https://github.com/microsoft/LongRoPE code]) | * 2025-02-27: [https://arxiv.org/abs/2502.20082 LongRoPE2: Near-Lossless LLM Context Window Scaling] ([https://github.com/microsoft/LongRoPE code]) | ||
| − | * [https://x.com/sundarpichai/status/1904579419496386736 2025-03-25]: Gemini 2.5 Pro [https://x.com/pvncher/status/1904685092053606715 1M] | + | * [https://x.com/sundarpichai/status/1904579419496386736 2025-03-25]: [https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/ Gemini 2.5 Pro] [https://x.com/pvncher/status/1904685092053606715 1M] |
| + | * 2025-04-05: Meta [https://ai.meta.com/blog/llama-4-multimodal-intelligence/ Llama 4] 10M | ||
==Extended Context== | ==Extended Context== | ||
| Line 79: | Line 74: | ||
* 2025-02: Meta: [https://arxiv.org/abs/2502.08524 LLM Pretraining with Continuous Concepts] (CoCoMix) | * 2025-02: Meta: [https://arxiv.org/abs/2502.08524 LLM Pretraining with Continuous Concepts] (CoCoMix) | ||
| − | =Diffusion Language Models= | + | =Altered Transformer= |
| + | |||
| + | ==Tokenization== | ||
| + | * 2024-04: [https://arxiv.org/abs/2404.19737 Better & Faster Large Language Models via Multi-token Prediction] | ||
| + | * 2024-12: [https://arxiv.org/abs/2412.06676 I Don't Know: Explicit Modeling of Uncertainty with an <nowiki>[IDK]</nowiki> Token] | ||
| + | * 2025-04: Meta: [https://arxiv.org/abs/2504.00927 Multi-Token Attention] | ||
| + | |||
| + | ==Generation Order== | ||
| + | * 2019-02: [https://arxiv.org/abs/1902.02192 Non-Monotonic Sequential Text Generation] | ||
| + | * 2019-04: [https://arxiv.org/abs/1904.09324 Mask-Predict: Parallel Decoding of Conditional Masked Language Models] | ||
| + | * 2019-06: [https://arxiv.org/abs/1906.09601 Sequence Generation: From Both Sides to the Middle] | ||
| + | * 2020-04: [https://arxiv.org/abs/2004.11579 Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order] | ||
| + | * 2021-12: [https://arxiv.org/abs/2112.10543 Spiral Language Modeling] | ||
| + | * 2023-10: [https://arxiv.org/abs/2310.09930 FiLM: Fill-in Language Models for Any-Order Generation] | ||
| + | * 2024-07: [https://arxiv.org/abs/2407.03582 Integrating Randomness in Large Language Models: A Linear Congruential Generator Approach for Generating Clinically Relevant Content] | ||
| + | |||
| + | ==Diffusion Language Models== | ||
* 2024-02: [https://arxiv.org/abs/2402.03687 Pard: Permutation-Invariant Autoregressive Diffusion for Graph Generation] | * 2024-02: [https://arxiv.org/abs/2402.03687 Pard: Permutation-Invariant Autoregressive Diffusion for Graph Generation] | ||
* 2025-02: [https://arxiv.org/abs/2502.09992 Large Language Diffusion Models] | * 2025-02: [https://arxiv.org/abs/2502.09992 Large Language Diffusion Models] | ||
* 2025-02: [https://www.inceptionlabs.ai/ Inception Labs] [https://www.inceptionlabs.ai/news Mercury] model ([https://chat.inceptionlabs.ai/ online demo]) | * 2025-02: [https://www.inceptionlabs.ai/ Inception Labs] [https://www.inceptionlabs.ai/news Mercury] model ([https://chat.inceptionlabs.ai/ online demo]) | ||
* 2025-03: [https://arxiv.org/abs/2503.09573 Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models] ([https://m-arriola.com/bd3lms/ project], [https://github.com/kuleshov-group/bd3lms code], [https://huggingface.co/collections/kuleshov-group/bd3-lms-67be95f81b96b15fec50d53f hf]) | * 2025-03: [https://arxiv.org/abs/2503.09573 Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models] ([https://m-arriola.com/bd3lms/ project], [https://github.com/kuleshov-group/bd3lms code], [https://huggingface.co/collections/kuleshov-group/bd3-lms-67be95f81b96b15fec50d53f hf]) | ||
| + | * 2025-04: [https://hkunlp.github.io/blog/2025/dream/ Dream 7B: Introducing Dream 7B, the most powerful open diffusion large language model to date] | ||
| + | * 2025-04: [https://dllm-reasoning.github.io/d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning] ([https://dllm-reasoning.github.io/media/preprint.pdf preprint], [https://github.com/dllm-reasoning/d1 code]) | ||
| + | |||
| + | ===Related: Image Synthesis via Autoregression/Diffusion=== | ||
| + | * 2023-10: [https://arxiv.org/abs/2310.01400 Sequential Data Generation with Groupwise Diffusion Process] | ||
| + | * 2024-02: [https://arxiv.org/abs/2402.09470 Rolling Diffusion Models] | ||
| + | * 2024-08: [https://arxiv.org/abs/2408.11039 Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model] | ||
| + | |||
| + | ==Sampling== | ||
| + | * 2024-10: [https://github.com/xjdr-alt/entropix entropix: Entropy Based Sampling and Parallel CoT Decoding] | ||
| + | * 2024-10: [https://arxiv.org/abs/2410.01104 softmax is not enough (for sharp out-of-distribution)] | ||
| + | |||
| + | =Missing Elements= | ||
| + | * Memory | ||
| + | * Continuous learning/update | ||
| + | * Robust contextual model | ||
| + | * Long-time-horizon coherence | ||
| + | * Fluid intelligence | ||
| + | * Agency | ||
=See Also= | =See Also= | ||
* [[Increasing AI Intelligence]] | * [[Increasing AI Intelligence]] | ||
Revision as of 12:02, 13 April 2025
Contents
System 2 Reasoning
See: Increasing AI Intelligence
Memory
LLM Weights Memory
- 2024-12: Memory Layers at Scale
Context Length
- 2020: Various ideas for scaling context window, including Longformer
- 2023-04-02: Discussion of ideas for how to scale context window
- 2023-05-11: Anthropic announces 100k window
- 2023-06-07: magic.dev claims 5M tokens coming soon
- 2023-07-05: Microsoft describes LongNet, with 1 billion token window
- 2023-07-11: Focused Transformer 256k
- 2023-11-06: GPT-4 turbo 128k
- 2023-11-22: Anthropic Claude 2.1 200k
- 2023-12-13: Mamba alternative
- 2024-01-04: LongLM to extend context window
- 2024-02-15: Gemini 1.5 1M tokens
- 2024-03-04: Anthropic Claude 3 200k
- 2024-03-08: Google claims Gemini 1.5 can scale to 10M
- 2024-04-10: Google preprint demonstrates infinite context length by using compressive memory
- 2024-04-12: Meta et al. demonstrate Megalodon that enables infinite context via a more efficient architecture
- 2024-04-14: Google presents TransformerFAM, which leverages a feedback loop so it attends to its own latent representations, acting as working memory and provides effectively infinite context
- 2024-10-31: What is Wrong with Perplexity for Long-context Language Modeling?
- 2025-01-14: MiniMax-01 4M (paper)
- 2025-01-27: Qwen2.5-1M (report)
- 2025-02-14: InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU 3M
- 2025-02-18: HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading (code)
- 2025-02-18: Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
- 2025-02-19: MoBA: Mixture of Block Attention for Long-Context LLMs
- 2025-02-27: LongRoPE2: Near-Lossless LLM Context Window Scaling (code)
- 2025-03-25: Gemini 2.5 Pro 1M
- 2025-04-05: Meta Llama 4 10M
Extended Context
Retrieval beyond RAG
See also: AI tools: Retrieval Augmented Generation (RAG)
- 2024-10: Microsoft: KBLaM: Knowledge Base augmented Language Model
- 2024-12: Let your LLM generate a few tokens and you will reduce the need for retrieval
- 2024-12: RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation
- 2025-03: Microsoft: Introducing KBLaM: Bringing plug-and-play external knowledge to LLMs
Working Memory
Episodic Memory
Updating Weights at Inference-time
- 2025-01: Transformer2: Self-adaptive LLMs
Parameters as Tokens
Internal Thought Representation Space
Visual Thinking
- 2025-01: VideoRAG: Retrieval-Augmented Generation over Video Corpus
- 2025-01: Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
Neural (non-token) Latent Representation
- 2024-11: Microsoft: DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving: LLMs invent their own inter-communication language
- 2024-12: Meta: Training Large Language Models to Reason in a Continuous Latent Space: feeding the latent representation directly back into the model, instead of tokenizing intermediate thoughts (Chain of Continuous Thought, a.k.a. Coconut)
- 2024-12: Meta: Large Concept Models: Language Modeling in a Sentence Representation Space: train a model that operates at a higher level of abstraction than typical word/token LLMs; model operates in a space of concept embeddings (more akin to full sentences than individual words)
- 2024-12: Meta: Byte Latent Transformer: Patches Scale Better Than Tokens: Instead of tokenization, dynamically convert input byte-stream into patches, yielding gains in compute efficiency, with minimal loss in performance
- 2024-12: Compressed Chain of Thought: Efficient Reasoning Through Dense Representations
- 2024-12: Google DeepMind: Deliberation in Latent Space via Differentiable Cache Augmentation
- 2024-12: LANG-JEPA: Learning to Think in Latent Space
- 2025-01: Efficient Reasoning with Hidden Thinking (code)
- 2025-02: Huginn-0125: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (code, model)
- 2025-02: Meta: LLM Pretraining with Continuous Concepts (CoCoMix)
Altered Transformer
Tokenization
- 2024-04: Better & Faster Large Language Models via Multi-token Prediction
- 2024-12: I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token
- 2025-04: Meta: Multi-Token Attention
Generation Order
- 2019-02: Non-Monotonic Sequential Text Generation
- 2019-04: Mask-Predict: Parallel Decoding of Conditional Masked Language Models
- 2019-06: Sequence Generation: From Both Sides to the Middle
- 2020-04: Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
- 2021-12: Spiral Language Modeling
- 2023-10: FiLM: Fill-in Language Models for Any-Order Generation
- 2024-07: Integrating Randomness in Large Language Models: A Linear Congruential Generator Approach for Generating Clinically Relevant Content
Diffusion Language Models
- 2024-02: Pard: Permutation-Invariant Autoregressive Diffusion for Graph Generation
- 2025-02: Large Language Diffusion Models
- 2025-02: Inception Labs Mercury model (online demo)
- 2025-03: Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (project, code, hf)
- 2025-04: Dream 7B: Introducing Dream 7B, the most powerful open diffusion large language model to date
- 2025-04: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (preprint, code)
Related: Image Synthesis via Autoregression/Diffusion
- 2023-10: Sequential Data Generation with Groupwise Diffusion Process
- 2024-02: Rolling Diffusion Models
- 2024-08: Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
Sampling
- 2024-10: entropix: Entropy Based Sampling and Parallel CoT Decoding
- 2024-10: softmax is not enough (for sharp out-of-distribution)
Missing Elements
- Memory
- Continuous learning/update
- Robust contextual model
- Long-time-horizon coherence
- Fluid intelligence
- Agency
