AI research trends

Novel Tokenization and/or Sampling

• 2024-10: entropix: Entropy Based Sampling and Parallel CoT Decoding
• 2024-12: I Don't Know: Explicit Modeling of Uncertainty with an [IDK Token]

System 2 Reasoning

See: Increasing AI Agent Intelligence

Memory

Context Length

TBD

Working Memory

• 2024-12: Improving Factuality with Explicit Working Memory

Episodic Memory

• 2024-03: Larimar: Large Language Models with Episodic Memory Control

Neural (non-token) Latent Representation

• 2024-11: Microsoft: DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving: LLMs invent their own inter-communication language
• 2024-12: Meta: Training Large Language Models to Reason in a Continuous Latent Space: feeding the latent representation directly back into the model, instead of tokenizing intermediate thoughts (Chain of Continuous Thought, a.k.a. Coconut)
• 2024-12: Meta: Large Concept Models: Language Modeling in a Sentence Representation Space: train a model that operates at a higher level of abstraction than typical word/token LLMs; model operates in a space of concept embeddings (more akin to full sentences than individual words)
• 2024-12: Meta: Byte Latent Transformer: Patches Scale Better Than Tokens: Instead of tokenization, dynamically convert input byte-stream into patches, yielding gains in compute efficiency, with minimal loss in performance
• 2024-12: Compressed Chain of Thought: Efficient Reasoning Through Dense Representations
• 2024-12: Google DeepMind: Deliberation in Latent Space via Differentiable Cache Augmentation
• 2024-12: LANG-JEPA: Learning to Think in Latent Space