AI tools

LLM

Open-weights LLM

• 2023-07Jul-18: Llama2 7B, 13B, 70B
• 2024-04Apr-18: Llama3 8B, 70B
• 2024-06Jun-14: Nemotron-4 340B
• 2024-07Jul-23: Llama 3.1 8B, 70B, 405B
• 2024-07Jul-24: Mistral Large 2 128B
• 2024-07Jul-31: Gemma 2 2B
• 2024-08Aug-08: Qwen2-Math (hf, github) 1.5B, 7B, 72B
• 2024-08Aug-14: Nous research Hermes 3 (technical report) 8B, 70B, 405B
• 2024-08Aug-19: Salesforce AI xGen-MM (BLIP-3): A Family of Open Large Multimodal Models (preprint, code)
• 2024-09Sep-04: OLMoE: Open Mixture-of-Experts Language Models (code) 7B model (uses 1B per input token)
• 2024-09Sep-05: Reflection 70B (demo): Trained using Reflection-Tuning, a technique developed to enable LLMs to fix their own mistakes.
• 2024-09Sep-06: DeepSeek-V2.5 238B mixture-of-experts (160 experts, 16B active params)
• 2024-09Sep-19: Microsoft GRadient-INformed (GRIN) MoE (demo, model, github) 6.6B
• 2024-09Sep-23: Nvidia Llama-3_1-Nemotron-51B-instruct 51B
• 2024-09Sep-25: Meta Llama 3.2 with visual and voice modalities 1B, 3B, 11B, 90B
• 2024-09Sep-25: Ai2 Molmo multi-modal models 1B, 7B, 72B
• 2024-10Oct-01: Nvidia NVLM-D-72B (includes vision)
• 2024-10Oct-16: Mistral Ministral-8B-Instruct-2410
• 2024-10Oct-16: Nvidia Llama-3.1-Nemotron-70B-Reward
• 2024-11Nov-04: Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent 389B (code, weights)
• 2024-11Nov-18: Mistral-Large-Instruct-2411) 123B; and Pixtral Large multimodal model 124B (weights)
• 2024-11Nov-22: Nvidia Hymba (blog): small and high-performance
• 2024-12Dec-06: Meta Llama 3.3 70B

For Coding

C.f. https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard

• 2024-10Oct-06: Abacus AI Dracarys2-72B-Instruct (optimized for coding, fine-tune of Qwen2.5-72B-Instruct)
• 2024-11Nov-09: OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (weights, preprint)
• 2024-11Nov-13: Qwen2.5-Coder

Reasoning

• 2024-11Nov-20: DeepSeek-R1-Lite-Preview (results, CoT)
• 2024-11Nov-23: Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
• 2024-11Nov-27: Alibaba Qwen QwQ 32B (model, demo)
• 2024-12Dec-04: Ruliad Deepthought 8B (demo)

Cloud LLM

• Groq cloud (very fast inference)

Multi-modal: Audio

• kyutai Open Science AI Lab chatbot moshi

Triage

• RouteLLM: Learning to Route LLMs with Preference Data

Retrieval Augmented Generation (RAG)

Reviews

• 2024-08: Graph Retrieval-Augmented Generation: A Survey
• 2024-09: Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
• List of RAG techniques
• Advanced RAG Cookbooks👨🏻‍💻

Analysis of RAG overall

• 2024-10: Is Semantic Chunking Worth the Computational Cost?

Approaches

• RAGFlow (code)
• GraphRAG (preprint, code, GraphRAG Accelerator for easy deployment on Azure)
• AutoMetaRAG (code)
• Verba: RAG for Weaviate vector database (code, video)
• 2024-10: Google Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models
• 2024-10: StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization: Reformats retrieved data into task-appropriate structures (table, graph, tree).
• 2024-10: Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval
• 2024-11: FastRAG: Retrieval Augmented Generation for Semi-structured Data
• 2024-11: Microsoft LazyGraphRAG: Setting a new standard for quality and cost
• 2024-11: Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models

Open-source Implementations

• kotaemon: An open-source clean & customizable RAG UI for chatting with your documents.
• LlamaIndex (code, docs, voice chat code)
• Nvidia ChatRTX with RAG
• Anthropic Customer Support Agent example
• LangChain and LangGraph (tutorial)
  • RAGBuilder: Automatically tunes RAG hyperparams
• WikiChat
  • WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
• Chonkie: No-nonsense RAG chunking library (open-source, lightweight, fast)
• autoflow: open source GraphRAG (Knowledge Graph), including conversational search page
• RAGLite
• nano-graphrag: A simple, easy-to-hack GraphRAG implementation

Web-based Tools

• SciSpace Chat with PDF (also available as a GPT).

Document Parsing

• Docling: converts multiple formats (PDF, DOCX, PPTX, Images, HTML) into Markdown and JSON
• Microsoft Markitdown: converts various formats (PDF, Word, Excel, PPT) to Markdown (available via web interface on replit)

PDF Conversion

• Grobid
• Chunkr (code)

Automatic Optimization

Analogous to Gradient Descent

• TextGrad: Automatic "Differentiation" via Text
• Symbolic Learning Enables Self-Evolving Agents

LLM for scoring/ranking

• GPTScore: Evaluate as You Desire
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
• Domain-specific chatbots for science using embeddings
• Large Language Models as Evaluators for Scientific Synthesis

LLM Agents

• See AI Agents.

Interfaces

Chatbot Frontend

Web (code)

• Steamlit
• Cohere Toolkit (code)
• LibreChat
• open-webui
• entropix frontend UI

Web (product)

• Chatbox

Desktop GUI

• AnythingLLM (docs, code): includes chat-with-docs, selection of LLM and vector db, etc.

Alternative Text Chatbot UI

• Loom provides a sort of tree-like structure for LLM coming up with branched writings.
• The Pantheon Interface is a new idea for how to interact with LLMs (live instance, code). In a traditional interaction, you prompt the bot and it replies in a turn-by-turn manner. Pantheon instead invites you to type out your thoughts, and various agents will asynchronously add comments or questions to spur along your brainstorming.

Conversational Audio Chatbot

• Swift is a fast AI voice assistant (code, live demo) uses:
  • Groq cloud running OpenAI Whisper for fast speech transcription.
  • Cartesia Sonic for fast speech synthesis
  • VAD to detect when user is talking
  • Vercel for app deployment
• RTVI-AI (code, demo), uses:
  • Groq
  • Llama 3.1
  • Daily
  • RTVI
• June: Local Voice Chatbot
  • Ollama
  • Hugging Face Transformers (for speech recognition)
  • Coqui TTS Toolkit
• kyutai Moshi chatbot (demo)
• Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming (model, code, demo)
• 2024-09Sep-11: Llama-3.1-8B-Omni (code), enabling end-to-end speech.
• 2024-10Oct-18: Meta Spirit LM: open source multimodal language model that freely mixes text and speech

Related Research

• Language Model Can Listen While Speaking

Commercial Systems

• HeyPi Talk
• Vapi
• Call Annie
• Bland AI
• DeepGram Voice AI

Speech Recognition (ASR) and Transcription

Lists

• Open ASR Leaderboard

Open Source

• DeepSpeech
• speechbrain
• Kaldi
• wav2vec 2.0
  • Paper: Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
• Whisper
  • Whisper medium.en
  • WhisperX (includes word-level timestamps and speaker diarization)
  • Distil Large v3 with MLX
  • 2024-10: whisper-large-v3-turbo distillation (demo, code)
• Nvidia Canary 1B
• 2024-09: Nvidia NeMo
• 2024-10: Rev AI models for transcription and diarization
• 2024-10: Moonshine (optimized for resource-constrained devices)

In Browser

• Whisper Timestamped: Multilingual speech recognition with word-level timestamps, running locally in browser

Phrase Endpointing and Voice Activity Detection (VAD)

I.e. how to determine when user is done talking, and bot should respond?

• Notes
  • Test settings
  • code
  • Silero VAD repo

Audio Cleanup

• Krisp AI: Noise cancellation, meeting summary, etc.

Text-to-speech (TTS)

Open Source

• Parler TTS (demo)
• Toucan (demo)
• MetaVoice (github)
• ChatTTS
• Camb.ai MARS5-TTS
• Coqui TTS Toolkit
• Fish Speech 1.4: multi-lingual, can clone voices (video, weights, demo)
• F5-TTS (demo): cloning, emotion, etc.
• MaskGCT (demo)
• Amphion: An Open-Source Audio, Music and Speech Generation Toolkit (code)

Cloud

• Elevenlabs ($50/million characters)
  • voice isolator
• Cartesia Sonic
• Neets AI ($1/million characters)

Vision

Visual Models

• CLIP
• Siglip
• Supervision
• Florence-2
• Nvidia MambaVision
• Meta Sapiens: Foundation for Human Vision Models (video input, can infer segmentation, pose, depth-map, and surface normals)

Multi-modal Models (language-vision/video)

• LLaVA-NeXT-Interleave (models, demo)
• SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models
• Nvidia NVEagle 13B, 7B (demo, preprint)
• 2024-08Aug-29: Qwen2-VL 7B, 2B (code, models): Can process videos up to 20 minutes in length
• 2024-09Sep-11: Mistral Pixtral 12B
• 2024-09Sep-17: NVLM 1.0
• 2024-12Dec-06: Nvidia NVILA: Efficient Frontier Visual Language Models

Optical character recognition (OCR)

• General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model (project, code, demo)
• Swift OCR: LLM Powered Fast OCR

Embedding

• A Comparison of Top Embedding Libraries for Generative AI

Time Series

• Stumpy: Python library, uses near-match subsequences for similarity and forecasting
• Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting
• From latent dynamics to meaningful representations
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
• A decoder-only foundation model for time-series forecasting
• TimeGPT-1
• Unified Training of Universal Time Series Forecasting Transformers
• xLSTMTime : Long-term Time Series Forecasting With xLSTM
• Salesforce: Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts (code, weights, blog)

Control

• PIDformer: Transformer Meets Control Theory

Forecasting

• Meta Kats (code): Forecasting (ARIMA, Prophet, Holt Winters, VAR), detection, feature extraction, simulation
• Context is Key: A Benchmark for Forecasting with Essential Textual Information

Data

Vector Database

Open Source

• milvus (open source with paid cloud option)
• Qdrant (open source with paid cloud option)
• Vespa (open source with paid cloud option)
• chroma
• LlamaIndex
• sqlite-vec

Commercial cloud

• pinecone
• weaviate

MySQL

• MySQL does not traditionally have support, but:
  • PlanetScale is working on it
  • mysql_vss (discussion)
  • tibd (discussion)

Database with Search

• Typesense (code)

Web Scraping

• Firecrawl
• Crawl4AI: Crawl Smarter, Faster, Freely. For AI.
• GitIngest: Turn any GitHub repository into a prompt-friendly text file, for inclusion in LLM's context. Available at: gitingest.com
• ScrapeGraphAI: You Only Scrape Once: web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.)

Hardware

AI Acceleration Hardware

• Nvidia GPUs
• Google TPU
• Tesla Dojo
• Cerebras
• Graphcore
• Untether AI
• SambaNova Systems
• Groq
• Deep Silicon: Combined hardware/software solution for accelerated AI (e.g. ternary math)
• Etched: Transformer ASICs

Cloud Training Compute

• Nebius AI
• Glaive AI

See Also

• AI agents
• AI understanding
• Robots