Difference between revisions of "AI tools"
KevinYager (talk | contribs) (→Approaches) |
KevinYager (talk | contribs) (→Approaches) |
||
| Line 55: | Line 55: | ||
===Approaches=== | ===Approaches=== | ||
* RAGFlow ([https://github.com/infiniflow/ragflow code]) | * RAGFlow ([https://github.com/infiniflow/ragflow code]) | ||
| − | * GraphRAG ([https://arxiv.org/abs/2404.16130 preprint], [https://github.com/microsoft/graphrag code] | + | * GraphRAG ([https://arxiv.org/abs/2404.16130 preprint], [https://github.com/microsoft/graphrag code], [https://github.com/Azure-Samples/graphrag-accelerator GraphRAG Accelerator] for easy deployment on Azure) |
| − | |||
* AutoMetaRAG ([https://github.com/darshil3011/AutoMetaRAG/tree/main code]) | * AutoMetaRAG ([https://github.com/darshil3011/AutoMetaRAG/tree/main code]) | ||
| − | * [https://verba.weaviate.io/ Verba]: RAG for [https://weaviate.io/ Weaviate] vector database | + | * [https://verba.weaviate.io/ Verba]: RAG for [https://weaviate.io/ Weaviate] vector database ([https://github.com/weaviate/verba code], [https://www.youtube.com/watch?v=UoowC-hsaf0 video]) |
| − | + | * 2024-10: Google [https://arxiv.org/abs/2410.07176 Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models] | |
| − | |||
| − | * Google | ||
| − | |||
* 2024-10: [https://arxiv.org/abs/2410.08815 StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization]: Reformats retrieved data into task-appropriate structures (table, graph, tree). | * 2024-10: [https://arxiv.org/abs/2410.08815 StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization]: Reformats retrieved data into task-appropriate structures (table, graph, tree). | ||
* 2024-10: [https://arxiv.org/abs/2410.13765 Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval] | * 2024-10: [https://arxiv.org/abs/2410.13765 Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval] | ||
Revision as of 13:56, 2 December 2024
Contents
- 1 LLM
- 2 LLM Agents
- 3 Interfaces
- 4 Speech Recognition (ASR) and Transcription
- 5 Text-to-speech (TTS)
- 6 Vision
- 7 Embedding
- 8 Time Series
- 9 Data
- 10 Hardware
- 11 See Also
LLM
Open-weights LLM
- 2023-07Jul-18: Llama2 7B, 13B, 70B
- 2024-04Apr-18: Llama3 8B, 70B
- 2024-06Jun-14: Nemotron-4 340B
- 2024-07Jul-23: Llama 3.1 8B, 70B, 405B
- 2024-07Jul-24: Mistral Large 2 128B
- 2024-07Jul-31: Gemma 2 2B
- 2024-08Aug-08: Qwen2-Math (hf, github) 1.5B, 7B, 72B
- 2024-08Aug-14: Nous research Hermes 3 (technical report) 8B, 70B, 405B
- 2024-08Aug-19: Salesforce AI xGen-MM (BLIP-3): A Family of Open Large Multimodal Models (preprint, code)
- 2024-09Sep-04: OLMoE: Open Mixture-of-Experts Language Models (code) 7B model (uses 1B per input token)
- 2024-09Sep-05: Reflection 70B (demo): Trained using Reflection-Tuning, a technique developed to enable LLMs to fix their own mistakes.
- 2024-09Sep-06: DeepSeek-V2.5 238B mixture-of-experts (160 experts, 16B active params)
- 2024-09Sep-19: Microsoft GRadient-INformed (GRIN) MoE (demo, model, github) 6.6B
- 2024-09Sep-23: Nvidia Llama-3_1-Nemotron-51B-instruct 51B
- 2024-09Sep-25: Meta Llama 3.2 with visual and voice modalities 1B, 3B, 11B, 90B
- 2024-09Sep-25: Ai2 Molmo multi-modal models 1B, 7B, 72B
- 2024-10Oct-01: Nvidia NVLM-D-72B (includes vision)
- 2024-10Oct-16: Mistral Ministral-8B-Instruct-2410
- 2024-10Oct-16: Nvidia Llama-3.1-Nemotron-70B-Reward
- 2024-11Nov-04: Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent 389B (code, weights)
- 2024-11Nov-18: Mistral-Large-Instruct-2411) 123B; and Pixtral Large multimodal model 124B (weights)
- 2024-11Nov-22: Nvidia Hymba (blog): small and high-performance
For Coding
C.f. https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard
- 2024-10Oct-06: Abacus AI Dracarys2-72B-Instruct (optimized for coding, fine-tune of Qwen2.5-72B-Instruct)
- 2024-11Nov-09: OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (weights, preprint)
- 2024-11Nov-13: Qwen2.5-Coder
Reasoning
- 2024-11Nov-20: DeepSeek-R1-Lite-Preview (results, CoT)
- 2024-11Nov-23: Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
- 2024-11Nov-27: Alibaba Qwen QwQ 32B (model, demo)
Cloud LLM
Multi-modal: Audio
- kyutai Open Science AI Lab chatbot moshi
Triage
Retrieval Augmented Generation (RAG)
Reviews
- 2024-08: Graph Retrieval-Augmented Generation: A Survey
- 2024-09: Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
- List of RAG techniques
Analysis of RAG overall
Approaches
- RAGFlow (code)
- GraphRAG (preprint, code, GraphRAG Accelerator for easy deployment on Azure)
- AutoMetaRAG (code)
- Verba: RAG for Weaviate vector database (code, video)
- 2024-10: Google Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models
- 2024-10: StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization: Reformats retrieved data into task-appropriate structures (table, graph, tree).
- 2024-10: Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval
- 2024-11: FastRAG: Retrieval Augmented Generation for Semi-structured Data
- 2024-11: Microsoft LazyGraphRAG: Setting a new standard for quality and cost
- 2024-11: Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models
Open-source Implementations
- kotaemon: An open-source clean & customizable RAG UI for chatting with your documents.
- LlamaIndex (code, docs, voice chat code)
- Nvidia ChatRTX with RAG
- Anthropic Customer Support Agent example
- LangChain and LangGraph (tutorial)
- RAGBuilder: Automatically tunes RAG hyperparams
- WikiChat
- Chonkie: No-nonsense RAG chunking library (open-source, lightweight, fast)
- autoflow: open source GraphRAG (Knowledge Graph), including conversational search page
- RAGLite
Web-based Tools
- SciSpace Chat with PDF (also available as a GPT).
Document Parsing
PDF Conversion
- Grobid
- Chunkr (code)
- Docling: converts multiple formats (PDF, DOCX, PPTX, Images, HTML) into Markdown and JSON
Automatic Optimization
Analogous to Gradient Descent
LLM for scoring/ranking
- GPTScore: Evaluate as You Desire
- Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
- Domain-specific chatbots for science using embeddings
- Large Language Models as Evaluators for Scientific Synthesis
LLM Agents
- See AI Agents.
Interfaces
Chatbot Frontend
Web
Desktop GUI
- AnythingLLM (docs, code): includes chat-with-docs, selection of LLM and vector db, etc.
Alternative Text Chatbot UI
- Loom provides a sort of tree-like structure for LLM coming up with branched writings.
- The Pantheon Interface is a new idea for how to interact with LLMs (live instance, code). In a traditional interaction, you prompt the bot and it replies in a turn-by-turn manner. Pantheon instead invites you to type out your thoughts, and various agents will asynchronously add comments or questions to spur along your brainstorming.
Conversational Audio Chatbot
- Swift is a fast AI voice assistant (code, live demo) uses:
- RTVI-AI (code, demo), uses:
- June: Local Voice Chatbot
- Ollama
- Hugging Face Transformers (for speech recognition)
- Coqui TTS Toolkit
- kyutai Moshi chatbot (demo)
- Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming (model, code, demo)
- 2024-09Sep-11: Llama-3.1-8B-Omni (code), enabling end-to-end speech.
- 2024-10Oct-18: Meta Spirit LM: open source multimodal language model that freely mixes text and speech
Related Research
Commercial Systems
Speech Recognition (ASR) and Transcription
Lists
Open Source
- DeepSpeech
- speechbrain
- Kaldi
- wav2vec 2.0
- Whisper
- Whisper medium.en
- WhisperX (includes word-level timestamps and speaker diarization)
- Distil Large v3 with MLX
- 2024-10: whisper-large-v3-turbo distillation (demo, code)
- Nvidia Canary 1B
- 2024-09: Nvidia NeMo
- 2024-10: Rev AI models for transcription and diarization
- 2024-10: Moonshine (optimized for resource-constrained devices)
In Browser
- Whisper Timestamped: Multilingual speech recognition with word-level timestamps, running locally in browser
Phrase Endpointing and Voice Activity Detection (VAD)
I.e. how to determine when user is done talking, and bot should respond?
Audio Cleanup
- Krisp AI: Noise cancellation, meeting summary, etc.
Text-to-speech (TTS)
Open Source
- Parler TTS (demo)
- Toucan (demo)
- MetaVoice (github)
- ChatTTS
- Camb.ai MARS5-TTS
- Coqui TTS Toolkit
- Fish Speech 1.4: multi-lingual, can clone voices (video, weights, demo)
- F5-TTS (demo): cloning, emotion, etc.
- MaskGCT (demo)
- Amphion: An Open-Source Audio, Music and Speech Generation Toolkit (code)
Cloud
- Elevenlabs ($50/million characters)
- Cartesia Sonic
- Neets AI ($1/million characters)
Vision
Visual Models
- CLIP
- Siglip
- Supervision
- Florence-2
- Nvidia MambaVision
- Meta Sapiens: Foundation for Human Vision Models (video input, can infer segmentation, pose, depth-map, and surface normals)
Multi-modal Models (language-vision/video)
- LLaVA-NeXT-Interleave (models, demo)
- SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models
- Nvidia NVEagle 13B, 7B (demo, preprint)
- 2024-08Aug-29: Qwen2-VL 7B, 2B (code, models): Can process videos up to 20 minutes in length
- 2024-09Sep-11: Mistral Pixtral 12B
- 2024-09Sep-17: NVLM 1.0
Optical character recognition (OCR)
Embedding
Time Series
- Stumpy: Python library, uses near-match subsequences for similarity and forecasting
- Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting
- From latent dynamics to meaningful representations
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
- A decoder-only foundation model for time-series forecasting
- TimeGPT-1
- Unified Training of Universal Time Series Forecasting Transformers
- xLSTMTime : Long-term Time Series Forecasting With xLSTM
- Salesforce: Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts (code, weights, blog)
Control
Forecasting
- Meta Kats (code): Forecasting (ARIMA, Prophet, Holt Winters, VAR), detection, feature extraction, simulation
Data
Vector Database
Open Source
- milvus (open source with paid cloud option)
- Qdrant (open source with paid cloud option)
- Vespa (open source with paid cloud option)
- chroma
- LlamaIndex
- sqlite-vec
Commercial cloud
MySQL
- MySQL does not traditionally have support, but:
- PlanetScale is working on it
- mysql_vss (discussion)
- tibd (discussion)
Database with Search
Web Scraping
Hardware
AI Acceleration Hardware
- Nvidia GPUs
- Google TPU
- Tesla Dojo
- Cerebras
- Graphcore
- Untether AI
- SambaNova Systems
- Groq
- Deep Silicon: Combined hardware/software solution for accelerated AI (e.g. ternary math)
- Etched: Transformer ASICs
