Difference between revisions of "AI tools"
KevinYager (talk | contribs) (→Open-weights LLM) |
KevinYager (talk | contribs) (→Time Series) |
||
| Line 321: | Line 321: | ||
* Salesforce: [https://arxiv.org/abs/2410.10469 Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts] ([https://github.com/SalesforceAIResearch/uni2ts/tree/main/project/moirai-moe-1 code], [https://huggingface.co/collections/Salesforce/moirai-r-models-65c8d3a94c51428c300e0742 weights], [https://www.salesforce.com/blog/time-series-morai-moe/ blog]) | * Salesforce: [https://arxiv.org/abs/2410.10469 Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts] ([https://github.com/SalesforceAIResearch/uni2ts/tree/main/project/moirai-moe-1 code], [https://huggingface.co/collections/Salesforce/moirai-r-models-65c8d3a94c51428c300e0742 weights], [https://www.salesforce.com/blog/time-series-morai-moe/ blog]) | ||
* IBM [https://huggingface.co/docs/transformers/en/model_doc/patchtsmixer PatchTSMixer] and [https://huggingface.co/docs/transformers/en/model_doc/patchtst PatchTST] (being [https://research.ibm.com/blog/time-series-AI-transformers used] for particle accelerators) | * IBM [https://huggingface.co/docs/transformers/en/model_doc/patchtsmixer PatchTSMixer] and [https://huggingface.co/docs/transformers/en/model_doc/patchtst PatchTST] (being [https://research.ibm.com/blog/time-series-AI-transformers used] for particle accelerators) | ||
| − | |||
==Control== | ==Control== | ||
| Line 329: | Line 328: | ||
* Meta [https://facebookresearch.github.io/Kats/ Kats] ([https://github.com/facebookresearch/Kats code]): Forecasting (ARIMA, Prophet, Holt Winters, VAR), detection, feature extraction, simulation | * Meta [https://facebookresearch.github.io/Kats/ Kats] ([https://github.com/facebookresearch/Kats code]): Forecasting (ARIMA, Prophet, Holt Winters, VAR), detection, feature extraction, simulation | ||
* [https://arxiv.org/abs/2410.18959 Context is Key: A Benchmark for Forecasting with Essential Textual Information] | * [https://arxiv.org/abs/2410.18959 Context is Key: A Benchmark for Forecasting with Essential Textual Information] | ||
| + | |||
| + | ==Anomaly Detection== | ||
| + | * 2024-10: [https://arxiv.org/abs/2410.05440 Can LLMs Understand Time Series Anomalies?] ([https://github.com/rose-stl-lab/anomllm code]) | ||
=Data= | =Data= | ||
Revision as of 09:35, 18 March 2025
Contents
LLM
Open-weights LLM
- 2023-07Jul-18: Llama2 7B, 13B, 70B
- 2024-04Apr-18: Llama3 8B, 70B
- 2024-06Jun-14: Nemotron-4 340B
- 2024-07Jul-23: Llama 3.1 8B, 70B, 405B
- 2024-07Jul-24: Mistral Large 2 128B
- 2024-07Jul-31: Gemma 2 2B
- 2024-08Aug-08: Qwen2-Math (hf, github) 1.5B, 7B, 72B
- 2024-08Aug-14: Nous research Hermes 3 (technical report) 8B, 70B, 405B
- 2024-08Aug-19: Salesforce AI xGen-MM (BLIP-3): A Family of Open Large Multimodal Models (preprint, code)
- 2024-09Sep-04: OLMoE: Open Mixture-of-Experts Language Models (code) 7B model (uses 1B per input token)
- 2024-09Sep-05: Reflection 70B (demo): Trained using Reflection-Tuning, a technique developed to enable LLMs to fix their own mistakes.
- 2024-09Sep-06: DeepSeek-V2.5 238B mixture-of-experts (160 experts, 16B active params)
- 2024-09Sep-19: Microsoft GRadient-INformed (GRIN) MoE (demo, model, github) 6.6B
- 2024-09Sep-23: Nvidia Llama-3_1-Nemotron-51B-instruct 51B
- 2024-09Sep-25: Meta Llama 3.2 with visual and voice modalities 1B, 3B, 11B, 90B
- 2024-09Sep-25: Ai2 Molmo multi-modal models 1B, 7B, 72B
- 2024-10Oct-01: Nvidia NVLM-D-72B (includes vision)
- 2024-10Oct-16: Mistral Ministral-8B-Instruct-2410
- 2024-10Oct-16: Nvidia Llama-3.1-Nemotron-70B-Reward
- 2024-11Nov-04: Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent 389B (code, weights)
- 2024-11Nov-18: Mistral-Large-Instruct-2411) 123B; and Pixtral Large multimodal model 124B (weights)
- 2024-11Nov-22: Nvidia Hymba (blog): small and high-performance
- 2024-12Dec-06: Meta Llama 3.3 70B
- 2024-12Dec-26: DeepSeek-V3-Base 671B
- 2025-01Jan-02: SmallThinker-3B-Preview (fine-tune of Qwen2.5-3b-Instruct)
- 2025-01Jan-08: Microsoft phi-4 15B
- 2025-01Jan-14: MiniMax-01, MiniMax-Text-01 and MiniMax-VL-01; 4M context length (paper)
- 2025-01Jan-27: Qwen2.5-1M (report)
- 2025-01Jan-27: DeepSeek Janus-Pro-7B (with image capabilities)
- 2025-03Mar-14: Cohere Command A (weights)
- 2025-03Mar-17: Mistral Small 3.1 24B (weights)
For Coding
Rankings: bigcode-models-leaderboard and CodeElo leaderboard
- 2024-10Oct-06: Abacus AI Dracarys2-72B-Instruct (optimized for coding, fine-tune of Qwen2.5-72B-Instruct)
- 2024-11Nov-09: OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (weights, preprint)
- 2024-11Nov-13: Qwen2.5-Coder
Reasoning
See also: Increasing AI Intelligence > Proactive Search > CoT reasoning model
- 2024-11Nov-20: DeepSeek-R1-Lite-Preview (results, CoT)
- 2024-11Nov-23: Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
- 2024-11Nov-27: Alibaba Qwen QwQ 32B (model, demo)
- 2024-12Dec-04: Ruliad Deepthought 8B (demo)
- 2024-12Dec-24: Qwen QvQ-72B-preview (visual reasoning)
- 2025-01Jan-10: LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs (preprint, code, weights)
- 2025-01Jan-20: DeepSeek-R1, DeepSeek-R1-Distill-Llama-70B, DeepSeek-R1-Distill-Qwen-32B, ... (paper)
- 2025-02Feb-10: Huginn-0125: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (code, model)
- 2025-02Feb-14: DeepHermes 3 - Llama-3.1 8B
- 2025-02Feb-24: Qwen QwQ-Max-Preview (online demo)
- 2025-03Mar-05: Qwen QwQ-32B (demo)
- 2025-03Mar-05: RWKV7-G1 "GooseOne" 0.1B (weights, preprint)
- 2025-03Mar-17: LG AI Research EXAONE Deep 2.4B, 7.8B, 32B (weights)
Agentic
- 2025-02Feb-18: Microsoft Magma-8B (preprint)
- 2025-02Feb-26: Convergence Proxy Lite
Multimodal
Language/Vision
- LLaVA-NeXT-Interleave (models, demo)
- SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models
- Nvidia NVEagle 13B, 7B (demo, preprint)
- 2024-08Aug-29: Qwen2-VL 7B, 2B (code, models): Can process videos up to 20 minutes in length
- 2024-09Sep-11: Mistral Pixtral 12B
- 2024-09Sep-17: NVLM 1.0
- 2024-12Dec-06: Nvidia NVILA: Efficient Frontier Visual Language Models
- 2025-01Jan-28: Qwen2.5-VL
- 2025-02Feb-18: Microsoft Magma-8B (preprint)
- 2025-03Mar-05: Cohere Aya 8B, 32B
- 2025-03Mar-12: Google Gemma 3 1B 4B, 12B, 27B (technical report)
Language/Vision/Speech
- 2025-02Feb-27: Microsoft Phi-4-multimodal-instruct (language, vision, speech)
Language/Audio
- 2025-03Mar-11: Mellow: a small audio language model for reasoning, 167M (paper)
- 2025-03Mar-12: Audio Flamingo 2 0.5B, 1.5B, 3B paper, code
Cloud LLM
Multi-modal: Audio
- kyutai Open Science AI Lab chatbot moshi
Triage
Retrieval Augmented Generation (RAG)
- See Also: Document Parsing
Reviews
- 2024-08: Graph Retrieval-Augmented Generation: A Survey
- 2024-09: Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
- 2024-12: A Survey of Query Optimization in Large Language Models
- 2025-01: Enhancing Retrieval-Augmented Generation: A Study of Best Practices
- 2025-01: Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (github)
- List of RAG techniques
- Advanced RAG Cookbooks👨🏻💻
- Awesome-GraphRAG (GraphRAG Survey)
Measuring RAG performance
- 2025-01: The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
Analysis of RAG overall
Approaches
- RAGFlow (code)
- GraphRAG (preprint, code, GraphRAG Accelerator for easy deployment on Azure)
- AutoMetaRAG (code)
- Verba: RAG for Weaviate vector database (code, video)
- Microsoft: PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generation
- 2024-10: Google Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models
- 2024-10: StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization: Reformats retrieved data into task-appropriate structures (table, graph, tree).
- 2024-10: Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval
- 2024-11: FastRAG: Retrieval Augmented Generation for Semi-structured Data
- 2024-11: Microsoft LazyGraphRAG: Setting a new standard for quality and cost
- 2024-11: Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models
- 2025-01: Search-o1: Agentic Search-Enhanced Large Reasoning Models (project, code)
- 2025-01: AutoRAG: RAG AutoML tool for automatically finding an optimal RAG pipeline for your data
- 2025-01: VideoRAG: Retrieval-Augmented Generation over Video Corpus
- 2025-02: DeepRAG: Thinking to Retrieval Step by Step for Large Language Models
- 2025-02: Multi-vector embeddings
Open-source Implementations
- kotaemon: An open-source clean & customizable RAG UI for chatting with your documents.
- LlamaIndex (code, docs, voice chat code)
- Nvidia ChatRTX with RAG
- Anthropic Customer Support Agent example
- LangChain and LangGraph (tutorial)
- RAGBuilder: Automatically tunes RAG hyperparams
- WikiChat
- Chonkie: No-nonsense RAG chunking library (open-source, lightweight, fast)
- autoflow: open source GraphRAG (Knowledge Graph), including conversational search page
- RAGLite
- nano-graphrag: A simple, easy-to-hack GraphRAG implementation
- Dabarqus
Web-based Tools
- SciSpace Chat with PDF (also available as a GPT).
Commercial Cloud Offerings
LLM for scoring/ranking
- GPTScore: Evaluate as You Desire
- Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
- Domain-specific chatbots for science using embeddings
- Large Language Models as Evaluators for Scientific Synthesis
LLM Agents
- See AI Agents.
Interfaces
Chatbot Frontend
Web (code)
Web (product)
Desktop GUI
- AnythingLLM (docs, code): includes chat-with-docs, selection of LLM and vector db, etc.
Alternative Text Chatbot UI
- Loom provides a sort of tree-like structure for LLM coming up with branched writings.
- The Pantheon Interface is a new idea for how to interact with LLMs (live instance, code). In a traditional interaction, you prompt the bot and it replies in a turn-by-turn manner. Pantheon instead invites you to type out your thoughts, and various agents will asynchronously add comments or questions to spur along your brainstorming.
Conversational Audio Chatbot
- Swift is a fast AI voice assistant (code, live demo) uses:
- RTVI-AI (code, demo), uses:
- June: Local Voice Chatbot
- Ollama
- Hugging Face Transformers (for speech recognition)
- Coqui TTS Toolkit
- kyutai Moshi chatbot (demo)
- Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming (model, code, demo)
- 2024-09Sep-11: Llama-3.1-8B-Omni (code), enabling end-to-end speech.
- 2024-10Oct-18: Meta Spirit LM: open source multimodal language model that freely mixes text and speech
- 2025-02Feb-28: Sesame (demo)
Turn Detection
- 2025-03: Smart Turn: Open-source
Related Research
Commercial Systems
Speech Recognition (ASR) and Transcription
Lists
Open Source
- DeepSpeech
- speechbrain
- Kaldi
- wav2vec 2.0
- Whisper
- Whisper medium.en
- WhisperX (includes word-level timestamps and speaker diarization)
- Distil Large v3 with MLX
- 2024-10: whisper-large-v3-turbo distillation (demo, code)
- Nvidia Canary 1B
- 2024-09: Nvidia NeMo
- 2024-10: Rev AI models for transcription and diarization
- 2024-10: Moonshine (optimized for resource-constrained devices)
In Browser
- Whisper Timestamped: Multilingual speech recognition with word-level timestamps, running locally in browser
Phrase Endpointing and Voice Activity Detection (VAD)
I.e. how to determine when user is done talking, and bot should respond?
Audio Cleanup
- Krisp AI: Noise cancellation, meeting summary, etc.
Text-to-speech (TTS)
Open Source
- Parler TTS (demo)
- Toucan (demo)
- MetaVoice (github)
- ChatTTS
- Camb.ai MARS5-TTS
- Coqui TTS Toolkit
- Fish Speech 1.4: multi-lingual, can clone voices (video, weights, demo)
- F5-TTS (demo): cloning, emotion, etc.
- MaskGCT (demo)
- Amphion: An Open-Source Audio, Music and Speech Generation Toolkit (code)
- Zyphra Zonos
- Fish Speech (includes voice cloning)
Cloud
- Elevenlabs ($50/million characters)
- Cartesia Sonic
- Neets AI ($1/million characters)
- Hailuo AI T2A-01-HD (try, API)
- Hume (can set emotion, give acting directions, etc.)
Text-to-audio
- 2024-12: TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization (code)
Vision
- Langfun library as a means of converting images into structured output.
- See also: Multimodal open-weights models
Visual Models
- CLIP
- Siglip
- Supervision
- Florence-2
- Nvidia MambaVision
- Meta Sapiens: Foundation for Human Vision Models (video input, can infer segmentation, pose, depth-map, and surface normals)
Depth
- 2024-06: Depth Anything V2 (code)
Superresolution
Related
Embedding
Text Embedding
- 2024-12: modernBERT
- 2025-02: NeoBERT
- 2025-03: gemini-embedding-exp-03-07
Image Embedding
Time Series
- Stumpy: Python library, uses near-match subsequences for similarity and forecasting
- Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting
- From latent dynamics to meaningful representations
- Review of Time Series Forecasting Methods and Their Applications to Particle Accelerators
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
- A decoder-only foundation model for time-series forecasting
- TimeGPT-1
- Unified Training of Universal Time Series Forecasting Transformers
- xLSTMTime : Long-term Time Series Forecasting With xLSTM
- Salesforce: Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts (code, weights, blog)
- IBM PatchTSMixer and PatchTST (being used for particle accelerators)
Control
Forecasting
- Meta Kats (code): Forecasting (ARIMA, Prophet, Holt Winters, VAR), detection, feature extraction, simulation
- Context is Key: A Benchmark for Forecasting with Essential Textual Information
Anomaly Detection
- 2024-10: Can LLMs Understand Time Series Anomalies? (code)
Data
- See also: Data Scraping and Document Parsing
Vector Database
Open Source
- milvus (open source with paid cloud option)
- Qdrant (open source with paid cloud option)
- Vespa (open source with paid cloud option)
- chroma
- LlamaIndex
- sqlite-vec
Commercial cloud
MySQL
- MySQL does not traditionally have support, but:
- PlanetScale is working on it
- mysql_vss (discussion)
- tibd (discussion)
