Difference between revisions of "AI Agents"
KevinYager (talk | contribs) (→Inter-agent communications) |
KevinYager (talk | contribs) (→Deep Research) |
||
| (121 intermediate revisions by the same user not shown) | |||
| Line 11: | Line 11: | ||
* [https://github.com/open-thought/system-2-research OpenThought - System 2 Research Links] | * [https://github.com/open-thought/system-2-research OpenThought - System 2 Research Links] | ||
* [https://github.com/hijkzzz/Awesome-LLM-Strawberry Awesome LLM Strawberry (OpenAI o1): Collection of research papers & blogs for OpenAI Strawberry(o1) and Reasoning] | * [https://github.com/hijkzzz/Awesome-LLM-Strawberry Awesome LLM Strawberry (OpenAI o1): Collection of research papers & blogs for OpenAI Strawberry(o1) and Reasoning] | ||
| + | * [https://github.com/e2b-dev/awesome-ai-agents Awesome AI Agents] | ||
===Analysis/Opinions=== | ===Analysis/Opinions=== | ||
* [https://arxiv.org/abs/2402.01817v3 LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks] | * [https://arxiv.org/abs/2402.01817v3 LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks] | ||
* [https://rasa.com/blog/cutting-ai-assistant-costs-the-power-of-enhancing-llms-with-business/ Cutting AI Assistant Costs by Up to 77.8%: The Power of Enhancing LLMs with Business Logic] | * [https://rasa.com/blog/cutting-ai-assistant-costs-the-power-of-enhancing-llms-with-business/ Cutting AI Assistant Costs by Up to 77.8%: The Power of Enhancing LLMs with Business Logic] | ||
| + | |||
| + | ===Guides=== | ||
| + | * Anthropic: [https://www.anthropic.com/research/building-effective-agents Building Effective Agents] | ||
| + | * Google: [https://www.kaggle.com/whitepaper-agents Agents] | ||
=AI Assistants= | =AI Assistants= | ||
| Line 20: | Line 25: | ||
==Components of AI Assistants== | ==Components of AI Assistants== | ||
| − | ===Information Retrieval=== | + | ===Agent Internal Workflow Management=== |
| + | * [https://github.com/langchain-ai/langchain LangChain] | ||
| + | * [https://github.com/pydantic/pydantic-ai Pydantic: Agent Framework / shim to use Pydantic with LLMs] | ||
| + | * [https://github.com/lmnr-ai/flow Flow: A lightweight task engine for building AI agents that prioritizes simplicity and flexibility] | ||
| + | * [https://llama-stack.readthedocs.io/en/latest/index.html llama-stack] | ||
| + | * [https://huggingface.co/blog/smolagents Huggingface] [https://github.com/huggingface/smolagents smolagents] | ||
| + | * [https://github.com/elizaOS/eliza Eliza] (includes multi-agent, interaction with docs, Discord, Twitter, etc.) | ||
| + | * [https://github.com/The-Pocket/PocketFlow Pocket Flow]: LLM Framework in 100 Lines | ||
| + | |||
| + | ===Information Retrieval (Memory)=== | ||
* See also [[AI_tools#Retrieval_Augmented_Generation_.28RAG.29|RAG]]. | * See also [[AI_tools#Retrieval_Augmented_Generation_.28RAG.29|RAG]]. | ||
| + | * 2024-09: PaperQA2: [https://paper.wikicrow.ai/ Language Models Achieve Superhuman Synthesis of Scientific Knowledge] ([https://x.com/SGRodriques/status/1833908643856818443 𝕏 post], [https://github.com/Future-House/paper-qa code]) | ||
* 2024-10: [https://arxiv.org/abs/2410.09713 Agentic Information Retrieval] | * 2024-10: [https://arxiv.org/abs/2410.09713 Agentic Information Retrieval] | ||
| + | * 2025-02: [https://arxiv.org/abs/2502.01142 DeepRAG: Thinking to Retrieval Step by Step for Large Language Models] | ||
| + | * [https://mem0.ai/ Mem0 AI]: Memory Layer for AI Agents; self-improving memory layer for LLM applications, enabling personalized. | ||
| + | |||
| + | ===Contextual Memory=== | ||
| + | * [https://github.com/memodb-io/memobase Memobase]: user profile-based memory (long-term user memory for genAI) applications) | ||
| + | |||
| + | ===Control (tool-use, computer use, etc.)=== | ||
| + | * See also: [[Human_Computer_Interaction#AI_Computer_Use]] | ||
| + | * [https://tavily.com/ Tavily]: Connect Your LLM to the Web: Empowering your AI applications with real-time, accurate search results tailored for LLMs and RAG | ||
| + | ===Model Context Protocol (MCP)=== | ||
| + | * '''Standards:''' | ||
| + | *# Anthropic [https://www.anthropic.com/news/model-context-protocol Model Context Protocol] (MCP) | ||
| + | *# [https://openai.github.io/openai-agents-python/mcp/ OpenAI Agents SDK] | ||
| + | * '''Tools:''' | ||
| + | ** [https://github.com/jlowin/fastmcp FastMCP]: The fast, Pythonic way to build MCP servers | ||
| + | ** [https://github.com/fleuristes/fleur/ Fleur]: A desktop app marketplace for Claude Desktop | ||
| + | * '''Servers:''' | ||
| + | ** '''Lists:''' | ||
| + | **# [https://github.com/modelcontextprotocol/servers Model Context Protocol servers] | ||
| + | **# [https://www.mcpt.com/ MCP Servers, One Managed Registry] | ||
| + | **# [https://github.com/punkpeye/awesome-mcp-servers Awesome MCP Servers] | ||
| + | ** '''Noteworthy:''' | ||
| + | **# [https://github.com/modelcontextprotocol/servers/tree/main/src/github Github MCP server] | ||
| + | **# [https://github.com/modelcontextprotocol/servers/tree/main/src/puppeteer Puppeteer] | ||
| + | **# [https://github.com/modelcontextprotocol/servers/tree/main/src/google-maps Google Maps MCP Server] | ||
| + | **# [https://github.com/modelcontextprotocol/servers/tree/main/src/slack Slack MCP Server] | ||
===Open-source=== | ===Open-source=== | ||
| Line 40: | Line 81: | ||
===Computer Use=== | ===Computer Use=== | ||
| − | * 2024-11: [https:// | + | * See: [[Human_Computer_Interaction#AI_Computer_Use]] |
| + | |||
| + | ===Software Engineering=== | ||
| + | * 2024-11: [https://github.com/MLSysOps/MLE-agent MLE-Agent: Your intelligent companion for seamless AI engineering and research] | ||
| + | * [https://github.com/OpenAutoCoder/Agentless Agentless]: agentless approach to automatically solve software development problems | ||
===Science Agents=== | ===Science Agents=== | ||
See [[Science Agents]]. | See [[Science Agents]]. | ||
| + | |||
| + | ===Medicine=== | ||
| + | * 2025-03: [https://news.microsoft.com/2025/03/03/microsoft-dragon-copilot-provides-the-healthcare-industrys-first-unified-voice-ai-assistant-that-enables-clinicians-to-streamline-clinical-documentation-surface-information-and-automate-task/ Microsoft Dragon Copilot]: streamline clinical workflows and paperwork | ||
===LLM-as-judge=== | ===LLM-as-judge=== | ||
| Line 49: | Line 97: | ||
* [https://www.philschmid.de/llm-evaluation LLM Evaluation doesn't need to be complicated] | * [https://www.philschmid.de/llm-evaluation LLM Evaluation doesn't need to be complicated] | ||
* [https://eugeneyan.com/writing/llm-evaluators/ Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)] | * [https://eugeneyan.com/writing/llm-evaluators/ Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)] | ||
| + | * [https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge Awesome-LLM-as-a-judge Survey] | ||
| + | * [https://github.com/haizelabs/Awesome-LLM-Judges haizelabs Awesome LLM Judges] | ||
* 2024-10: [https://arxiv.org/abs/2410.10934 Agent-as-a-Judge: Evaluate Agents with Agents] | * 2024-10: [https://arxiv.org/abs/2410.10934 Agent-as-a-Judge: Evaluate Agents with Agents] | ||
| + | * 2024-11: [https://arxiv.org/abs/2411.15594 A Survey on LLM-as-a-Judge] | ||
| + | * 2024-12: [https://arxiv.org/abs/2412.05579 LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods] | ||
| + | * 2025-03: [https://arxiv.org/abs/2503.19877 Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators] | ||
| + | |||
| + | ===Deep Research=== | ||
| + | * Google [https://blog.google/products/gemini/google-gemini-deep-research/ Deep Research] | ||
| + | * OpenAI [https://openai.com/index/introducing-deep-research/ Deep Research] | ||
| + | * Perplexity: | ||
| + | ** [https://www.perplexity.ai/ Search] | ||
| + | ** [https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research Deep Research] | ||
| + | * [https://exa.ai/ Exa AI]: | ||
| + | ** [https://exa.ai/websets Websets]: Web research agent | ||
| + | ** [https://demo.exa.ai/deepseekchat Web-search agent] powered by DeepSeek ([https://github.com/exa-labs/exa-deepseek-chat code]) or [https://o3minichat.exa.ai/ o3-mini] ([https://github.com/exa-labs/exa-o3mini-chat code]) | ||
| + | * [https://www.firecrawl.dev/ Firecrawl] [https://x.com/nickscamara_/status/1886287956291338689 wip] | ||
| + | * [https://x.com/mattshumer_ Matt Shumer] [https://github.com/mshumer/OpenDeepResearcher OpenDeepResearcher] | ||
| + | * [https://github.com/zilliztech/deep-searcher DeepSearcher] (operate on local data) | ||
| + | * [https://github.com/nickscamara nickscamara] [https://github.com/nickscamara/open-deep-research open-deep-research] | ||
| + | * [https://x.com/dzhng dzhng] [https://github.com/dzhng/deep-research deep-research] | ||
| + | * [https://huggingface.co/ huggingface] [https://huggingface.co/blog/open-deep-research open-Deep-research ([https://github.com/huggingface/smolagents/tree/main/examples/open_deep_research code]) | ||
| + | * xAI Grok 3 Deep Search | ||
| + | * [https://liner.com/news/introducing-deepresearch Liner Deep Research] | ||
| + | * [https://allenai.org/ Allen AI] (AI2) [https://paperfinder.allen.ai/chat Paper Finder] | ||
| + | * 2025-03: [https://arxiv.org/abs/2503.20201 Open Deep Search: Democratizing Search with Open-source Reasoning Agents] ([https://github.com/sentient-agi/OpenDeepSearch code]) | ||
=Advanced Workflows= | =Advanced Workflows= | ||
| Line 59: | Line 132: | ||
* [https://arxiv.org/abs/2409.05556 SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning] | * [https://arxiv.org/abs/2409.05556 SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning] | ||
** [https://github.com/lamm-mit/SciAgentsDiscovery code] | ** [https://github.com/lamm-mit/SciAgentsDiscovery code] | ||
| + | |||
| + | ===Streamline Administrative Tasks=== | ||
| + | * 2025-02: [https://er.educause.edu/articles/2025/2/ushering-in-a-new-era-of-ai-driven-data-insights-at-uc-san-diego Ushering in a New Era of AI-Driven Data Insights at UC San Diego] | ||
| + | |||
| + | ===Author Research Articles=== | ||
| + | * 2024-02: STORM: [https://arxiv.org/abs/2402.14207 Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models] ([https://www.aihero.dev/storm-generate-high-quality-articles-based-on-real-research discussion/analysis]) | ||
===Software Development Workflows=== | ===Software Development Workflows=== | ||
| Line 72: | Line 151: | ||
## [https://www.cursor.com/ Cursor] | ## [https://www.cursor.com/ Cursor] | ||
## [https://codeium.com/ Codeium] [https://codeium.com/windsurf Windsurf] (with "Cascade" AI Agent) | ## [https://codeium.com/ Codeium] [https://codeium.com/windsurf Windsurf] (with "Cascade" AI Agent) | ||
| + | ## ByteDance [https://www.trae.ai/ Trae AI] | ||
| + | ## [https://www.tabnine.com/ Tabnine] | ||
| + | ## [https://marketplace.visualstudio.com/items?itemName=Traycer.traycer-vscode Traycer] | ||
| + | ## [https://idx.dev/ IDX]: free | ||
| + | ## [https://github.com/codestoryai/aide Aide]: open-source AI-native code editor (fork of VS Code) | ||
| + | ## [https://www.continue.dev/ continue.dev]: open-source code assistant | ||
| + | ## [https://trypear.ai/ Pear AI]: open-source code editor | ||
| + | ## [https://haystackeditor.com/ Haystack Editor]: canvas UI | ||
| + | ## [https://onlook.com/ Onlook]: for designers | ||
# AI-assisted IDE, where the AI generates and manages the dev environment | # AI-assisted IDE, where the AI generates and manages the dev environment | ||
## [https://replit.com/ Replit] | ## [https://replit.com/ Replit] | ||
| Line 80: | Line 168: | ||
# Prompt-to-product | # Prompt-to-product | ||
## [https://githubnext.com/projects/github-spark Github Spark] ([https://x.com/ashtom/status/1851333075374051725 demo video]) | ## [https://githubnext.com/projects/github-spark Github Spark] ([https://x.com/ashtom/status/1851333075374051725 demo video]) | ||
| + | ## [https://www.create.xyz/ Create.xyz]: text-to-app, replicate product from link | ||
| + | ## [https://a0.dev/ a0.dev]: generate mobil apps (from your phone) | ||
| + | ## [https://softgen.ai/ Softgen]: web app developer | ||
| + | ## [https://wrapifai.com/ wrapifai]: build form-based apps | ||
| + | ## [https://lovable.dev/ Lovable]: web app (from text, screenshot, etc.) | ||
| + | ## [https://v0.dev/ Vercel v0] | ||
| + | ## [https://x.com/johnrushx/status/1625179509728198665 MarsX] ([https://x.com/johnrushx John Rush]): SaaS builder | ||
| + | ## [https://webdraw.com/ Webdraw]: turn sketches into web apps | ||
| + | ## [https://www.tempo.new/ Tempo Labs]: build React apps | ||
| + | ## [https://databutton.com/ Databutton]: no-code software development | ||
| + | ## [https://base44.com/ base44]: no-code dashboard apps | ||
| + | ## [https://www.theorigin.ai/ Origin AI] | ||
# Semi-autonomous software engineer agents | # Semi-autonomous software engineer agents | ||
## [https://www.cognition.ai/blog/introducing-devin Devin] (Cognition AI) | ## [https://www.cognition.ai/blog/introducing-devin Devin] (Cognition AI) | ||
| − | ## [https://aws.amazon.com/q/ Amazon Q] | + | ## [https://aws.amazon.com/q/ Amazon Q] (and CodeWhisperer) |
## [https://honeycomb.sh/ Honeycomb] | ## [https://honeycomb.sh/ Honeycomb] | ||
| + | ## [https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview Claude Code] | ||
For a review of the current state of software-engineering agentic approaches, see: | For a review of the current state of software-engineering agentic approaches, see: | ||
| Line 100: | Line 201: | ||
==Inference-compute Reasoning== | ==Inference-compute Reasoning== | ||
* [https://nousresearch.com/#popup-menu-anchor Nous Research]: [https://nousresearch.com/introducing-the-forge-reasoning-api-beta-and-nous-chat-an-evolution-in-llm-inference/ Forge Reasoning API Beta] | * [https://nousresearch.com/#popup-menu-anchor Nous Research]: [https://nousresearch.com/introducing-the-forge-reasoning-api-beta-and-nous-chat-an-evolution-in-llm-inference/ Forge Reasoning API Beta] | ||
| + | |||
| + | ==AI Assistant== | ||
| + | * [https://convergence.ai/ Convergence] [https://proxy.convergence.ai/ Proxy] | ||
==Agentic Systems== | ==Agentic Systems== | ||
| Line 105: | Line 209: | ||
* [https://www.cognition.ai/ Cognition AI]: [https://www.cognition.ai/blog/introducing-devin Devin] software engineer (14% SWE-Agent) | * [https://www.cognition.ai/ Cognition AI]: [https://www.cognition.ai/blog/introducing-devin Devin] software engineer (14% SWE-Agent) | ||
* [https://honeycomb.sh/ Honeycomb] ([https://honeycomb.sh/blog/swe-bench-technical-report 22% SWE-Agent]) | * [https://honeycomb.sh/ Honeycomb] ([https://honeycomb.sh/blog/swe-bench-technical-report 22% SWE-Agent]) | ||
| + | * [https://www.factory.ai/ Factory AI] | ||
=Increasing AI Agent Intelligence= | =Increasing AI Agent Intelligence= | ||
| + | See: [[Increasing AI Intelligence]] | ||
| − | == | + | =Multi-agent orchestration= |
| − | + | ==Research== | |
| − | + | * 2025-03: [https://arxiv.org/abs/2503.13657 Why Do Multi-Agent LLM Systems Fail?] | |
| − | == | + | * 2025-03: [https://arxiv.org/abs/2503.15478 SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks] |
| − | * | ||
| − | |||
| − | * | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | === | + | ===Organization Schemes=== |
| − | * | + | * 2025-03: [https://arxiv.org/abs/2503.02390 ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Tasks] |
| − | === | + | ===Societies and Communities of AI agents=== |
| − | * 2024- | + | * 2024-12: [https://arxiv.org/abs/2412.10270 Cultural Evolution of Cooperation among LLM Agents] |
| − | === | + | ===Domain-specific=== |
| − | + | * 2024-12: [https://arxiv.org/abs/2412.20138 TradingAgents: Multi-Agents LLM Financial Trading Framework] | |
| − | + | * 2025-01: [https://arxiv.org/abs/2501.04227 Agent Laboratory: Using LLM Agents as Research Assistants] | |
| − | * 2024- | ||
| − | |||
| − | * | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
==Research demos== | ==Research demos== | ||
* [https://github.com/camel-ai/camel Camel] | * [https://github.com/camel-ai/camel Camel] | ||
| Line 276: | Line 249: | ||
* 2024-06: [https://arxiv.org/abs/2406.11638 MASAI: Modular Architecture for Software-engineering AI Agents] | * 2024-06: [https://arxiv.org/abs/2406.11638 MASAI: Modular Architecture for Software-engineering AI Agents] | ||
* 2024-10: [https://arxiv.org/abs/2410.08164 Agent S: An Open Agentic Framework that Uses Computers Like a Human] ([https://github.com/simular-ai/Agent-S code]) | * 2024-10: [https://arxiv.org/abs/2410.08164 Agent S: An Open Agentic Framework that Uses Computers Like a Human] ([https://github.com/simular-ai/Agent-S code]) | ||
| + | * 2024-10: [https://arxiv.org/abs/2410.20424 AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions] | ||
| + | * 2025-02: [https://arxiv.org/abs/2502.16111 PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving] | ||
===Related work=== | ===Related work=== | ||
| Line 298: | Line 273: | ||
* Amazon AWS [https://github.com/awslabs/multi-agent-orchestrator Multi-Agent Orchestrator] | * Amazon AWS [https://github.com/awslabs/multi-agent-orchestrator Multi-Agent Orchestrator] | ||
* [https://github.com/kaiban-ai/KaibanJS KaibanJS]: Kanban for AI Agents? (Takes inspiration from [https://en.wikipedia.org/wiki/Kanban Kanban] visual [https://www.atlassian.com/agile/kanban work management].) | * [https://github.com/kaiban-ai/KaibanJS KaibanJS]: Kanban for AI Agents? (Takes inspiration from [https://en.wikipedia.org/wiki/Kanban Kanban] visual [https://www.atlassian.com/agile/kanban work management].) | ||
| + | * [https://github.com/Thytu/Agentarium Agentarium] | ||
| + | * [https://orchestra.org/ Orchestra] ([https://docs.orchestra.org/orchestra/introduction docs], [https://docs.orchestra.org/orchestra/introduction code]) | ||
| + | * [https://github.com/HKUDS/AutoAgent AutoAgent]: Fully-Automated & Zero-Code LLM Agent Framework | ||
| + | * [https://mastra.ai/ Mastra] ([https://github.com/mastra-ai/mastra github]): opinionated Typescript framework for AI applications (primitives for workflows, agents, RAG, integrations and evals) | ||
| + | * [https://github.com/orra-dev/orra Orra]: multi-agent applications with complex real-world interactions | ||
| + | * [https://github.com/gensx-inc/gensx/blob/main/README.md GenSX] | ||
| + | * Cloudflare [https://developers.cloudflare.com/agents/ agents-sdk] ([https://blog.cloudflare.com/build-ai-agents-on-cloudflare/ info], [https://github.com/cloudflare/agents code]) | ||
| + | * OpenAI [https://platform.openai.com/docs/api-reference/responses responses API] and [https://platform.openai.com/docs/guides/agents agents SDK] | ||
==Open Source Systems== | ==Open Source Systems== | ||
| Line 327: | Line 310: | ||
* [https://ottogrid.ai/ Otto Grid] | * [https://ottogrid.ai/ Otto Grid] | ||
* [https://www.paradigmai.com/ Paradigm] | * [https://www.paradigmai.com/ Paradigm] | ||
| + | * [https://www.superworker.ai/ Superworker AI] | ||
==Cloud solutions== | ==Cloud solutions== | ||
| Line 342: | Line 326: | ||
=Optimization= | =Optimization= | ||
| + | ===Reviews=== | ||
| + | * 2024-12: [https://arxiv.org/abs/2412.11936 A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges] | ||
| + | * 2025-03: [https://arxiv.org/abs/2503.16416 Survey on Evaluation of LLM-based Agents] | ||
| + | |||
===Metrics, Benchmarks=== | ===Metrics, Benchmarks=== | ||
| + | * 2019-11: [https://arxiv.org/abs/1911.01547 On the Measure of Intelligence] | ||
* 2022-06: [https://arxiv.org/abs/2206.10498 PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change] | * 2022-06: [https://arxiv.org/abs/2206.10498 PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change] | ||
| + | * 2023-06: [https://arxiv.org/abs/2306.05836 Can Large Language Models Infer Causation from Correlation?] (challenging Corr2Cause task) | ||
| + | * 2024-01: [https://microsoft.github.io/autogen/0.2/blog/2024/01/25/AutoGenBench/ AutoGenBench -- A Tool for Measuring and Evaluating AutoGen Agents] | ||
* 2024-04: AutoRace ([https://github.com/maitrix-org/llm-reasoners code]): [https://arxiv.org/abs/2404.05221 LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models] | * 2024-04: AutoRace ([https://github.com/maitrix-org/llm-reasoners code]): [https://arxiv.org/abs/2404.05221 LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models] | ||
* 2024-04: [https://arxiv.org/abs/2404.07972 OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments] ([https://os-world.github.io/ github]) | * 2024-04: [https://arxiv.org/abs/2404.07972 OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments] ([https://os-world.github.io/ github]) | ||
| Line 355: | Line 346: | ||
* 2024-10: SimpleAQ: [https://cdn.openai.com/papers/simpleqa.pdf Measuring short-form factuality in large language models] ([https://openai.com/index/introducing-simpleqa/ announcement], [https://github.com/openai/simple-evals code]) | * 2024-10: SimpleAQ: [https://cdn.openai.com/papers/simpleqa.pdf Measuring short-form factuality in large language models] ([https://openai.com/index/introducing-simpleqa/ announcement], [https://github.com/openai/simple-evals code]) | ||
* 2024-11: [https://metr.org/AI_R_D_Evaluation_Report.pdf RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts] ([https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/ blog], [https://github.com/METR/ai-rd-tasks/tree/main code]) | * 2024-11: [https://metr.org/AI_R_D_Evaluation_Report.pdf RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts] ([https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/ blog], [https://github.com/METR/ai-rd-tasks/tree/main code]) | ||
| + | * 2024-11: [https://arxiv.org/abs/2411.10323 The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use] ([https://github.com/showlab/computer_use_ootb code]) | ||
| + | * 2024-11: [https://arxiv.org/abs/2411.13543 BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games] | ||
| + | * 2024-12: [https://arxiv.org/abs/2412.14161 TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks] ([https://github.com/TheAgentCompany/TheAgentCompany code], [https://the-agent-company.com/ project], [https://the-agent-company.com/#/leaderboard leaderboard]) | ||
| + | * 2025-01: [https://codeelo-bench.github.io/ CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings] ([https://arxiv.org/abs/2501.01257 preprint], [https://codeelo-bench.github.io/#leaderboard-table leaderboard]) | ||
| + | * 2025-02: [https://static.scale.com/uploads/654197dc94d34f66c0f5184e/EnigmaEval%20v4.pdf ENIGMAEVAL:A Benchmark of Long Multimodal Reasoning Challenges] ([https://scale.com/leaderboard/enigma_eval leaderboard]) | ||
| + | * 2025-02: [https://sites.google.com/view/mlgym MLGym: A New Framework and Benchmark for Advancing AI Research Agents] ([https://arxiv.org/abs/2502.14499 paper], [https://github.com/facebookresearch/MLGym code]) | ||
| + | * 2025-02: [https://arxiv.org/abs/2502.18356 WebGames: Challenging General-Purpose Web-Browsing AI Agents] | ||
| + | * 2025-03: ColBench: [https://arxiv.org/abs/2503.15478 SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks] | ||
| + | |||
| + | ===Evaluation Schemes=== | ||
| + | * 2024-12: [https://arxiv.org/abs/2412.10424 LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation] | ||
| + | * 2025-01: [https://github.com/marquisdepolis/LLMRank LLMRank ("SlopRank")]: LLMs evaluate each other, allowing top model (for a given prompt/problem) to be inferred from a large number of recommendations. | ||
| + | |||
| + | ===Multi-agent=== | ||
| + | * 2024-12: [https://arxiv.org/abs/2412.10270 Cultural Evolution of Cooperation among LLM Agents] | ||
| + | * [https://github.com/lechmazur/step_game/ Multi-Agent Step Race Benchmark: Assessing LLM Collaboration and Deception Under Pressure] | ||
===Agent Challenges=== | ===Agent Challenges=== | ||
* [https://github.com/aidanmclaughlin/Aidan-Bench Aidan-Bench]: Test creativity by having a particular LLM generate long sequence of outputs (meant to be different), and measuring how long it can go before duplications appear. | * [https://github.com/aidanmclaughlin/Aidan-Bench Aidan-Bench]: Test creativity by having a particular LLM generate long sequence of outputs (meant to be different), and measuring how long it can go before duplications appear. | ||
| + | ** NeurIPS 2024 paper/poster: [https://openreview.net/pdf?id=fz969ahcvJ AidanBench: Evaluating Novel Idea Generation on Open-Ended Questions] | ||
* [https://x.com/paul_cal/status/1850262678712856764 Pictionary]: LLM suggests prompt, multiple LLMs generate outputs, LLM judges; allows raking of the generation abilities. | * [https://x.com/paul_cal/status/1850262678712856764 Pictionary]: LLM suggests prompt, multiple LLMs generate outputs, LLM judges; allows raking of the generation abilities. | ||
* [https://github.com/mc-bench/orchestrator MC-bench]: Request LLMs to build an elaborate structure in Minecraft; outputs can be A/B tested by human judges. | * [https://github.com/mc-bench/orchestrator MC-bench]: Request LLMs to build an elaborate structure in Minecraft; outputs can be A/B tested by human judges. | ||
| Line 369: | Line 377: | ||
=See Also= | =See Also= | ||
* [[Science Agents]] | * [[Science Agents]] | ||
| + | * [[Increasing AI Intelligence]] | ||
* [[AI tools]] | * [[AI tools]] | ||
* [[AI understanding]] | * [[AI understanding]] | ||
* [[Robots]] | * [[Robots]] | ||
* [[Exocortex]] | * [[Exocortex]] | ||
Latest revision as of 16:53, 31 March 2025
Contents
- 1 Reviews & Perspectives
- 2 AI Assistants
- 3 Advanced Workflows
- 4 Corporate AI Agent Ventures
- 5 Increasing AI Agent Intelligence
- 6 Multi-agent orchestration
- 7 Optimization
- 8 See Also
Reviews & Perspectives
Published
- 2024-04: LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models (code)
- 2024-08: From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future
- 2024-09: Towards a Science Exocortex
- 2024-09: Large Language Model-Based Agents for Software Engineering: A Survey
- 2024-09: Agents in Software Engineering: Survey, Landscape, and Vision
Continually updating
- OpenThought - System 2 Research Links
- Awesome LLM Strawberry (OpenAI o1): Collection of research papers & blogs for OpenAI Strawberry(o1) and Reasoning
- Awesome AI Agents
Analysis/Opinions
- LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
- Cutting AI Assistant Costs by Up to 77.8%: The Power of Enhancing LLMs with Business Logic
Guides
- Anthropic: Building Effective Agents
- Google: Agents
AI Assistants
Components of AI Assistants
Agent Internal Workflow Management
- LangChain
- Pydantic: Agent Framework / shim to use Pydantic with LLMs
- Flow: A lightweight task engine for building AI agents that prioritizes simplicity and flexibility
- llama-stack
- Huggingface smolagents
- Eliza (includes multi-agent, interaction with docs, Discord, Twitter, etc.)
- Pocket Flow: LLM Framework in 100 Lines
Information Retrieval (Memory)
- See also RAG.
- 2024-09: PaperQA2: Language Models Achieve Superhuman Synthesis of Scientific Knowledge (𝕏 post, code)
- 2024-10: Agentic Information Retrieval
- 2025-02: DeepRAG: Thinking to Retrieval Step by Step for Large Language Models
- Mem0 AI: Memory Layer for AI Agents; self-improving memory layer for LLM applications, enabling personalized.
Contextual Memory
- Memobase: user profile-based memory (long-term user memory for genAI) applications)
Control (tool-use, computer use, etc.)
- See also: Human_Computer_Interaction#AI_Computer_Use
- Tavily: Connect Your LLM to the Web: Empowering your AI applications with real-time, accurate search results tailored for LLMs and RAG
Model Context Protocol (MCP)
- Standards:
- Anthropic Model Context Protocol (MCP)
- OpenAI Agents SDK
- Tools:
- Servers:
Open-source
- Khoj (code): self-hostable AI assistant
- RAGapp: Agentic RAG for enterprise
- STORM: Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking
- Can write (e.g.) Wikipedia-style articles
- code
- Preprint: Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models
Personalities/Personas
- 2023-10: Generative Agents: Interactive Simulacra of Human Behavior
- 2024-11: Microsoft TinyTroupe 🤠🤓🥸🧐: LLM-powered multiagent persona simulation for imagination enhancement and business insights
- 2024-11: Generative Agent Simulations of 1,000 People (code)
Specific Uses for AI Assistants
Computer Use
Software Engineering
- 2024-11: MLE-Agent: Your intelligent companion for seamless AI engineering and research
- Agentless: agentless approach to automatically solve software development problems
Science Agents
See Science Agents.
Medicine
- 2025-03: Microsoft Dragon Copilot: streamline clinical workflows and paperwork
LLM-as-judge
- List of papers.
- LLM Evaluation doesn't need to be complicated
- Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
- Awesome-LLM-as-a-judge Survey
- haizelabs Awesome LLM Judges
- 2024-10: Agent-as-a-Judge: Evaluate Agents with Agents
- 2024-11: A Survey on LLM-as-a-Judge
- 2024-12: LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- 2025-03: Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators
Deep Research
- Google Deep Research
- OpenAI Deep Research
- Perplexity:
- Exa AI:
- Websets: Web research agent
- Web-search agent powered by DeepSeek (code) or o3-mini (code)
- Firecrawl wip
- Matt Shumer OpenDeepResearcher
- DeepSearcher (operate on local data)
- nickscamara open-deep-research
- dzhng deep-research
- huggingface open-Deep-research ([https://github.com/huggingface/smolagents/tree/main/examples/open_deep_research code)
- xAI Grok 3 Deep Search
- Liner Deep Research
- Allen AI (AI2) Paper Finder
- 2025-03: Open Deep Search: Democratizing Search with Open-source Reasoning Agents (code)
Advanced Workflows
- Salesforce DEI: meta-system that leverages a diversity of SWE agents
- Sakana AI: AI Scientist
- SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning
Streamline Administrative Tasks
Author Research Articles
- 2024-02: STORM: Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models (discussion/analysis)
Software Development Workflows
Several paradigms of AI-assisted coding have arisen:
- Manual, human driven
- AI-aided through chat/dialogue, where the human asks for code and then copies it into the project
- API calls to an LLM, which generates code and inserts the file into the project
- LLM-integration into the IDE
- Copilot
- Qodo (Codium) & AlphaCodium (preprint, code)
- Cursor
- Codeium Windsurf (with "Cascade" AI Agent)
- ByteDance Trae AI
- Tabnine
- Traycer
- IDX: free
- Aide: open-source AI-native code editor (fork of VS Code)
- continue.dev: open-source code assistant
- Pear AI: open-source code editor
- Haystack Editor: canvas UI
- Onlook: for designers
- AI-assisted IDE, where the AI generates and manages the dev environment
- Replit
- Aider (code): Pair programming on commandline
- Pythagora
- StackBlitz bolt.new
- Cline (formerly Claude Dev)
- Prompt-to-product
- Github Spark (demo video)
- Create.xyz: text-to-app, replicate product from link
- a0.dev: generate mobil apps (from your phone)
- Softgen: web app developer
- wrapifai: build form-based apps
- Lovable: web app (from text, screenshot, etc.)
- Vercel v0
- MarsX (John Rush): SaaS builder
- Webdraw: turn sketches into web apps
- Tempo Labs: build React apps
- Databutton: no-code software development
- base44: no-code dashboard apps
- Origin AI
- Semi-autonomous software engineer agents
- Devin (Cognition AI)
- Amazon Q (and CodeWhisperer)
- Honeycomb
- Claude Code
For a review of the current state of software-engineering agentic approaches, see:
- 2024-08: From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future
- 2024-09: Large Language Model-Based Agents for Software Engineering: A Survey
- 2024-09: Agents in Software Engineering: Survey, Landscape, and Vision
Corporate AI Agent Ventures
Mundane Workflows and Capabilities
- Payman AI: AI to Human platform that allows AI to pay people for what it needs
- VoiceFlow: Build customer experiences with AI
- Mistral AI: genAI applications
- Taskade: Task/milestone software with AI agent workflows
- Covalent: Building a Multi-Agent Prompt Refining Application
Inference-compute Reasoning
AI Assistant
Agentic Systems
- Topology AI
- Cognition AI: Devin software engineer (14% SWE-Agent)
- Honeycomb (22% SWE-Agent)
- Factory AI
Increasing AI Agent Intelligence
See: Increasing AI Intelligence
Multi-agent orchestration
Research
- 2025-03: Why Do Multi-Agent LLM Systems Fail?
- 2025-03: SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
Organization Schemes
Societies and Communities of AI agents
Domain-specific
- 2024-12: TradingAgents: Multi-Agents LLM Financial Trading Framework
- 2025-01: Agent Laboratory: Using LLM Agents as Research Assistants
Research demos
- Camel
- LoopGPT
- JARVIS
- OpenAGI
- AutoGen
- preprint: AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
- Agent-E: Browser (eventually computer) automation (code, preprint, demo video)
- AutoGen Studio: GUI for agent workflows (code)
- Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
- AG2 (previously AutoGen) (code, docs, Discord)
- TaskWeaver
- MetaGPT
- AutoGPT (code); and AutoGPT Platform
- Optima
- 2024-04: LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models (code)
- 2024-06: MASAI: Modular Architecture for Software-engineering AI Agents
- 2024-10: Agent S: An Open Agentic Framework that Uses Computers Like a Human (code)
- 2024-10: AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions
- 2025-02: PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving
Related work
Inter-agent communications
- 2024-10: Agora: A Scalable Communication Protocol for Networks of Large Language Models (preprint): disparate agents auto-negotiate communication protocol
- 2024-11: DroidSpeak: Enhancing Cross-LLM Communication: Exploits caches of embeddings and key-values, to allow context to be more easily transferred between AIs (without consuming context window)
- 2024-11: Anthropic describes Model Context Protocol: an open standard for secure, two-way connections between data sources and AI (intro, quickstart, code)
Architectures
Open Source Frameworks
- LangChain
- ell (code, docs)
- AgentOps AI AgentStack
- Agent UI
- kyegomez swarms
- OpenAI Swarm (cookbook)
- Amazon AWS Multi-Agent Orchestrator
- KaibanJS: Kanban for AI Agents? (Takes inspiration from Kanban visual work management.)
- Agentarium
- Orchestra (docs, code)
- AutoAgent: Fully-Automated & Zero-Code LLM Agent Framework
- Mastra (github): opinionated Typescript framework for AI applications (primitives for workflows, agents, RAG, integrations and evals)
- Orra: multi-agent applications with complex real-world interactions
- GenSX
- Cloudflare agents-sdk (info, code)
- OpenAI responses API and agents SDK
Open Source Systems
- ControlFlow
- OpenHands (formerly OpenDevin)
- code: platform for autonomous software engineers, powered by AI and LLMs
- Report: OpenDevin: An Open Platform for AI Software Developers as Generalist Agents
Commercial Automation Frameworks
- Lutra: Automation and integration with various web systems.
- Gumloop
- TextQL: Enterprise Virtual Data Analyst
- Athena intelligence: Analytics platform
- Nexus GPT: Business co-pilot
- Multi-On: AI agent that acts on your behalf
- Firecrawl: Turn websites into LLM-ready data
- Reworkd: End-to-end data extraction
- Lindy: Custom AI Assistants to automate business workflows
- E.g. use Slack
- Bardeen: Automate workflows
- Abacus: AI Agents
- LlamaIndex: (𝕏, code, docs, Discord)
- MultiOn AI: Agent Q (paper) automated planning and execution
Spreadsheet
Cloud solutions
- Numbers Station Meadow: agentic framework for data workflows (code).
- CrewAI says they provide multi-agent automations (code).
- LangChain introduced LangGraph to help build agents, and LangGraph Cloud as a service for running those agents.
- LangGraph Studio is an IDE for agent workflows
- C3 AI enterprise platform
- Deepset AI Haystack (docs, code)
Frameworks
- Google Project Oscar
- Agent: Gaby (for "Go AI bot") (code, documentation) helps with issue tracking.
- OpenPlexity-Pages: Data-aggregator implementation (like Perplexity) based on CrewAI
Optimization
Reviews
- 2024-12: A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges
- 2025-03: Survey on Evaluation of LLM-based Agents
Metrics, Benchmarks
- 2019-11: On the Measure of Intelligence
- 2022-06: PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
- 2023-06: Can Large Language Models Infer Causation from Correlation? (challenging Corr2Cause task)
- 2024-01: AutoGenBench -- A Tool for Measuring and Evaluating AutoGen Agents
- 2024-04: AutoRace (code): LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
- 2024-04: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (github)
- 2024-07: AI Agents That Matter
- 2024-09: CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark (leaderboard)
- 2024-09: LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench
- 2024-09: On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability
- 2024-10: MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
- 2024-10: WorFBench: Benchmarking Agentic Workflow Generation
- 2024-10: VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models
- 2024-10: SimpleAQ: Measuring short-form factuality in large language models (announcement, code)
- 2024-11: RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (blog, code)
- 2024-11: The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use (code)
- 2024-11: BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
- 2024-12: TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (code, project, leaderboard)
- 2025-01: CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings (preprint, leaderboard)
- 2025-02: ENIGMAEVAL:A Benchmark of Long Multimodal Reasoning Challenges (leaderboard)
- 2025-02: MLGym: A New Framework and Benchmark for Advancing AI Research Agents (paper, code)
- 2025-02: WebGames: Challenging General-Purpose Web-Browsing AI Agents
- 2025-03: ColBench: SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
Evaluation Schemes
- 2024-12: LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation
- 2025-01: LLMRank ("SlopRank"): LLMs evaluate each other, allowing top model (for a given prompt/problem) to be inferred from a large number of recommendations.
Multi-agent
- 2024-12: Cultural Evolution of Cooperation among LLM Agents
- Multi-Agent Step Race Benchmark: Assessing LLM Collaboration and Deception Under Pressure
Agent Challenges
- Aidan-Bench: Test creativity by having a particular LLM generate long sequence of outputs (meant to be different), and measuring how long it can go before duplications appear.
- NeurIPS 2024 paper/poster: AidanBench: Evaluating Novel Idea Generation on Open-Ended Questions
- Pictionary: LLM suggests prompt, multiple LLMs generate outputs, LLM judges; allows raking of the generation abilities.
- MC-bench: Request LLMs to build an elaborate structure in Minecraft; outputs can be A/B tested by human judges.
Automated Improvement
- 2024-06: EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms
- 2024-06: Symbolic Learning Enables Self-Evolving Agents
- 2024-08: Automated Design of Agentic Systems (ADAS code)
- 2024-08: Self-Taught Evaluators: Iterative self-improvement through generation of synthetic data and evaluation
