Difference between revisions of "AI Agents"

From GISAXS
Jump to: navigation, search
(→‎LLM-as-judge)
(→‎Metrics, Benchmarks)
 
(13 intermediate revisions by the same user not shown)
Line 7: Line 7:
 
* 2024-09: [https://www.arxiv.org/abs/2409.02977 Large Language Model-Based Agents for Software Engineering: A Survey]
 
* 2024-09: [https://www.arxiv.org/abs/2409.02977 Large Language Model-Based Agents for Software Engineering: A Survey]
 
* 2024-09: [https://arxiv.org/abs/2409.09030 Agents in Software Engineering: Survey, Landscape, and Vision]
 
* 2024-09: [https://arxiv.org/abs/2409.09030 Agents in Software Engineering: Survey, Landscape, and Vision]
 +
* 2025-04: [https://arxiv.org/abs/2504.01990 Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems]
 +
* 2025-04: [https://arxiv.org/abs/2503.19213 A Survey of Large Language Model Agents for Question Answering]
  
 
===Continually updating===
 
===Continually updating===
Line 60: Line 62:
 
**# [https://github.com/punkpeye/awesome-mcp-servers Awesome MCP Servers]
 
**# [https://github.com/punkpeye/awesome-mcp-servers Awesome MCP Servers]
 
** '''Noteworthy:'''
 
** '''Noteworthy:'''
−
**# [https://github.com/modelcontextprotocol/servers/tree/main/src/github Github MCP server]
+
**# Official [https://github.com/github/github-mcp-server Github MCP server]
 +
**# Unofficial [https://github.com/modelcontextprotocol/servers/tree/main/src/github Github MCP server]
 
**# [https://github.com/modelcontextprotocol/servers/tree/main/src/puppeteer Puppeteer]
 
**# [https://github.com/modelcontextprotocol/servers/tree/main/src/puppeteer Puppeteer]
 
**# [https://github.com/modelcontextprotocol/servers/tree/main/src/google-maps Google Maps MCP Server]
 
**# [https://github.com/modelcontextprotocol/servers/tree/main/src/google-maps Google Maps MCP Server]
Line 66: Line 69:
 
**# [https://zapier.com/mcp Zapier MCP Servers] (Slack, Google Sheets, Notion, etc.)
 
**# [https://zapier.com/mcp Zapier MCP Servers] (Slack, Google Sheets, Notion, etc.)
 
**# [https://github.com/awslabs/mcp AWS MCP Servers]
 
**# [https://github.com/awslabs/mcp AWS MCP Servers]
 +
**# [https://x.com/elevenlabsio/status/1909300782673101265 ElevenLabs]
 +
 +
===Agent2Agent Protocol (A2A)===
 +
* Google [https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/ announcement]
  
 
===Open-source===
 
===Open-source===
Line 94: Line 101:
 
===Medicine===
 
===Medicine===
 
* 2025-03: [https://news.microsoft.com/2025/03/03/microsoft-dragon-copilot-provides-the-healthcare-industrys-first-unified-voice-ai-assistant-that-enables-clinicians-to-streamline-clinical-documentation-surface-information-and-automate-task/ Microsoft Dragon Copilot]: streamline clinical workflows and paperwork
 
* 2025-03: [https://news.microsoft.com/2025/03/03/microsoft-dragon-copilot-provides-the-healthcare-industrys-first-unified-voice-ai-assistant-that-enables-clinicians-to-streamline-clinical-documentation-surface-information-and-automate-task/ Microsoft Dragon Copilot]: streamline clinical workflows and paperwork
 +
* 2025-04: [https://arxiv.org/abs/2504.05186 Training state-of-the-art pathology foundation models with orders of magnitude less data]
 +
* 2025-04: [https://www.nature.com/articles/s41586-025-08866-7?linkId=13898052 Towards conversational diagnostic artificial intelligence]
 +
* 2025-04: [https://www.nature.com/articles/s41586-025-08869-4?linkId=13898054 Towards accurate differential diagnosis with large language models]
  
 
===LLM-as-judge===
 
===LLM-as-judge===
Line 127: Line 137:
 
* 2025-03: [https://arxiv.org/abs/2503.20201 Open Deep Search: Democratizing Search with Open-source Reasoning Agents] ([https://github.com/sentient-agi/OpenDeepSearch code])
 
* 2025-03: [https://arxiv.org/abs/2503.20201 Open Deep Search: Democratizing Search with Open-source Reasoning Agents] ([https://github.com/sentient-agi/OpenDeepSearch code])
 
* [https://convergence.ai/welcome Convergence AI] Deep Work (swarms for web-based tasks)
 
* [https://convergence.ai/welcome Convergence AI] Deep Work (swarms for web-based tasks)
 +
* 2025-04: [https://arxiv.org/abs/2504.03160 DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments]
  
 
=Advanced Workflows=
 
=Advanced Workflows=
Line 166: Line 177:
 
## [https://www.all-hands.dev/ All Hands AI]
 
## [https://www.all-hands.dev/ All Hands AI]
 
## [https://app.devin.ai/ Devin 2.0] ([https://cognition.ai/ Cognition AI])
 
## [https://app.devin.ai/ Devin 2.0] ([https://cognition.ai/ Cognition AI])
 +
## Google [https://firebase.google.com/docs/studio Firebase Studio]
 
# AI-assisted IDE, where the AI generates and manages the dev environment
 
# AI-assisted IDE, where the AI generates and manages the dev environment
 
## [https://replit.com/ Replit]
 
## [https://replit.com/ Replit]
Line 218: Line 230:
 
* [https://www.factory.ai/ Factory AI]
 
* [https://www.factory.ai/ Factory AI]
 
* [https://convergence.ai/welcome Convergence AI] Deep Work (swarms for web-based tasks)
 
* [https://convergence.ai/welcome Convergence AI] Deep Work (swarms for web-based tasks)
 +
* [https://agents.cloudflare.com/ Cloudflare Agents]
  
 
=Increasing AI Agent Intelligence=
 
=Increasing AI Agent Intelligence=
Line 262: Line 275:
 
===Related work===
 
===Related work===
 
* 2024-07: [https://arxiv.org/abs/2407.18416 PersonaGym: Evaluating Persona Agents and LLMs]
 
* 2024-07: [https://arxiv.org/abs/2407.18416 PersonaGym: Evaluating Persona Agents and LLMs]
 +
* 2025-01: [https://arxiv.org/abs/2501.13946 Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks]
  
 
===Inter-agent communications===
 
===Inter-agent communications===
Line 289: Line 303:
 
* Cloudflare [https://developers.cloudflare.com/agents/ agents-sdk] ([https://blog.cloudflare.com/build-ai-agents-on-cloudflare/ info], [https://github.com/cloudflare/agents code])
 
* Cloudflare [https://developers.cloudflare.com/agents/ agents-sdk] ([https://blog.cloudflare.com/build-ai-agents-on-cloudflare/ info], [https://github.com/cloudflare/agents code])
 
* OpenAI [https://platform.openai.com/docs/api-reference/responses responses API] and [https://platform.openai.com/docs/guides/agents agents SDK]
 
* OpenAI [https://platform.openai.com/docs/api-reference/responses responses API] and [https://platform.openai.com/docs/guides/agents agents SDK]
 +
* Google [https://google.github.io/adk-docs/ Agent Development Kit]
  
 
==Open Source Systems==
 
==Open Source Systems==
Line 313: Line 328:
 
* [https://www.llamaindex.ai/ LlamaIndex]: ([https://x.com/llama_index 𝕏], [https://github.com/run-llama/llama_index code], [https://docs.llamaindex.ai/en/stable/ docs], [https://discord.com/invite/dGcwcsnxhU Discord])
 
* [https://www.llamaindex.ai/ LlamaIndex]: ([https://x.com/llama_index 𝕏], [https://github.com/run-llama/llama_index code], [https://docs.llamaindex.ai/en/stable/ docs], [https://discord.com/invite/dGcwcsnxhU Discord])
 
* [https://www.multion.ai/ MultiOn AI]: [https://www.multion.ai/blog/introducing-agent-q-research-breakthrough-for-the-next-generation-of-ai-agents-with-planning-and-self-healing-capabilities Agent Q] ([https://multion-research.s3.us-east-2.amazonaws.com/AgentQ.pdf paper]) automated planning and execution
 
* [https://www.multion.ai/ MultiOn AI]: [https://www.multion.ai/blog/introducing-agent-q-research-breakthrough-for-the-next-generation-of-ai-agents-with-planning-and-self-healing-capabilities Agent Q] ([https://multion-research.s3.us-east-2.amazonaws.com/AgentQ.pdf paper]) automated planning and execution
 +
* Google [https://cloud.google.com/products/agentspace Agentspace]
 +
 +
===Multi-agent Handoff/Collaboration===
 +
* [https://www.maskara.ai/ Maskara AI]
  
 
===Spreadsheet===
 
===Spreadsheet===
Line 362: Line 381:
 
* 2025-02: [https://arxiv.org/abs/2502.18356 WebGames: Challenging General-Purpose Web-Browsing AI Agents]
 
* 2025-02: [https://arxiv.org/abs/2502.18356 WebGames: Challenging General-Purpose Web-Browsing AI Agents]
 
* 2025-03: ColBench: [https://arxiv.org/abs/2503.15478 SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks]
 
* 2025-03: ColBench: [https://arxiv.org/abs/2503.15478 SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks]
 +
* 2025-04 OpenAI [https://openai.com/index/browsecomp/ BrowseComp: a benchmark for browsing agents]
  
 
===Evaluation Schemes===
 
===Evaluation Schemes===

Latest revision as of 14:25, 10 April 2025

Reviews & Perspectives

Published

Continually updating

Analysis/Opinions

Guides

AI Assistants

Components of AI Assistants

Agent Internal Workflow Management

Information Retrieval (Memory)

Contextual Memory

  • Memobase: user profile-based memory (long-term user memory for genAI) applications)

Control (tool-use, computer use, etc.)

Model Context Protocol (MCP)

Agent2Agent Protocol (A2A)

Open-source

Personalities/Personas

Specific Uses for AI Assistants

Computer Use

Software Engineering

Science Agents

See Science Agents.

Medicine

LLM-as-judge

Deep Research

Advanced Workflows

Streamline Administrative Tasks

Author Research Articles

Software Development Workflows

Several paradigms of AI-assisted coding have arisen:

  1. Manual, human driven
  2. AI-aided through chat/dialogue, where the human asks for code and then copies it into the project
    1. OpenAI ChatGPT
    2. Anthropic Claude
  3. API calls to an LLM, which generates code and inserts the file into the project
  4. LLM-integration into the IDE
    1. Copilot
    2. Qodo (Codium) & AlphaCodium (preprint, code)
    3. Cursor
    4. Codeium Windsurf (with "Cascade" AI Agent)
    5. ByteDance Trae AI
    6. Tabnine
    7. Traycer
    8. IDX: free
    9. Aide: open-source AI-native code editor (fork of VS Code)
    10. continue.dev: open-source code assistant
    11. Pear AI: open-source code editor
    12. Haystack Editor: canvas UI
    13. Onlook: for designers
    14. All Hands AI
    15. Devin 2.0 (Cognition AI)
    16. Google Firebase Studio
  5. AI-assisted IDE, where the AI generates and manages the dev environment
    1. Replit
    2. Aider (code): Pair programming on commandline
    3. Pythagora
    4. StackBlitz bolt.new
    5. Cline (formerly Claude Dev)
  6. Prompt-to-product
    1. Github Spark (demo video)
    2. Create.xyz: text-to-app, replicate product from link
    3. a0.dev: generate mobil apps (from your phone)
    4. Softgen: web app developer
    5. wrapifai: build form-based apps
    6. Lovable: web app (from text, screenshot, etc.)
    7. Vercel v0
    8. MarsX (John Rush): SaaS builder
    9. Webdraw: turn sketches into web apps
    10. Tempo Labs: build React apps
    11. Databutton: no-code software development
    12. base44: no-code dashboard apps
    13. Origin AI
  7. Semi-autonomous software engineer agents
    1. Devin (Cognition AI)
    2. Amazon Q (and CodeWhisperer)
    3. Honeycomb
    4. Claude Code

For a review of the current state of software-engineering agentic approaches, see:

Corporate AI Agent Ventures

Mundane Workflows and Capabilities

Inference-compute Reasoning

AI Assistant

Agentic Systems

Increasing AI Agent Intelligence

See: Increasing AI Intelligence

Multi-agent orchestration

Research

Organization Schemes

Societies and Communities of AI agents

Domain-specific

Research demos

Related work

Inter-agent communications

Architectures

Open Source Frameworks

Open Source Systems

Commercial Automation Frameworks

Multi-agent Handoff/Collaboration

Spreadsheet

Cloud solutions

Frameworks

Optimization

Reviews

Metrics, Benchmarks

Evaluation Schemes

Multi-agent

Agent Challenges

  • Aidan-Bench: Test creativity by having a particular LLM generate long sequence of outputs (meant to be different), and measuring how long it can go before duplications appear.
  • Pictionary: LLM suggests prompt, multiple LLMs generate outputs, LLM judges; allows raking of the generation abilities.
  • MC-bench: Request LLMs to build an elaborate structure in Minecraft; outputs can be A/B tested by human judges.

Automated Improvement

See Also

Retrieved from "http://gisaxs.com/index.php?title=AI_Agents&oldid=7541"