Difference between revisions of "Human Computer Interaction"
KevinYager (talk | contribs) (→Browser) |
KevinYager (talk | contribs) (→Full Desktop GUI) |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 74: | Line 74: | ||
* [https://github.com/camel-ai/owl OWL (Optimized Workforce Learning)]: General Multi-Agent Assistance in Real-World Task Automation | * [https://github.com/camel-ai/owl OWL (Optimized Workforce Learning)]: General Multi-Agent Assistance in Real-World Task Automation | ||
* OpenAI [https://platform.openai.com/docs/api-reference/responses responses API] and [https://platform.openai.com/docs/guides/agents agents SDK] | * OpenAI [https://platform.openai.com/docs/api-reference/responses responses API] and [https://platform.openai.com/docs/guides/agents agents SDK] | ||
| + | * [https://github.com/bytebot-ai/bytebot?tab=readme-ov-file bytebot]: The computer use container | ||
==Screen Record== | ==Screen Record== | ||
| Line 82: | Line 83: | ||
* OpenAI [https://openai.com/index/introducing-operator/ Operator] | * OpenAI [https://openai.com/index/introducing-operator/ Operator] | ||
* [https://convergence.ai/ Convergence AI] [https://proxy.convergence.ai/ Proxy] ([https://x.com/ai_for_success/status/1883824921396322493 examples]) | * [https://convergence.ai/ Convergence AI] [https://proxy.convergence.ai/ Proxy] ([https://x.com/ai_for_success/status/1883824921396322493 examples]) | ||
| + | * [https://computer-agent.ai/ screenpipe Computer Agent] | ||
| + | * [https://generalagents.com/ General Agents Co.] [https://generalagents.com/ace/ Ace] | ||
Latest revision as of 13:26, 12 April 2025
A.k.a. HCI
Contents
Smart Wearables
Pendants, etc.
- Rewind pendant
- Compass AI pendant
- Humane AI pin ($700)
- Rabbit R1 ($200)
- OpenInterpreter natural-language control fob ($100, code and schematics available)
- Limitless Pendant for AI transcription ($100)
- Iyo One earbuds ($600)
- Omi open-source wearable ($90)
Smart Glasses
- Meta Ray-Ban smart glasses
- XReal Air2
- RayNeo X2 Lite
- Digilens Argo
- Brilliant Frame smart glasses ($350)
- Based Hardware open-source AI wearables, including glasses
- Snap Spectacles AR glasses (specs)
- Meta Orion Augmented Reality glasses
- Rokid Glasses (release news)
- Even Realities G1
- INMO Air 3 (video)
- Solos AirGo V smart glasses, with vision/camera, ChatGPT integration
- Raven Resonance
- Halliday ($430)
VRs
UIs tailored to AI
Example products with AI-first interfaces
- Thread of examples
- granola: AI notepad for meetings.
- attio: A next-generation CRM platform that leverages AI to automate complex go-to-market tasks and enhance customer relationship management.
- rabbitholes.ai: An AI-powered platform that facilitates deep, explorative conversations on an infinite canvas, enabling users to learn faster and delve deeper into topics.
- tldraw.com: A free, instant collaborative whiteboarding tool for creating diagrams, flowcharts, and sketches with real-time collaboration.
- herostuff.com: An AI-driven marketplace that allows users to scan, price, and list items for sale quickly using AI technology.
- krea.ai: A platform that simplifies generative AI, enabling users to create and enhance images and videos for free.
- superrandom.studio/venngenn: An AI tool that generates unique and creative images based on user prompts, offering various style and environment modifiers.
- scrapybara.com/playground: An experimental AI-powered playground that allows users to interact with AI agents for various tasks, including web scraping and data extraction.
- sdk.vercel.ai: An open-source AI SDK for TypeScript that provides tools to build AI-powered products, supporting multiple AI providers and frameworks.
- midday.ai: A business management platform designed for freelancers, offering features like invoicing, time tracking, financial overviews, and an AI assistant.
- dupe.com: A platform that helps users find similar products at lower prices, aiming to provide affordable alternatives to popular items.
- Granola: The AI notepad for people in back-to-back meetings
AI Computer Use
Research
- 2024-11: The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use (code)
- 2025-01: Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments
- 2025-02: PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC (video, code)
Browser
- Helium: Light-weight web automation with Python; library for automating browsers (Chrome, Firefox) actions (enter values, click buttons, etc.)
- Browser-Use (app)
- Lightpanda Browser: open-source browser made for headless usage
- Steel: The open-source browser API for AI agents & apps (build live web agents and browser automation tools)
- stagehand: An AI web browsing framework focused on simplicity and extensibility
- Skyvern: Automate Browser-based workflows using LLMs and Computer Vision
- Amazon Nova Act
- Browserable: Open source browser automation library for AI agents
Full Desktop GUI
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis (huggingface, paper)
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use (paper, huggingface, leaderboard)
- Microsoft OmniParser v2 (code) for Windows 11 VMs
- ByteDance UI-TARS Desktop: GUI agent for controlling computers with natural language
- Manus AI
- LangManus: open-source agent (based on LangChain and LangGraph)
- OWL (Optimized Workforce Learning): General Multi-Agent Assistance in Real-World Task Automation
- OpenAI responses API and agents SDK
- bytebot: The computer use container
Screen Record
- screenpipe (github): AI app store powered by 24/7 desktop history
Computer Use Agents
- Anthropic Computer Use
- OpenAI Operator
- Convergence AI Proxy (examples)
- screenpipe Computer Agent
- General Agents Co. Ace
