Difference between revisions of "Human Computer Interaction"
KevinYager (talk | contribs) (→Full Desktop GUI) |
KevinYager (talk | contribs) (→Screen Record) |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 45: | Line 45: | ||
=AI Computer Use= | =AI Computer Use= | ||
| + | ==Research== | ||
| + | * 2024-11: [https://arxiv.org/abs/2411.10323 The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use] ([https://github.com/showlab/computer_use_ootb code]) | ||
| + | * 2025-01: [https://arxiv.org/abs/2501.10893 Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments] | ||
| + | |||
==Browser== | ==Browser== | ||
* [https://github.com/mherrmann/helium Helium]: Light-weight web automation with Python; library for automating browsers (Chrome, Firefox) actions (enter values, click buttons, etc.) | * [https://github.com/mherrmann/helium Helium]: Light-weight web automation with Python; library for automating browsers (Chrome, Firefox) actions (enter values, click buttons, etc.) | ||
| Line 57: | Line 61: | ||
* [https://huggingface.co/blog/Ziyang/screenspot-pro ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use] ([https://likaixin2000.github.io/papers/ScreenSpot_Pro.pdf paper], [https://huggingface.co/datasets/likaixin/ScreenSpot-Pro huggingface], [https://gui-agent.github.io/grounding-leaderboard/ leaderboard]) | * [https://huggingface.co/blog/Ziyang/screenspot-pro ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use] ([https://likaixin2000.github.io/papers/ScreenSpot_Pro.pdf paper], [https://huggingface.co/datasets/likaixin/ScreenSpot-Pro huggingface], [https://gui-agent.github.io/grounding-leaderboard/ leaderboard]) | ||
* Microsoft [https://huggingface.co/microsoft/OmniParser-v2.0 OmniParser v2] ([https://github.com/microsoft/OmniParser/tree/master/omnitool code]) for Windows 11 VMs | * Microsoft [https://huggingface.co/microsoft/OmniParser-v2.0 OmniParser v2] ([https://github.com/microsoft/OmniParser/tree/master/omnitool code]) for Windows 11 VMs | ||
| + | |||
| + | ==Screen Record== | ||
| + | * [https://screenpi.pe/ screenpipe] ([https://github.com/mediar-ai/screenpipe github]): AI app store powered by 24/7 desktop history | ||
==Computer Use Agents== | ==Computer Use Agents== | ||
Latest revision as of 09:14, 19 February 2025
A.k.a. HCI
Contents
Smart Wearables
Pendants, etc.
- Rewind pendant
- Compass AI pendant
- Humane AI pin ($700)
- Rabbit R1 ($200)
- OpenInterpreter natural-language control fob ($100, code and schematics available)
- Limitless Pendant for AI transcription ($100)
- Iyo One earbuds ($600)
- Omi open-source wearable ($90)
Smart Glasses
- Meta Ray-Ban smart glasses
- XReal Air2
- RayNeo X2 Lite
- Digilens Argo
- Brilliant Frame smart glasses ($350)
- Based Hardware open-source AI wearables, including glasses
- Snap Spectacles AR glasses (specs)
- Meta Orion Augmented Reality glasses
- Rokid Glasses (release news)
- Even Realities G1
- INMO Air 3 (video)
- Solos AirGo V smart glasses, with vision/camera, ChatGPT integration
- Raven Resonance
UIs tailored to AI
Example products with AI-first interfaces
- Thread of examples
- granola: AI notepad for meetings.
- attio: A next-generation CRM platform that leverages AI to automate complex go-to-market tasks and enhance customer relationship management.
- rabbitholes.ai: An AI-powered platform that facilitates deep, explorative conversations on an infinite canvas, enabling users to learn faster and delve deeper into topics.
- tldraw.com: A free, instant collaborative whiteboarding tool for creating diagrams, flowcharts, and sketches with real-time collaboration.
- herostuff.com: An AI-driven marketplace that allows users to scan, price, and list items for sale quickly using AI technology.
- krea.ai: A platform that simplifies generative AI, enabling users to create and enhance images and videos for free.
- superrandom.studio/venngenn: An AI tool that generates unique and creative images based on user prompts, offering various style and environment modifiers.
- scrapybara.com/playground: An experimental AI-powered playground that allows users to interact with AI agents for various tasks, including web scraping and data extraction.
- sdk.vercel.ai: An open-source AI SDK for TypeScript that provides tools to build AI-powered products, supporting multiple AI providers and frameworks.
- midday.ai: A business management platform designed for freelancers, offering features like invoicing, time tracking, financial overviews, and an AI assistant.
- dupe.com: A platform that helps users find similar products at lower prices, aiming to provide affordable alternatives to popular items.
- Granola: The AI notepad for people in back-to-back meetings
AI Computer Use
Research
- 2024-11: The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use (code)
- 2025-01: Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments
Browser
- Helium: Light-weight web automation with Python; library for automating browsers (Chrome, Firefox) actions (enter values, click buttons, etc.)
- Browser-Use (app)
- Lightpanda Browser: open-source browser made for headless usage
- Steel: The open-source browser API for AI agents & apps (build live web agents and browser automation tools)
- stagehand: An AI web browsing framework focused on simplicity and extensibility
- Skyvern: Automate Browser-based workflows using LLMs and Computer Vision
Full Desktop GUI
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis (huggingface, paper)
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use (paper, huggingface, leaderboard)
- Microsoft OmniParser v2 (code) for Windows 11 VMs
Screen Record
- screenpipe (github): AI app store powered by 24/7 desktop history
Computer Use Agents
- Anthropic Computer Use
- OpenAI Operator
- Convergence AI Proxy (examples)
