Difference between revisions of "Human Computer Interaction"

From GISAXS
Jump to: navigation, search
(AI Computer Use)
(Browser)
 
Line 49: Line 49:
 
* [https://github.com/steel-dev/steel-browser Steel]: The open-source browser API for AI agents & apps (build live web agents and browser automation tools)
 
* [https://github.com/steel-dev/steel-browser Steel]: The open-source browser API for AI agents & apps (build live web agents and browser automation tools)
 
* [https://github.com/browserbase/stagehand stagehand]: An AI web browsing framework focused on simplicity and extensibility
 
* [https://github.com/browserbase/stagehand stagehand]: An AI web browsing framework focused on simplicity and extensibility
 +
* [https://github.com/Skyvern-AI/skyvern Skyvern]: Automate Browser-based workflows using LLMs and Computer Vision
  
 
==Full Desktop GUI==
 
==Full Desktop GUI==
 
* [https://qiushisun.github.io/OS-Genesis-Home/ OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis] ([https://huggingface.co/papers/2412.19723 huggingface], [https://arxiv.org/abs/2412.19723 paper])
 
* [https://qiushisun.github.io/OS-Genesis-Home/ OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis] ([https://huggingface.co/papers/2412.19723 huggingface], [https://arxiv.org/abs/2412.19723 paper])
 
* [https://huggingface.co/blog/Ziyang/screenspot-pro ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use] ([https://likaixin2000.github.io/papers/ScreenSpot_Pro.pdf paper], [https://huggingface.co/datasets/likaixin/ScreenSpot-Pro huggingface], [https://gui-agent.github.io/grounding-leaderboard/ leaderboard])
 
* [https://huggingface.co/blog/Ziyang/screenspot-pro ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use] ([https://likaixin2000.github.io/papers/ScreenSpot_Pro.pdf paper], [https://huggingface.co/datasets/likaixin/ScreenSpot-Pro huggingface], [https://gui-agent.github.io/grounding-leaderboard/ leaderboard])

Latest revision as of 15:23, 19 January 2025

Smart Wearables

Pendants, etc.

Smart Glasses

UIs tailored to AI

Example products with AI-first interfaces

  • Thread of examples
    • granola: AI notepad for meetings.
    • attio: A next-generation CRM platform that leverages AI to automate complex go-to-market tasks and enhance customer relationship management.
    • rabbitholes.ai: An AI-powered platform that facilitates deep, explorative conversations on an infinite canvas, enabling users to learn faster and delve deeper into topics.
    • tldraw.com: A free, instant collaborative whiteboarding tool for creating diagrams, flowcharts, and sketches with real-time collaboration.
    • herostuff.com: An AI-driven marketplace that allows users to scan, price, and list items for sale quickly using AI technology.
    • krea.ai: A platform that simplifies generative AI, enabling users to create and enhance images and videos for free.
    • superrandom.studio/venngenn: An AI tool that generates unique and creative images based on user prompts, offering various style and environment modifiers.
    • scrapybara.com/playground: An experimental AI-powered playground that allows users to interact with AI agents for various tasks, including web scraping and data extraction.
    • sdk.vercel.ai: An open-source AI SDK for TypeScript that provides tools to build AI-powered products, supporting multiple AI providers and frameworks.
    • midday.ai: A business management platform designed for freelancers, offering features like invoicing, time tracking, financial overviews, and an AI assistant.
    • dupe.com: A platform that helps users find similar products at lower prices, aiming to provide affordable alternatives to popular items.
  • Granola: The AI notepad for people in back-to-back meetings

AI Computer Use

Browser

  • Helium: Light-weight web automation with Python; library for automating browsers (Chrome, Firefox) actions (enter values, click buttons, etc.)
  • Browser-Use (app)
  • Lightpanda Browser: open-source browser made for headless usage
  • Steel: The open-source browser API for AI agents & apps (build live web agents and browser automation tools)
  • stagehand: An AI web browsing framework focused on simplicity and extensibility
  • Skyvern: Automate Browser-based workflows using LLMs and Computer Vision

Full Desktop GUI