Open Interface

Control Your Computer Using LLMs

Open Interface

Self-drives your computer by sending your requests to an LLM backend (GPT-4o, Gemini, etc) to figure out the required steps.
Automatically executes these steps by simulating keyboard and mouse input.
Course-corrects by sending the LLM backend updated screenshots of the progress as needed.

<div align="center"> <h4>Full Autopilot for All Computers Using LLMs</h4>

GitHub code size in bytes
GitHub Repo stars
GitHub

</div>

<ins>Demo</ins> 💻

"Solve Today's Wordle" 
 
clipped, 2x

<details> <summary><a href="https://github.com/AmberSahdev/Open-Interface/blob/main/MEDIA.md#demos">More Demos</a></summary> <ul> <li> "Make me a meal plan in Google Docs" <img src="assets/meal_plan_demo_2x.gif" style="margin: 5px; border-radius: 10px;"> </li> <li> "Write a Web App" <img src="assets/code_web_app_demo_2x.gif" style="margin: 5px; border-radius: 10px;"> </li> </ul> </details> <hr>

<ins>Install</ins> 💽

<details> <summary><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Apple_Computer_Logo_rainbow.svg/960px-Apple_Computer_Logo_rainbow.svg.png?20250629104313" alt="MacOS Logo" width="13" height="15"> MacOS</summary> <ul> <li>Download the MacOS binary from the latest <a href="https://github.com/AmberSahdev/Open-Interface/releases/latest">release</a>.</li> <li>Unzip the file and move Open Interface to the Applications Folder. <img src="assets/macos_unzip_move_to_applications.png" width="350" style="border-radius: 10px; border: 3px solid black;"> </li> </ul> <details> <summary>Apple Silicon M-Series Macs</summary> <ul> <li> Open Interface will ask you for Accessibility access to operate your keyboard and mouse for you, and Screen Recording access to take screenshots to assess its progress. </li> <li> In case it doesn't, manually add these permission via System Settings -> Privacy and Security <img src="assets/mac_m3_accessibility.png" width="400" style="margin: 5px; border-radius: 10px; border: 3px solid black;"> <img src="assets/mac_m3_screenrecording.png" width="400" style="margin: 5px; border-radius: 10px; border: 3px solid black;"> </li> </ul> </details> <details> <summary>Intel Macs</summary> <ul> <li> Launch the app from the Applications folder. You might face the standard Mac "Open Interface cannot be opened" error. <img src="assets/macos_unverified_developer.png" width="200" style="border-radius: 10px; border: 3px solid black;"> In that case, press <ins>"Cancel"</ins>. Then go to System Preferences -> Security and Privacy -> Open Anyway. <img src="assets/macos_system_preferences.png" width="100" style="border-radius: 10px; border: 3px solid black;">   <img src="assets/macos_security.png" width="100" style="border-radius: 10px; border: 3px solid black;">   <img src="assets/macos_open_anyway.png" width="400" style="border-radius: 10px; border: 3px solid black;"> </li> <li> Open Interface will also need Accessibility access to operate your keyboard and mouse for you, and Screen Recording access to take screenshots to assess its progress. <img src="assets/macos_accessibility.png" width="400" style="margin: 5px; border-radius: 10px; border: 3px solid black;"> <img src="assets/macos_screen_recording.png" width="400" style="margin: 5px; border-radius: 10px; border: 3px solid black;"> </li> </ul> </details> <ul> <li>Lastly, checkout the <a href="#setup">Setup</a> section to connect Open Interface to LLMs (OpenAI GPT-4V)</li> </ul> </details> <details> <summary><img src="https://upload.wikimedia.org/wikipedia/commons/a/af/Tux.png" alt="Linux Logo" width="18" height="18"> Linux</summary> <ul> <li>Linux binary has been tested on Ubuntu 20.04 so far.</li> <li>Download the Linux zip file from the latest <a href="https://github.com/AmberSahdev/Open-Interface/releases/latest">release</a>.</li> <li> Extract the executable and checkout the <a href="https://github.com/AmberSahdev/Open-Interface?tab=readme-ov-file#setup">Setup</a> section to connect Open Interface to LLMs, such as OpenAI GPT-4V.</li> </ul> </details> <details> <summary><img src="https://upload.wikimedia.org/wikipedia/commons/5/5f/Windows_logo_-_2012.svg" alt="Linux Logo" width="15" height="15"> Windows</summary> <ul> <li>Windows binary has been tested on Windows 10.</li> <li>Download the Windows zip file from the latest <a href="https://github.com/AmberSahdev/Open-Interface/releases/latest">release</a>.</li> <li>Unzip the folder, move the exe to the desired location, double click to open, and voila.</li> <li>Checkout the <a href="https://github.com/AmberSahdev/Open-Interface?tab=readme-ov-file#setup">Setup</a> section to connect Open Interface to LLMs (OpenAI GPT-4V)</li> </ul> </details> <details> <summary><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Python-logo-notext.svg/120px-Python-logo-notext.svg.png?20250701090410" alt="Python Logo" width="15" height="15"> Run as a Script</summary> <ul> <li>Clone the repo <code>git clone https://github.com/AmberSahdev/Open-Interface.git</code></li> <li>Enter the directory <code>cd Open-Interface</code></li> <li>Optionally use a Python virtual environment <ul> <li>Note: pyenv handles tkinter installation weirdly so you may have to debug for your own system yourself.</li> <li><code>pyenv local 3.12.2</code></li> <li><code>python -m venv .venv</code></li> <li><code>source .venv/bin/activate</code></li> </ul> </li> <li>Install dependencies <code>pip install -r requirements.txt</code></li> <li>Run the app using <code>python app/app.py</code></li> </ul> </details>

<ins id="setup">Setup</ins> 🛠️

<details> <summary>Set up the OpenAI API key</summary>

Get your OpenAI API key
- Open Interface needs access to GPT-4o to perform user requests. GPT-4o keys can be downloaded from your OpenAI account at platform.openai.com/settings/organization/api-keys.
- Follow the steps here to add balance to your OpenAI account. To unlock GPT-4o a minimum payment of $5 is needed.
- More info
Save the API key in Open Interface settings
- In Open Interface, go to the Settings menu on the top right and enter the key you received from OpenAI into the text field like so: 
<picture> <img src="assets/set_openai_api_key.png" align="middle" alt="Set API key in settings" width="400"> </picture>
After setting the API key for the first time you'll need to restart the app.

</details> <details> <summary>Set up the Google Gemini API key</summary>

Go to Settings -> Advanced Settings and select the Gemini model you wish to use.
Get your Google Gemini API key from https://aistudio.google.com/app/apikey.
Save the API key in Open Interface settings.
Save the settings and restart the app.

</details> <details> <summary>Optional: Setup a Custom LLM</summary>

Open Interface supports using other OpenAI API style LLMs (such as Llava) as a backend and can be configured easily in the Advanced Settings window.
Enter the custom base url and model name in the Advanced Settings window and the API key in the Settings window as needed.
NOTE - If you're using Llama:
- You may need to enter a random string like "xxx" in the API key input box.
- You may need to append /v1/ to the base URL.
 
 <picture>
 <img src="assets/advanced_settings.png" align="middle" alt="Set API key in settings" width="400">
 </picture>
If your LLM does not support an OpenAI style API, you can use a library like this to convert it to one.
You will need to restart the app after these changes.

</details> <hr>

<ins>Stuff It’s Error-Prone At, For Now</ins> 😬

Accurate spatial-reasoning and hence clicking buttons.
Keeping track of itself in tabular contexts, like Excel and Google Sheets, for similar reasons as stated above.
Navigating complex GUI-rich applications like Counter-Strike, Spotify, Garage Band, etc due to heavy reliance on cursor actions.

<ins>The Future</ins> 🔮

(with better models trained on video walkthroughs like Youtube tutorials)

"Create a couple of bass samples for me in Garage Band for my latest project."
"Read this design document for a new feature, edit the code on Github, and submit it for review."
"Find my friends' music taste from Spotify and create a party playlist for tonight's event."
"Take the pictures from my Tahoe trip and make a White Lotus type montage in iMovie."

<ins>Notes</ins> 📝

Cost Estimation: $0.0005 - $0.002 per LLM request depending on the model used. 
(User requests can require between two to a few dozen LLM backend calls depending on the request's complexity.)
You can interrupt the app anytime by pressing the Stop button, or by dragging your cursor to any of the screen corners.
Open Interface can only see your primary display when using multiple monitors. Therefore, if the cursor/focus is on a secondary screen, it might keep retrying the same actions as it is unable to see its progress.

<hr>

<ins>System Diagram</ins> 🖼️

+----------------------------------------------------+
| App                                                |
|                                                    |
|    +-------+                                       |
|    |  GUI  |                                       |
|    +-------+                                       |
|        ^                                           |
|        |                                           |
|        v                                           |
|  +-----------+  (Screenshot + Goal)  +-----------+ |
|  |           | --------------------> |           | |
|  |    Core   |                       |    LLM    | |
|  |           | <-------------------- |  (GPT-4o) | |
|  +-----------+    (Instructions)     +-----------+ |
|        |                                           |
|        v                                           |
|  +-------------+                                   |
|  | Interpreter |                                   |
|  +-------------+                                   |
|        |                                           |
|        v                                           |
|  +-------------+                                   |
|  |   Executer  |                                   |
|  +-------------+                                   |
+----------------------------------------------------+