Screenshots
Learn how to use screenshots as triggers, and capture information from them.
Last updated
Learn how to use screenshots as triggers, and capture information from them.
Last updated
In Hotkeys / Triggers, you can set a hotkey to capture a part of your screen and send it to your AI action for processing. Let's dive into the default screenshot processor and see how it works. It goes beyond the scope of this documentation to fully explain all the possibilities, rather we will explain the basics.
After you finish the Quick Start, you should be ready to try out your hotkey. In Hotkeys / Triggers you can see which action will be run when you tap the hotkey.
Under Advanced Settings you can configure more. For instance, you can specify whether the screenshot itself is sent to the AI, or if Inbox AI should try to extract the text itself using an on-device image-to-text processor and just send that. Sending the screenshot required a vision-capable AI model, and takes more effort and time, but often yields better results.
When you tap the hotkey, draw a rectangle around the text you want to capture, and when done, the image (or text from the image) will be sent to the selected action. By default this is the "Extract Tasks & Add (Apple Reminders)" action, but you can change this as you please.
After this, a notification will pop up, and your reminder will appear in Apple Reminders.
To customize the behavior, you can select a pre-built example or fully build your own AI actions. Let's look at the current default. Go to "Actions", and open the "Apple Reminders" folder in the sidebar, and select "Extract Tasks & Add" below it.
This is an Ask AI action, which uses the default AI provider. The prompt defines its behavior and it gives the AI access to two follow-up actions:
Add Apple Reminder
Send Notification
In the prompt, you can specify the behavior you are looking for. The {{ value }} placeholders in the prompt will be replaced by variables before sending them to the AI. In this case the {{ originalInput | process }} is the extracted text from the screenshot (the image itself is also attached if you enabled that option). You can also see some time information being added, as the AI does not know what time it currently is. For more information on templating, see the Templating page.
In this case the AI can choose the run the built-in "Add Apple Reminder" action, but if you explore the other examples, you can customize this to nearly anything on your Mac or on the Web.
