Repository navigation
docs: add Apify Actor run example task page #5021
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
Jakub-Vacek
wants to merge
4
commits into
triggerdotdev:main
Choose a base branch
from
apify:docs/735-actor-run-example-task-page
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
d514885
docs(apify): add Apify Actor run example guide
daveomri e1da756
docs(apify): link the Apify actor run examples
daveomri d1e54bd
docs(apify): change wording and add utm token to console link
daveomri d40f824
docs: note that Apify Actor run examples aren't retry-safe
daveomri File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,221 @@ | ||
| --- | ||
| title: "Run Apify Actors with Trigger.dev" | ||
| sidebarTitle: "Apify Actor run" | ||
| description: "These examples demonstrate how to run an Apify Actor from a Trigger.dev task, either waiting for the results or resuming the run with a webhook." | ||
| --- | ||
|
|
||
| import ScrapingWarning from "/snippets/web-scraping-warning.mdx"; | ||
|
|
||
| ## Overview | ||
|
|
||
| [Apify](https://apify.com) is a platform for web scraping and browser automation. Its [Apify Store](https://apify.com/store) has thousands of ready-made scrapers, called Actors, for sites like Google Maps, Amazon and Instagram. Actors run on Apify's infrastructure and proxies, so your task calls the Apify API instead of scraping from Trigger.dev's servers. | ||
|
|
||
| Here are two ways to run an Actor from a Trigger.dev task: | ||
|
|
||
| - [Example 1](#example-1-run-an-actor-and-wait-for-the-results): start the Actor and wait for it to finish. Use this for Actors that finish in seconds. | ||
| - [Example 2](#example-2-run-a-long-actor-and-resume-on-a-webhook): start the Actor and pause the run until Apify calls back with a webhook. Use this for anything longer. | ||
|
|
||
| <ScrapingWarning /> | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - A project with [Trigger.dev initialized](/quick-start) | ||
| - An [Apify](https://apify.com) account and an [Apify API token](https://console.apify.com/settings/integrations?utm_source=trigger.dev&utm_medium=integrations) | ||
| - The `apify-client` package installed in your project: | ||
|
|
||
| ```bash | ||
| npm install apify-client | ||
| ``` | ||
|
|
||
| ### Set your API token | ||
|
|
||
| Add `APIFY_TOKEN` to your `.env` file for local development, and to your Trigger.dev dashboard on the Environment Variables page: | ||
|
|
||
| ```bash | ||
| APIFY_TOKEN="<your-apify-token>" | ||
| ``` | ||
|
|
||
| ### Mark apify-client as external | ||
|
|
||
| Add `apify-client` to [`external`](/config/config-file#external) in your `trigger.config.ts` file. When it's bundled, `apify-client` fails on its first API call with `TypeError: ProxyAgent is not a constructor`. | ||
|
|
||
| ```ts trigger.config.ts | ||
| import { defineConfig } from "@trigger.dev/sdk"; | ||
|
|
||
| export default defineConfig({ | ||
| project: "<project ref>", | ||
| // Your other config settings... | ||
| build: { | ||
| external: ["apify-client"], | ||
| }, | ||
| }); | ||
| ``` | ||
|
|
||
| ### Create the Apify client | ||
|
|
||
| Both examples share this client. The `x-apify-integration-platform` header tells Apify the requests come from Trigger.dev. | ||
|
|
||
| ```ts trigger/apify.ts | ||
| import { ApifyClient } from "apify-client"; | ||
|
|
||
| export const apify = new ApifyClient({ | ||
| token: process.env.APIFY_TOKEN, | ||
| requestInterceptors: [ | ||
| (config) => { | ||
| // Identifies requests from Trigger.dev tasks to Apify | ||
| config.headers = { ...config.headers, "x-apify-integration-platform": "trigger.dev" }; | ||
| return config; | ||
| }, | ||
| ], | ||
| }); | ||
| ``` | ||
|
|
||
| ## Example 1: run an Actor and wait for the results | ||
|
|
||
| This task starts an Actor, waits for it to finish and returns the items from the run's default dataset. Your run stays active and counts towards compute usage while it waits, so keep this for Actors that finish in seconds and use [Example 2](#example-2-run-a-long-actor-and-resume-on-a-webhook) for anything longer. | ||
|
|
||
| ### Key features | ||
|
|
||
| - `timeout` stops the Actor on Apify before `waitSecs` runs out, so the task never retries while an earlier Actor run is still going. | ||
| - `maxItems` caps how many results a pay-per-result Actor charges you for. Nobody is watching a background job, so a cap stops a runaway crawl. | ||
| - Throwing an error on a failed or timed-out Actor run lets your [retry settings](/errors-retrying) start the Actor again. An aborted run throws an `AbortTaskRunError` instead, because someone stopped it on purpose. | ||
| - `call()` copies the Actor's log into your run logs. Pass `log: null` in the options to turn this off. | ||
|
|
||
| ### Task code | ||
|
|
||
| ```ts trigger/apify-actor-run.ts | ||
| import { AbortTaskRunError, logger, task } from "@trigger.dev/sdk"; | ||
| import { apify } from "./apify"; | ||
|
|
||
| export const apifyActorRun = task({ | ||
| id: "apify-actor-run", | ||
| maxDuration: 600, | ||
| retry: { maxAttempts: 3 }, | ||
| run: async (payload: { actorId: string; input: Record<string, unknown> }) => { | ||
| const run = await apify.actor(payload.actorId).call(payload.input, { | ||
| timeout: 300, // Apify stops the Actor after 5 minutes, so it doesn't keep running after the task gives up | ||
| waitSecs: 360, | ||
| maxItems: 1000, // cap spend on pay-per-result Actors | ||
| }); | ||
|
|
||
| if (run.status === "ABORTED") { | ||
| // Someone stopped the Actor on purpose, so don't start it again | ||
| throw new AbortTaskRunError(`Actor run ${run.id} was aborted`); | ||
| } | ||
|
|
||
| if (run.status !== "SUCCEEDED") { | ||
| throw new Error(`Actor run ${run.id} finished with status ${run.status}`); | ||
| } | ||
|
|
||
| const { items } = await apify.dataset(run.defaultDatasetId).listItems(); | ||
|
|
||
| logger.info("Actor run finished", { runId: run.id, itemCount: items.length }); | ||
|
|
||
| return { runId: run.id, datasetId: run.defaultDatasetId, items }; | ||
| }, | ||
| }); | ||
| ``` | ||
|
|
||
| ### Testing your task | ||
|
|
||
| You can test your task by triggering it from the Trigger.dev dashboard. This payload runs [Website Content Crawler](https://apify.com/apify/website-content-crawler) on three pages: | ||
|
|
||
| ```json | ||
| { | ||
| "actorId": "apify/website-content-crawler", | ||
| "input": { | ||
| "startUrls": [{ "url": "https://docs.apify.com/academy" }], | ||
| "maxCrawlPages": 3, | ||
| "crawlerType": "cheerio", | ||
| "proxyConfiguration": { "useApifyProxy": true } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Example 2: run a long Actor and resume on a webhook | ||
|
|
||
| This task starts an Actor with an [ad-hoc webhook](https://docs.apify.com/platform/integrations/webhooks/ad-hoc-webhooks) that points at a [wait token](/wait-for-token) callback URL. The run pauses on `wait.forToken()` until the Actor finishes, and [paused runs don't count towards compute usage](/wait), however long the Actor takes. | ||
|
|
||
| ### Key features | ||
|
|
||
| - The webhook covers all four ways an Actor run can end, so a failed, aborted or timed-out Actor resumes your run with an error instead of leaving it waiting until the token times out. | ||
| - `payloadTemplate` sends only `eventType` and `resource` (the Actor run). Whatever Apify posts to `token.url` becomes the token's result, so the `ApifyWebhookPayload` type matches this template, not Apify's default webhook payload. | ||
| - The token `timeout` is longer than the Actor's `timeout`, so the Actor always finishes, and the webhook fires, before the token expires. | ||
| - Webhook event types use underscores (`ACTOR.RUN.TIMED_OUT`), but a run's `status` uses a hyphen (`TIMED-OUT`). Compare `eventType` with event types and `status` with statuses. | ||
|
|
||
| ### Task code | ||
|
|
||
| ```ts trigger/apify-long-actor-run.ts | ||
| import { AbortTaskRunError, logger, task, wait } from "@trigger.dev/sdk"; | ||
| import { apify } from "./apify"; | ||
|
|
||
| // Shape produced by the payloadTemplate below, not Apify's default webhook payload | ||
| type ApifyWebhookPayload = { | ||
| eventType: string; | ||
| resource: { id: string; status: string; defaultDatasetId: string }; | ||
| }; | ||
|
|
||
| export const apifyLongActorRun = task({ | ||
| id: "apify-long-actor-run", | ||
| retry: { maxAttempts: 3 }, | ||
| run: async (payload: { actorId: string; input: Record<string, unknown> }) => { | ||
| const token = await wait.createToken({ timeout: "1h" }); | ||
|
|
||
| const run = await apify.actor(payload.actorId).start(payload.input, { | ||
| timeout: 3000, // 50 minutes, shorter than the token timeout | ||
| maxItems: 1000, | ||
| webhooks: [ | ||
| { | ||
| eventTypes: [ | ||
| "ACTOR.RUN.SUCCEEDED", | ||
| "ACTOR.RUN.FAILED", | ||
| "ACTOR.RUN.ABORTED", | ||
| "ACTOR.RUN.TIMED_OUT", | ||
| ], | ||
| requestUrl: token.url, | ||
| payloadTemplate: `{ "eventType": {{eventType}}, "resource": {{resource}} }`, | ||
| }, | ||
| ], | ||
| }); | ||
|
|
||
| logger.info("Actor run started, waiting for webhook", { runId: run.id }); | ||
|
|
||
| const result = await wait.forToken<ApifyWebhookPayload>(token).unwrap(); | ||
|
|
||
| if (result.eventType === "ACTOR.RUN.ABORTED") { | ||
| // Someone stopped the Actor on purpose, so don't start it again | ||
| throw new AbortTaskRunError(`Actor run ${result.resource.id} was aborted`); | ||
| } | ||
|
|
||
| if (result.eventType !== "ACTOR.RUN.SUCCEEDED") { | ||
| throw new Error(`Actor run ${result.resource.id} finished as ${result.eventType}`); | ||
| } | ||
|
|
||
| const { items } = await apify.dataset(result.resource.defaultDatasetId).listItems(); | ||
|
|
||
| logger.info("Actor run finished", { runId: result.resource.id, itemCount: items.length }); | ||
|
|
||
| return { runId: result.resource.id, datasetId: result.resource.defaultDatasetId, items }; | ||
| }, | ||
| }); | ||
| ``` | ||
|
|
||
| ### Testing your task | ||
|
|
||
| Trigger the task from the Trigger.dev dashboard with the same payload as Example 1. The run trace shows `wait.forToken()` covering the time the Actor was running, and the run resumes once Apify sends the webhook. | ||
|
|
||
| <Note> | ||
| Cancelling a Trigger.dev run doesn't stop the Actor run it started. Abort the Actor run in | ||
| [Apify Console](https://console.apify.com?utm_source=trigger.dev&utm_medium=integrations) or with `apify.run(runId).abort()`. | ||
| </Note> | ||
|
|
||
| <Note> | ||
| These examples aren't retry-safe: retries always start a new Actor run. In production, store the Actor run ID and resume that run on retry rather than starting over. | ||
| </Note> | ||
|
|
||
| ## Learn more | ||
|
|
||
| - [Wait for token](/wait-for-token) | ||
| - [Apify ad-hoc webhooks](https://docs.apify.com/platform/integrations/webhooks/ad-hoc-webhooks) | ||
| - [Apify JavaScript client](https://docs.apify.com/api/client/js/) | ||
| - [Apify Store](https://apify.com/store) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Make Actor starts retry-safe. Both tasks can start a second Actor when an attempt fails after the first Actor starts. In the short task,
waitSecscan expire while the Actor remains READY or RUNNING. In the long task, the webhook wait or dataset request can fail afterstart()succeeds. Persist the Actor run ID across attempts and resume or reconcile that run before starting another. (docs.apify.com)docs/guides/examples/apify-actor-run.mdx#L95-L95: resume or stop an unfinished Actor before a retry can call it again.docs/guides/examples/apify-actor-run.mdx#L164-L164: reuse the previously started Actor run when a later step fails.📍 Affects 1 file
docs/guides/examples/apify-actor-run.mdx#L95-L95(this comment)docs/guides/examples/apify-actor-run.mdx#L164-L164There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CC @daveomri this fells bit nitpicky - maybe we could left comment in code stating that this is example not production code
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, these examples aren't production code, but rather an example.
Added a note stating this fact (with proposed fix).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.