Integration · Data transformation
Apache Spark + OrchestrAI
Catalog exported 2026-09-02 · Apache Spark website
Submit and manage Apache Spark jobs from chat, and create, optimize, and vacuum Delta Lake and Iceberg tables.
OrchestrAI exposes 8 Apache Spark operations: 2 are low-risk (read-only or low-impact), and 6 create or modify resources and run only after you confirm the plan. 4 of them also carry a step-level approval gate.
What teams use it for
Data engineers use this to launch a spark-submit for a backfill job, check whether an application is still running, and kill one that has been stuck for hours, all without shelling into the cluster. On the table side, OrchestrAI creates Delta Lake and Apache Iceberg tables, compacts a Delta table with OPTIMIZE, and runs VACUUM, which is rated high risk with confirmation because it permanently removes files older than the retention window. It does not fetch driver or executor logs, show stage-level metrics, or read Delta table history, so diagnosing a slow job still means opening the Spark UI.
Every Apache Spark operation, with its risk level
| Operation | What it does | Risk | Step-level approval |
|---|---|---|---|
Get Spark Application Status |
Get Spark application status | Low risk | No |
List Spark Applications |
List running Spark applications | Low risk | No |
Create Delta Table |
Create Delta Lake table | Creates resources | Yes |
Create Iceberg Table |
Create Apache Iceberg table | Creates resources | Yes |
Kill Spark Application |
Kill Spark application | Creates resources | Yes |
Optimize Delta Table |
Optimize Delta Lake table | Creates resources | No |
Submit Spark Job |
Submit Spark job via spark-submit | Creates resources | No |
Vacuum Delta Table |
Vacuum Delta Lake table to remove old files | Modifies existing | Yes |
Risk tiers come from the catalog: low is read-only or low-impact, medium creates resources and is reversible, high modifies existing resources, destructive may lose data. Every plan that creates or changes resources is shown with its cost estimate and waits for your confirmation. Operations marked with a step-level approval pause again on their own step. Destructive operations require a typed risk phrase.
What you connect
A Apache Spark credential (stored as spark).
Connected-service tokens are envelope-encrypted with a per-record key wrapped by a cloud KMS.
Prompts that work
- Submit the nightly_dedupe.py job to the Spark cluster with 8 executors and 16g executor memory
- List the running Spark applications and kill the one named backfill-2026-08 if it is still going
- Optimize the Delta table sales.transactions and then vacuum it with a 168 hour retention
Before anything runs
Every mutation shows its plan, cost estimate, and blast radius, then waits for your confirmation. Destructive operations require a typed risk phrase. Credentials are minted per run through OIDC federation and discarded afterward; nothing you create here is invisible later, because every resource lands in the desired-state ledger where drift is detected and can be converged. Details on the security page.
Frequently asked questions
- Can OrchestrAI vacuum a Delta Lake table?
- Yes, delta_vacuum removes files past the retention window and is rated high risk with confirmation, because the files cannot be recovered afterwards. delta_optimize, which compacts small files, is medium risk.
- Does OrchestrAI support Apache Iceberg as well as Delta Lake?
- Table creation is supported for both formats and waits for confirmation. Optimize and vacuum operations exist only for Delta Lake.
- How does OrchestrAI authenticate to Apache Spark?
- You add a Apache Spark credential once in the connections screen. It is envelope-encrypted with a per-record key wrapped by a cloud KMS and is only decrypted inside the run that needs it.
Related integrations
Try it on your own account
Connect your cloud read-only and see your resources, drift, and costs before anything runs. $5 minimum to start. Unused credits refunded in your first 14 days.
Unused credits refunded in your first 14 days.