Pipeline authoring
A visual DAG editor and a CLI, both reading and writing the same .tf files. Transforms in SQL or PySpark.
Self-hosted lakehouse platform
The lakehouse
you actually own.
Write transforms in SQL, or PySpark when SQL isn’t enough. Clavesa compiles them to the cheapest AWS compute that fits: Lambda, Fargate, or EMR Serverless. The whole stack is Terraform in your repo.
Built on AWS-native primitives. No Glue Jobs. No Databricks runtime. No Snowflake credits.
$ clavesa workspace init demo
building runner image ......... ✓
$ clavesa ui
serving http://localhost:8080 ... ✓ ready
Laptop to a working catalog in two commands. No AWS required.
01 What it is
Everything one engineer needs to run a production lakehouse on AWS.
A single Go binary. Author pipelines, run distributed Spark, land every output as a queryable table, and watch the whole thing, all from one tool that reads and writes the Terraform in your repo.
Distributed execution
One PySpark runtime, identical on your laptop and on Lambda, Fargate, or EMR Serverless.
Data lakehouse
Every transform output is a Delta table in Glue Data Catalog, queryable from Athena with no DDL.
Observability
Run history, lineage, freshness SLAs, and SQL-driven dashboards over the same catalog as your data.
Local–cloud parity
Every capability works against deployed pipelines and local-only ones alike. Develop offline; deploy when ready.
No premium services in the data path.
Step Functions, Lambda, S3, and Glue Data Catalog do the work. You pay AWS list price and nothing else.
02 How it works
One engine. Your laptop, then your account.
The same PySpark runtime runs locally and in the cloud, so the SQL you test on a 3-million-row sample produces the same Delta table when it deploys. You pick the compute target per transform, in the same Terraform that defines the pipeline.
localyour laptop
→
lambdasmall filters
→
fargate100M-row joins
→
emr-serverlessmulti-TB shuffles
EMR Serverless runs a multi-terabyte shuffle at roughly 5× cheaper than Glue at the same scale.
SELECT
payment_type,
COUNT(*) AS trips,
ROUND(SUM(total_amount), 2) AS revenue,
ROUND(AVG(tip_amount / NULLIF(fare_amount, 0)) * 100, 1) AS avg_tip_pct
FROM trips
GROUP BY payment_type
ORDER BY revenue DESCWrite the transform. Ship it.
That runs on the Spark runner locally and lands a Delta table you can browse in the catalog. When you want it in the cloud, one command applies the infrastructure and every pipeline in it.
$ AWS_PROFILE=prod clavesa deploy
workspace ..... ✓
pipeline demo . ✓ appliedNothing is centrally hosted. Nothing leaves your account.
03 Why it exists
A fourth shape.
There are three usual ways to run a data platform, each with a real tradeoff. Clavesa is the one that keeps the ergonomics without the bill.
Hosted SaaS
Fivetran, dbt Cloud, Airbyte Cloud
Fast to start, but your data leaves your account, you pay per row, and nothing runs offline.
Hosted runtime
Databricks, Snowflake, Dagster Cloud
Powerful, but you pay DBU- or credit-hour rates for Spark wrapped in a vendor.
Roll your own
Airflow + dbt + Glue
Cheap at scale, expensive in engineer-weeks, and prone to local-vs-production drift.
Clavesλ
a platform you own
The authoring ergonomics of a hosted runtime, with the cost structure of rolling your own.
04 Built for the way you work
SQL-first, PySpark when SQL isn’t enough
Analytics engineers write SQL and drop to PySpark only for the transforms that need it. One language for most of the warehouse, one runtime for the rest.
Right-sized backend per transform
A small filter runs on Lambda; a 100M-row join on Fargate; a multi-TB shuffle on EMR Serverless. You choose, in the Terraform that defines the pipeline.
One engine, local and cloud
PySpark runs on Lambda, on EMR Serverless, and on your laptop. The same SQL produces the same Delta table in every target.
Observability without a second stack
Run history, lineage, and freshness live as Delta tables in the same catalog as your data. Query them with the same SQL:
SELECT * FROM clavesa_<pipeline>.runs
WHERE status = 'FAILED'05 Get started
Two commands to a working dashboard.
Runs fully offline on macOS or Linux. You need Docker for the Spark runner; add Terraform and AWS credentials only when you want to deploy.
$ brew install --cask vesahyp/clavesa/clavesa
$ clavesa version$ clavesa workspace init demo
$ clavesa ui # opens the Catalog