Built for scale

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management. It enables data analysts and engineers to transform their data by writing simple SQL queries and YAML configuration while Optimus handles dependency management, scheduling and all other aspects of running transformation jobs at scale.

Zero dependency

Optimus is written in Go and compiles into a single binary with no external dependencies, and requires a very minimal memory footprint.

Warehouse management

Optimus allows you to create and manage your data warehouse tables and views through YAML based configuration.

Extensible

With the ease of plugin development build your own plugins. Optimus support Python transformation and allows for writing custom plugins.

CLI

Optimus comes with a CLI which allows you to interact with workflows effectively. You can create, run, replay jobs and more.

Proven

Battle tested at large scale across multiple companies. Largest deployment runs thousands of workflows on multiple data sources.

Workflows

Optimus provides industry-proven workflows using git and REST/GRPC based specification management for data warehouse management.

Key features

Optimus is an ETL orchestration tool that helps manage warehouse resources and schedule transformation over cron interval. Warehouses like Bigquery can be used to create, update, read, delete different types of resources(dataset/table/standard view). Similarly, jobs can be SQL transformations taking inputs from single/multiple source tables executing over fixed schedule interval. Optimus was made from start to be extensible, which is, adding support of different kind of warehouses, transformers can be done easily.

Scheduling

Optimus provides an easy way to schedule your SQL transformation through a YAML based configuration.

Dependency resolution

Optimus parses your data transformation queries and builds a dependency graphs automaticaly instead of users defining it in DAGs.

Dry runs

Before SQL query is scheduled for transformation, during deployment query will be dry-run to make sure it passes basic sanity checks.

Powerful templating

Optimus provides query compile time templating with variables, loop, if statements, macros, etc for allowing users to write complex tranformation logic.

Cross tenant dependency

Optimus is a multi-tenant service. With more than two tenants registered Optimus can resolve cross tenant dependencies automatically.

Hooks

Optimus provides hooks for post tranformation logic to extend the functionality of the transformation. e,g. You can sink BigQuery tables to Kafka.

Workflow

With Optimus data teams work directly with the data warehouse and data catalogs. Optimus provides a set of workflows which can be used to build data transformation pipelines, reporting, operational, machine learning workflows.

Develop

Write your specifications in git using Optimus CLI or use Optimus APIs to prgramtically submit specifications through SDK.

Test

Test your workflows prior to production with linting, dry runs and local execution from your machines.

Deploy

Deploy your workflows safely with Optimus CLI and APIs to production.