Flow through data pipelines with Flowman

Flowman is a powerful and open source data build tool powered by Apache Spark that follows a declarative approach to simplify the act of writing complex ETL, ELT and data transformation applications. The strong focus on transformation and schema management reduces your development efforts for creating robust data pipelines.

Being built on top of Apache Spark, Flowman can be run as a standalone application but can also scale by using compute clusters (Hadoop & Kubernetes) to process any amounts of data.

Focus on business logic instead of Spark boilerplate code!

Transform data with Flowman and use it for BI, ML or Analyitcs

How you will benefit from Flowman

Flowman blog and change log

Flowman 0.30.0 released!

January 4, 2023

The new version 0.30.0 of Flowman has been released with better control over execution plans and with a new “observe” mapping to capture data dependent metrics as records flow through the system. Spark 3.2 is now officially supported on Cloudera CDP 7.1.

Flowman 0.29.0 released!

November 8, 2022

A new version of Flowman has been released containing some new features like a new “iterativeSql” mapping and a new Maven module “flowman-spark-dependencies”, which simplifies creating and building Flowman projects with Maven.

Flowman 0.28.0 released!

October 7, 2022

A new version of Flowman has been released containing minor but important improvements for working with MariaDB, MySQL and SQL Server / Azure SQL databases as sinks. Moreover the workflow for creating a new Flowman project has been simplified by adding a new Maven archetype.

Flowman 0.27.0 released!

September 9, 2022

The new version 0.27.0 of Flowman has been released. Among many improvements and new features, this release takes the support for working with JDBC data sources and sinks even further. You can now execute arbitrary SQL commands as part of the build process to provide a way to handle database specific features.

Flowman 0.26.0 released!

July 27, 2022

This latest release of Flowman contains a lot of work with a strong focus on improving working with JDBC targets like MariaDB/MySQL, Postgres, MS SQL Server, Azure SQL and Oracle. For example, column collations and comments are now correctly propagated into relational databases, changing the primary key is now supported and much more.

Also Spark 3.3 is now officially supported, albeit not much tested so far. Moreover many small bug fixes and enhancements help to make Flowman more robust and versatile.

Flowman 0.25.0 released!

June 31, 2022

This latest release contains a couple of smaller changes and fixes plus a new relation for creating and managing views in SQL databases, which are accessed via JDBC. This addition strengthens Flowmans position in environments where data resides in classical relational databases instead of HDFS or object storages.

Flowman 0.24.0 released!

Mai 4, 2022

The latest Flowman release now provides a YAML/JSON to enable syntax highlighting and auto complete in code editors like IntelliJ and Visual Studio Code.

Projects delivered with Flowman

Online Adversting

Online advertising produces huge amounts of data on a daily basis. In order to provide meaningful insights, all this data needs to be integrated and aggregated to meaningful dimensions. Flowman has been implemented successfully to create multiple pre-aggregated data marts. By relying on the declarative specification, business experts can be easily involved for reviewing.

Financial Services

Flowman has been successfully implemented in a microservice project in the financial service industry. The project uses Kafka for intra-service communication and Flowman is used to collect and process relevant messages directly from Kafka without the need to connect all services individually.

Customer facing reporting

The art of making sense of millions of detailed records from multiple source systems by providing a high level and holistic view is at the core of customer facing reporting in B2B scenarios. Flowman is the right tool for integrating different data sources, applying complex business logic and storing aggregated tables into your reporting backend.

Cookie	Duration	Description
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
CookieLawInfoConsent	1 year	Records the default button state of the corresponding category & the status of CCPA. It works only in coordination with the primary cookie.
elementor	never	This cookie is used by the website's WordPress theme. It allows the website owner to implement or change the website's content in real-time.
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.