Flowman 0.27.0 released

Email
LinkedIn
Facebook
Twitter

We are happy to announce a fresh and exciting release of Flowman. The focus of this release is again on improving JDBC support. In particular the following important features have been implemented

  • New ‘ jdbcCommand’ target for executing arbitrary SQL statements for JDBC sinks
  • Support direct SQL statements in JDBC relations for creating tables
  • Upgrade Delta Lake to 2.0.0/2.1.0 (for Spark 3.2 and 3.3 respectively)
  • Better error messages

Of course, this release also contains a bunch of smaller bug fixes and other improvements.

Detailed Changes

  • github-232: [BUG] Column descriptions should be propagates in UNIONs
  • github-233: [BUG] Missing Hadoop dependencies for S3, Delta, etc
  • github-235: Implement new rest hook with fine control
  • github-229: A build target should not fail if Impala “COMPUTE STATS” fails
  • github-236: ‘copy’ target should not apply output schema
  • github-237: jdbcQuery relation should use fields “sql” and “file” instead of “query”
  • github-239: Allow optional SQL statement for creating jdbcTable
  • github-238: Implement new ‘jdbcCommand’ target
  • github-240: [BUG] Data quality checks in documentation should not fail on NULL values
  • github-241: Throw an error on duplicate entity definitions
  • github-220: Upgrade Delta-Lake to 2.0 / 2.1
  • github-242: Switch to Spark 3.3 as default
  • github-243: Use alternative Spark MS SQL Connector for Spark 3.3
  • github-244: Generate project HTML documentation with optional external CSS file

Download

As usual, you can download the latest version from the Download section or directly from GitHub.

Flowman at Smartclip

smartclip is a successful and growing company specialized for online video advertisement. More importantly, smartclip was one of the first companies implementing

Flowman 1.0.0 released

Flowman version 1.0.0 has finally arrived. For several years, multiple companies are using Flowman in production as a robust and reliable solution

Flowman 1.0 has landed

We are excited and proud to announce the official release of Flowman 1.0. Flowman is a tool for performing complex data transformations