Flowman 0.27.0 released

We are happy to announce a fresh and exciting release of Flowman. The focus of this release is again on improving JDBC support. In particular the following important features have been implemented

• New ‘ jdbcCommand’ target for executing arbitrary SQL statements for JDBC sinks
• Support direct SQL statements in JDBC relations for creating tables
• Upgrade Delta Lake to 2.0.0/2.1.0 (for Spark 3.2 and 3.3 respectively)
• Better error messages

Of course, this release also contains a bunch of smaller bug fixes and other improvements.

Detailed Changes

• github-232: [BUG] Column descriptions should be propagates in UNIONs
• github-233: [BUG] Missing Hadoop dependencies for S3, Delta, etc
• github-235: Implement new rest hook with fine control
• github-229: A build target should not fail if Impala “COMPUTE STATS” fails
• github-236: ‘copy’ target should not apply output schema
• github-237: jdbcQuery relation should use fields “sql” and “file” instead of “query”
• github-239: Allow optional SQL statement for creating jdbcTable
• github-238: Implement new ‘jdbcCommand’ target
• github-240: [BUG] Data quality checks in documentation should not fail on NULL values
• github-241: Throw an error on duplicate entity definitions
• github-220: Upgrade Delta-Lake to 2.0 / 2.1
• github-242: Switch to Spark 3.3 as default
• github-243: Use alternative Spark MS SQL Connector for Spark 3.3
• github-244: Generate project HTML documentation with optional external CSS file

Download

As usual, you can download the latest version from the Download section or directly from GitHub.