Frequently asked questions about Flowman (FAQ)

• Files either on local file system or distributed storage (HDFS) or blob storage (S3, ABS, etc)
• Hive tables and views
• Relational databases (MySQL, MariaDB, MS SQL Server, PostgreSQL, …)

• Plain text
• CSV
• Fixed width format
• Hadoop Sequence files
• JSON files
• Avro files
• Parquet
• ORC
• Delta Lake

Flowman supports many relational databases via JDBC connectivity.

• MySQL
• MariaDB
• PostgreSQL
• Oracle
• MS SQL Server
• Azure SQL

New databases can be implemented on request without much effort.

Actually Flowman is neither a replacement for schedulers like Apache Airflow or Oozie, nor does it contain a job scheduler which automatically starts the execution of jobs at specific times. Since job scheduling is an overarching topic which is required to run many different tools, this is not a shortcoming of Flowman itself, but rather a design decision to exclude this feature since other excellent tools already exist.

This means you can use any existing scheduler which supports starting a bash script (since this is what the Flowman executables essentially are), so for example Oozie or Airflow work just fine.

Since Flowman version 0.30.0, AWS EMR is fully supported as a deployment target. You can either deploy Flowman to the master node and access it via ssh, or you can deploy Flowman as a step function of your EMR cluster. Flowman also supports EMR Serverless, but no interactive console access is possible.

Since Flowman version 1.0.0, Flowman supports Azure Synapse Spark as a deployment target. This means that you can create a fat jar containing all required Flowman libraries and your project, copy the jar file to Azure Blob Storage and then run Flowman as a Spark job inside Azure Synapse.