Dremio

Connect Dremio to the S3 bucket.

Format the Parquet folder as a physical table.

Create a Virtual Data Set (VDS) to join data with a SQL database.

Enable 'Aggregate Reflection' on the VDS.

Connect Tableau to Dremio via native connector to enjoy <1s dashboard refreshes.

Use Nessie to create a new branch named 'experiment_v1'.

Modify the data on the 'experiment_v1' branch to test feature engineering.

Point the ML model training script to the new branch.

Verify results against the 'main' production branch using a simple SQL switch.

Delete the branch when finished without impacting the master table.

Add both S3 and ADLS as sources in Dremio.

Write a standard SQL JOIN query spanning both sources.

Create a VDS representing the federated view.

Apply row-level security to ensure users only see regional data.

Expose the federated view to the corporate reporting tool.

Offload historical data to S3 in Iceberg format.

Map the Iceberg tables in Dremio to match the warehouse schema.

Redirect BI queries from the warehouse to Dremio.

Use Dremio's SQL engine to maintain the same syntax for end-users.

Decommission the expensive legacy warehouse storage.

Enable Arrow Flight endpoint in Dremio.

Use the dremio-flight-python library to connect.

Execute a query to retrieve training data.

Stream data directly into Arrow buffers for immediate GPU processing.

Reduce data loading time from hours to minutes.