Tag: Data Engineering
All the articles with the tag "Data Engineering".
- 4 MIN READ•Sep 16, 2025
The Endgame – Building an Autonomous Optimization Pipeline for Apache Iceberg
Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools for a self-healing lakehouse.
Apache IcebergAutomationCompaction - 3 MIN READ•Sep 9, 2025
Managing Large-Scale Optimizations – Parallelism, Checkpointing, and Fail Recovery
Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability and performance.
Apache IcebergOptimizationCompaction - 9 MIN READ•Sep 5, 2025
Unlocking the Power of Agentic AI with Apache Iceberg and Dremio
Unlocking the Power of Agentic AI with Apache Iceberg and Dremio
Data LakehouseData EngineeringAgentic AI - 4 MIN READ•Sep 2, 2025
Hidden Pitfalls – Compaction and Partition Evolution in Apache Iceberg
Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction performance. Learn how to handle it effectively.
Apache IcebergPartition EvolutionTable Optimization - 4 MIN READ•Aug 19, 2025
Designing the Ideal Cadence for Compaction and Snapshot Expiration
Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.
Apache IcebergTable OptimizationCompaction - 6 MIN READ•Jul 3, 2025
How to Discover or Organize Lakehouse & Apache Iceberg Meetups
Guide on How to Be Part of the Lakehouse Community
Data LakehouseData Engineering - 10 MIN READ•Jun 23, 2025
What is an API? And Why Data Architecture Depends on Them
Hive, Glue, Iceberg Rest and so many more!
Data LakehouseData Engineering - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | What is Data Engineering?
Introduction to the terms in data engineering
Data EngineeringDatabasesData Lakes - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion
Introduction to the terms in data engineering
Data EngineeringDatabasesData Lakes - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines
Introduction to the terms in data engineering
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Batch Processing Fundamentals
Introduction to the terms in data engineering
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Streaming Data Fundamentals
Introduction to the terms in data engineering
Data EngineeringDatabasesData Lakes