ApacheCon North America 2014

Apache Falcon is a framework for simplifying data management and pipeline processing in Apache Hadoop. It enables users to automate the movement and processing of datasets for ingest, pipelines, disaster recovery and data retention use cases. Instead of hard-coding complex dataset and pipeline processing logic, users can now rely on Apache Falcon for these functions, maximizing reuse and consistency across Hadoop applications.

Apache Falcon simplifies the development and management of data processing pipelines with introduction of higher layer of abstractions for users to work with such as Data Sets, Process and Infrastructure entities that are expressed using declarative language.

The presentation covers detailed design and architecture along with case studies on the usage of Falcon in production. We also look at how this compares against solutions if we took a silo-ed approach.