ApacheCon North America 2014

A long time ago in a galaxy far, far away only the chosen few could deploy and operate a fully functional Hadoop cluster. Vendors were taking pride in rationalizing this experience to their customers by creating various distributions including Apache Hadoop. It all changed when Cloudera decided to support Apache Bigtop as the first 100% community driven bigdata management distribution of Apache Hadoop. Today, most major commercial distribution of Apache Hadoop are based on Bigtop. Bigtop has won the Hadoop distributions war and is offering a superset of packaged components. In this talk we will focus on practical advice of how to deploy and start operating a Hadoop cluster using Bigtop’s packages and deployment code. We will dive into the details of using packages of Hadoop ecosystem provided by Bigtop and how to build data management pipelines in support your enterprise applications.

Apache Hadoop users can drive adoption within their organization by implementing Apache Sentry Role Based Access Control (RBAC). This talk will discuss the problems associated with using Apache Hadoop's low level authorization primitives in the context of Apache Hive and Apache SOLR, and how Apache Sentry addresses those problems. As well as how to implement Apache Sentry on your cluster, and the Apache Sentry roadmap, followed by a demo of Apache Sentry using Apache Hive and Apache SOLR.


- Concepts
- Practical concerns
- Installation
- Configuration
- Administration

Apache Falcon is a framework for simplifying data management and pipeline processing in Apache Hadoop. It enables users to automate the movement and processing of datasets for ingest, pipelines, disaster recovery and data retention use cases. Instead of hard-coding complex dataset and pipeline processing logic, users can now rely on Apache Falcon for these functions, maximizing reuse and consistency across Hadoop applications.

Apache Falcon simplifies the development and management of data processing pipelines with introduction of higher layer of abstractions for users to work with such as Data Sets, Process and Infrastructure entities that are expressed using declarative language.

The presentation covers detailed design and architecture along with case studies on the usage of Falcon in production. We also look at how this compares against solutions if we took a silo-ed approach.

Apache Sqoop is a tool that was created to efficiently transfer big data between entire Hadoop ecosystem (components such as HDFS, Hive or HBase) and structured data stores (such as relational databases, data warehouses, or NoSQL systems). The popularity of Sqoop in enterprise systems confirms that Sqoop does bulk transfer admirably.

In the meantime, we have encountered many new challenges that have outgrown the abilities of the current infrastructure. To fulfill more data integration use cases, as well as become easier to manage and operate, a new generation of Sqoop has been created. With focus on ease of use, ease of extension, and security Sqoop 2 was born. This session will dive into Sqoop 2 architecture, describing differences between Sqoop 1, and the benefits that the new architecture brings.

Kafka is a publish/subscribe messaging system that, while young, forms a vital core for data flow inside many organizations, including LinkedIn. We will discuss Kafka from an Operations point of view, including the use cases for Kafka and the tools LinkedIn has been developing to improve the management of deployed clusters. We'll also talk about some of the challenges of managing a multi-tenant data service and how to avoid getting woken up at 3 AM.

MQTT is a publish/subscribe protocol used in the Internet of Things to send telemetry data. Unlike Kafka, MQTT clients are simple and easy to implement. It is designed for devices with strict memory and power constraints. However, MQTT has an Achilles heel when it comes to scaling to high loads. Tim will review both protocols and dissect the performance characteristics of both protocols and discuss how his team at 2lemetry used Kafka to address MQTT's scaling problems.

Interest in analyzing the real-time web has reached a fever pitch among academics and corporate executives. Researchers and professionals tasked with capturing and analyzing high volumes of real-time social data have a plethora of open-source databases and machine learning libraries to choose from, but often spend a large fraction of their time writing code (and manually performing) ingestion, cleansing, normalization, and data management.

Apache Streams seeks to break these problems down into self-contained modules based on simple interfaces and foster a community-based approach to connecting and harmonizing data sources and services. Implementers can compose a data workflow from streams components and run their workflow in real-time or batch modes, using a variety of storage services (Kafka, HDFS, Cassandra, etc...) and execution engines (Tomcat, Storm, Amazon Kinesis, etc...)