Hadoop Weekly Issue #85

Hadoop Weekly Issue #85

31 August 2014

This week’s issue features a lot of good technical content covering Apache Storm and Apache Spark. There are also a number of releases—Apache Flink, Apache Phoenix, Cloudera Enterprise, and Luigi. In addition, Hortonworks announced a technical preview of Apache Kafka support for HDP, and SequenceIQ unveiled Periscope, an open-source tool for YARN cluster auto-scaling.

Technical

The eBay blog has a post about NameNode Quality of Service. While running a large cluster, they’ve found that certain jobs can cause major issues by overwhelming the NameNode with too many RPCs. To combat that, they’ve worked on the FairCallQueue, which replaces the NameNode RPC handler’s FIFO queue. The post details the status of the implementation and shows how the implementation performs in their tests.

http://www.ebaytechblog.com/2014/08/21/quality-of-service-in-hadoop/

A number of organizations are working hard to enhance Apache Storm on several fronts. The fronts include security/multi-tenancy (Kerberos authentication, Hadoop security integration, user isolation), scalability improvements, high availability for the Nimbus service, and enhanced language/tooling support. The Hortonworks blog has an in-depth article discussing these features and more.

http://hortonworks.com/blog/the-future-of-apache-storm/

On the topic of Storm, the Hortonworks blog has posted more Hadoop Summit curated content covering Storm. It highlights seven presentations, which cover scaling Storm, Pig on Storm, R and Storm, and more.

http://hortonworks.com/blog/hadoop-summit-curated-content-apache-storm/

Apache Storm integrates well with Apache Kafka (more below in an announcement from Hortonworks), and this tutorial builds a local environment using Docker and Fig for testing. It uses that environment to build a system for streaming log data through Kafka, using the Trident API to implement an exponentially weighted moving average, and sending alerts from Storm using XMPP.

http://alvinhenrick.com/2014/08/18/apache-storm-and-kafka-cluster-with-docker/

Continuing on the Storm theme, the Hortonworks blog has a post on performing micro-batching with Storm. The post focuses on implementing micro-batching with the Storm APIs (the Trident API provides micro-batching, too). This post describes three different ways to implement micro-batching and provides an example implementation using the “tick tuples” approach.

http://hortonworks.com/blog/apache-storm-design-pattern-micro-batching/

Switching gears to the first of several articles on Apache Spark, this post covers Bayesian Machine Learning on Apache Spark. It discusses integrating the PyMC framework with Apache Spark to implement Markov Chain Monte Carlo (MCMC) methods. There are five parts to the post—an introduction to MCMC methods, an overview of the PyMC python package and its API, integrating PyMC with Apache Spark, using the integration for topic modeling with MCMC, and performing distributed LDA on Spark with PyMC.

http://blog.cloudera.com/blog/2014/08/bayesian-machine-learning-on-apache-spark/

An upcoming release of Apache Spark will contain implementations of several common statistics functions found in many statistical computing packages like R and SciPy.stats. This post describes the new implementations, which cover correlations (spearman and pearson), hypothesis testing (chi-squared), stratified sampling, and random data generation.

http://databricks.com/blog/2014/08/27/statistics-functionality-in-spark.html

The Lambda Architecture is a popular idea for building hybrid batch and speed (near-realtime) data processing systems. This tutorial provides an example of implementing this type of system using Apache Spark. In addition to the normal batch operation, Spark also has a micro-batch mode called Spark streaming. The same data processing function can be used by both the normal and streaming operation, as is demonstrated in the post. The accompanying source code (written in Scala) is available on github.

http://blog.cloudera.com/blog/2014/08/building-lambda-architecture-with-spark-streaming/

Cloudera has posted a new roadmap for Impala, its SQL on Hadoop system. The post recaps the features of Impala 1.2, 1.3, and 1.4, and it describes what will be delivered in version 2.0 (by end of 2014) and version 2.1 (in 2015). The highlights for 2.0 include new analytic window functions, spilling of queries to disk, and subqueries. The highlights of version 2.1 include long-anticipated support for nested data, CRUD for HBase, and an exciting feature for folks running in AWS—Amazon S3 integration.

http://blog.cloudera.com/blog/2014/08/whats-next-for-impala-focus-on-advanced-sql-functionality/

The Hortonworks blog has a post on HTTPS for HDFS. The implementation makes use of client certificate for HTTPS client authentication, which in turn are verified by the HDFS daemons. It has details on the configuration changes required to enable and setup HTTPS as well as a walkthrough of the various SSL certificates that need to be generated (complete with example keytool invocations).

http://hortonworks.com/blog/deploying-https-hdfs/

News

Qubole has written about how they see Hadoop complementing an existing data warehouse (DW) deployment. They suggest a DW is more appropriate for structured data, whereas (large amounts) of unstructured data are better handled by Hadoop. Workloads that require SLAs/predictable runtimes should use the DW, but Hadoop is good at ad hoc or fluctuating workloads (and this is a key area where Qubole’s cloud offering adds additional flexibility). It can be hard to find the right place to draw this line, so it’s interesting to hear from a vendor (who is likely working with customers for something like this).

http://www.qubole.com/hadoop-and-data-warehouse/

As someone who has run a production Hadoop cluster, a number of the points and anecdotes in this article ring true. The overarching theme is that Hadoop is not particularly good at meeting SLAs partly because it’s easy to use Hadoop in unpredictable ways. The article has quotes from some Pepperdata folks about how their cluster orchestration software helps solve these types of issues.

http://www.zdnet.com/from-hadoops-earliest-days-to-post-yarn-why-some-issues-just-wont-go-away-7000032884/

This article is focussed on Hadoop for non-engineers, particularly folks in the healthcare industry. After giving a brief intro to the key components of Hadoop, it talks a bit about some of the implications to the healthcare industry. Specifically, there are a number of types of analyses that can be powered by Hadoop which couldn’t be done before. But with that said, there’s an interesting point that rings true in nearly every profession. The industry isn’t being held back by the data processing systems—the barrier is in acting on the data to improve healthcare.

http://www.hitechanswers.net/much-hadoop-something/

Big news in the development process of Hadoop this week—the Apache Hadoop codebase has migrated from SVN to git.

http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201408.mbox/%3CCALwhT94H3bakqx_ARqxR3G8FB0f7mH-zXmQBnU6sfDj5D79jWw%40mail.gmail.com%3E

Once a Hadoop cluster gets to a certain size, there are ultimately conflicts related to required native libraries on the cluster. Docker, which provides a mechanism to running an isolated environment on a linux host, has great promise for solving this and other types of issues. GigaOm has an article that describes the state of the YARN and Docker integration, which is being lead by Altiscale.

http://gigaom.com/2014/08/29/hadoop-jobs-should-soon-be-able-to-run-easily-inside-docker-containers/

Releases

Luigi, the batch processing framework for Hadoop, released version 1.0.17 this week. The new release has a number of fixes and improvements, including support for storing data in ElasticSearch, support for loading JSON data into redshift, an FTP task, and a new luigi command.

https://pypi.python.org/pypi/luigi

Apache Flink (incubating), previously known as Stratosphere, has released version 0.6. Flink is a data processing engine built atop of YARN, targeting iterative processing and data streaming. The release includes over 100 resolved tickets, which cover things like support for POJO and a new AvroOutputFormat.

http://flink.incubator.apache.org/news/2014/08/26/release-0.6.html

Cloudera Enterprise 5.1.2 (which includes CDH 5.1.2 and Cloudera Manager 5.1.2) and CDH 5.0.4 were released. The CE 5.1.2 release includes a number of fixes covering nearly every component in the CDH stack. CDH 5.0.4 also includes a number of fixes across the stack.

http://community.cloudera.com/t5/Release-Announcements/Announcing-Cloudera-Enterprise-5-1-2-CDH-5-1-2-Cloudera-Manager/m-p/17806#U17806

Apache Phoenix 3.1 (for HBase 0.94.4+) and 4.1 (for 0.98.1+) were released earlier this week. Both releases contain a number of bug fixes, use of nested tables in queries, a Pig loader, and more. On top of that, the 4.1 release supports distributed tracing and local indexes.

https://blogs.apache.org/phoenix/entry/announcing_phoenix_3_1_and

Hortonworks has announced a technical preview of Apache Kafka 0.8.1.1 for HDP 2.1. A post on the Hortonworks blog introduces Kafka and explains how it fits well with Apache Storm.

http://hortonworks.com/blog/announcing-apache-kafka-technical-preview-hdp-2-1/

Periscope is a new open-source tool from SequenceIQ for auto-scaling and enforcing SLAs for YARN clusters. For a static cluster, it offers tools to enforce time-based and cluster capacity SLAs. In cloud environments, it can increase cluster capacity by spinning up new nodes. The code is available on github as part of a public beta. It’s used internally at SequenceIQ, but it relies on unreleased features of Apache Hadoop and Apache Ambari.

http://blog.sequenceiq.com/blog/2014/08/27/announcing-periscope/

Events

Curated by Mortar Data ( http://www.mortardata.com )

UNITED STATES

California

What is Practical Data Science? Co-hosted with Palo Alto Data Science Foundation (Menlo Park) – Thursday, September 4
http://www.meetup.com/Hadoop-Talks/events/198019592/

Resistance Is Futile: What You Need to Know about Big Data (San Francisco) – Thursday, September 4
http://www.meetup.com/San-Francisco-Bay-Area-Microsoft-BI-User-Group/events/179128012/

Texas

Apache Spark Night – Show and Tell (Austin) – Tuesday, September 2
http://www.meetup.com/austin-data-geeks/events/189913182/

Introduction to Hadoop Course, Part 1: Hadoop and Its Ecosystem (Austin) – Saturday, September 6
http://www.meetup.com/Big-Data-Revealed-Austin/events/186518172/

Arizona

Understanding Your Customer’s Buying Journey Using Path Analysis on Hadoop (Phoenix) – Wednesday, September 3
http://www.meetup.com/Phoenix-Hadoop-User-Group/events/192459242/

CANADA

Hadoop Security Deep Dive (Toronto) – Thursday, September 4
http://www.meetup.com/TorontoHUG/events/202031802/

SWEDEN

Managing Hadoop Workflows in the Enterprise + Jumpstart your Big Data Projects (Stockholm) – Monday, September 1
http://www.meetup.com/stockholm-hug/events/201722192/

NEW ZEALAND

Large Datasets with WEKA + Big Data Use Cases & Industry Trends (Auckland) – Tuesday, September 2
http://www.meetup.com/NZ-Big-Data/events/200044182/

AUSTRIA

OCG Meetup: Hadoop (Vienna) – Thursday, September 4
http://www.meetup.com/Austrian-Cloud-and-Big-Data-Forum/events/196975542/

MEXICO

SQL en Hadoop: Un Gran Paso Adelante! (Mexico City) – Friday, September 5
http://www.meetup.com/Big-Data-Developers-in-Mexico-City/events/194328882/

INDIA

Bangalore Hadoop – Big Data Meetup (Bangalore) – Saturday, September 6
http://www.meetup.com/Bangalore-Hadoop-Meetups/events/202452242/

[…]

Read More…

Performance improvements for photo serving | code.flickr.com

Performance improvements for photo serving | code.flickr.com:

We’ve been working to make Flickr faster for our users around the world. Since the primary photo storage locations are in the US, and information on the internet travels at a finite speed, the farther away a Flickr user is located from the US, the slower Flickr’s response time will be. Recently, we looked at opportunities to improve this situation. One of the improvements involves keeping temporary copies of recently viewed photos in locations nearer to users.  The other improvement aims to get a benefit from these caches even when a user views a photo that is not already in the cache.

[…]

Read More…

Hadoop Weekly Issue #84

Hadoop Weekly Issue #84

24 August 2014

This week’s edition has a lot of great technical content from prominent Hadoop vendors Hortonworks and Cloudera as well as newcomer SequenceIQ. There are also a couple of interesting articles based on real-world experience covering an A/B testing platform and Apache Zookeeper. Those types of articles tend to be quite good but more difficult to find—as always, if you have suggestions for the newsletter please send them my way!

Technical

Hortonworks has posted a video series on the most recent release of their distribution, HDP 2.1. The videos, which are recordings of several webinars, cover a large number components including YARN, HDFS, Hive, and Ambari.

http://hortonworks.com/blog/discover-hdp-2-1-series-webinar-wrap/

A guest post on the Hortonworks blog describes how SAS is working to bring their High-Performance Analytics (HPA) and LASR Analytics Server to YARN. The systems were originally built to run on as MPI applications in which SSH was used to launch processes. With YARN, HPA uses the framework for process management, and there are improvements like enforcing CPU and memory limitations.

http://hortonworks.com/blog/apache-hadoop-helps-sas/

The Hortonworks blog has a post on an in-progress feature called container delegation. Before diving into container delegation, the post gives an intro to YARN’s resource and workload management. The new feature will be used, for among other things, to provide additional per-query resources to a long-running application.

http://hortonworks.com/blog/evolving-apache-hadoop-yarn-provide-resource-workload-management-services/

The SequenceIQ blog has a post on the YARN FairScheduler. The post has an introduction to the FairScheduler, the scheduling challenges, and some of its configuration options. Using an example test and an R-based analysis tool (which is open-sourced), the post finds that the FairScheduler is good at maintaining fairness.

http://blog.sequenceiq.com/blog/2014/08/16/fairplay/

The Hortonworks blog has had a number of security related posts in the past week. This post summarizes the coverage, which includes posts on Apache Argus and Apache Knox. It also discusses posts from some partner vendors—Protegrity, Voltage Secruity, and Dataguise. Finally, it touches on some new Hadoop features—Transparent Data Encryption for HDFS and a Key Provider API and accompanying Key Management Server.

http://hortonworks.com/blog/continued-innovation-hadoop-security/

Apache Spark ships with the spark-submit script for submitting a job to a Spark cluster. Sometimes, it’s useful or necessary to programmatically submit a job. This post describes how to write a Scala program to do so, and how to invoke the resulting binary jar.

http://blog.sequenceiq.com/blog/2014/08/22/spark-submit-in-java/

This post serves as a a good introduction to partitioning of Hive tables. It outlines the motivation and benefits of partitioning and includes several tips and best practices.

http://blog.cloudera.com/blog/2014/08/improving-query-performance-using-partitioning-in-apache-hive/

The Cloudera blog has a post with several tips and examples for writing powerful Hive queries. It includes example queries with the LAG and LEAD analytics function as well as using LATERAL VIEW and a UDTF to execute nested SQL queries. It also suggests some ways of organizing data, including the notion of a “supernova schema” which is somewhat akin to a materialized star-schema as a single table.

http://blog.cloudera.com/blog/2014/08/how-to-count-events-like-a-data-scientist/

DZone has published a cheat sheet for Apache Hadoop. It includes things like HDFS architecture, HDFS command line examples, an overview of YARN, and an introduction to MapReduce. It also covers Pig and Hive as well as providing links to several ecosystem projects.

http://refcardz.dzone.com/refcardz/getting-started-apache-hadoop

Camille Fournier, Zookeeper PMC and Rent the Runway CTO, spoke on using Zookeeper in the wild. Her talk covers a number of systems that use Zookeeper as well as a number that do not. One of her conclusions is that, while Zookeeper has a number of use-cases, it’s not always the best tool for the job.

http://www.infoq.com/presentations/zookeeper-use-case

The Pinterest engineering blog has a post on their A/B analytics platform. The post covers the implementation, which uses Kafka, Storm, MapReduce, HBase, and more. There’s an overview of the MapReduce workflow, the serving of metrics via HBase, and real-time processing via Storm. There’s also a discussion of statistical significance and group validation via chi-square.

http://engineering.pinterest.com/post/95378137929/scalable-a-b-experiments-at-pinterest

News

A new book on Apache Flume is in early release and available as an eBook from O’Reilly. The book is aimed at developers deploying and customizing Flume.

http://blog.cloudera.com/blog/2014/08/the-new-apache-flume-book-is-in-early-release/

Allied Market Research recently released a report on the Hadoop-as-a-Service (HaaS) market. It expects that market to growth rapidly to $16.1B by 2020. The report notes that HaaS doubled from 2012 to 2013, and it expects that HaaS will become more and more competitive with on-premises deployments.

http://channelnomics.com/2014/08/20/hadoop-service-reach-16-1-billion/

TPCx-HS is a new benchmark specification aimed at measuring the Hadoop Runtime, Hadoop Filesystem API implementations, and MapReduce layers. It is claimed to be the first “Industry Standard Big Data Benchmark,” and there are already plans for additional. The ODBMS blog has an interview with Francois Raab, the author of the TPC-C Benchmark, and Yanpei Chen of the Performance Engineering Team at Cloudera. In the interview, they discuss some plans for big data benchmarks in more detail.

http://www.odbms.org/2014/08/tpc-launches-tpcx-hs-vendor-neutral-industry-standard-big-data-benchmark/
http://www.odbms.org/blog/2014/08/big-data-benchmarks-interview-francois-raab-yanpei-chen/

Using Apache BigTop, CDH5 has been tested in conjunction with GlusterFS 3.3 (specifically its glusterfs-hadoop FileSystem). There are some more details on the implementation in a guest post on the Cloudera blog.

http://blog.cloudera.com/blog/2014/08/running-cdh-5-on-glusterfs-3-3/

The MapR blog has a transcript and video of a recent presentation by their CEO John Schroeder where he spends 5 minutes talking about several applications of Hadoop. He talks about the Aadhaar project’s biometric database, health care, advertising, music personalization, and MinuteSort.

https://www.mapr.com/blog/hadoop-5-minutes-or-less

Releases

Version 0.16.0 of the Kite SDK was released. This release adds support for Apache Spark, adds a new command-line ETL tool, fixes generation of Parquet Hive tables on Hive 0.13+, and adds a new parent pom for Kite SDK apps written for CDH5.

http://community.cloudera.com/t5/Release-Announcements/Announcing-Kite-SDK-0-16-0/m-p/17396

The folks at SequenceIQ have released a new docker image for Apache Hadoop 2.5.0. Like previous versions, their are psuedo-distributed and fully distributed variants of the image. The image uses Apache Ambari to provision a cluster.

http://blog.sequenceiq.com/blog/2014/08/18/hadoop-2-5-0-docker/

Microsoft made some announcements about their Azure cloud services this week. Among them, they announced the general availability of Apache HBase for HDInsight. The service had been in preview since June.

http://gigaom.com/2014/08/21/microsoft-unveils-documentdb-a-nosql-database-built-for-azure/

Spindle is a new analytics platform recently open-sourced by Adobe Research. It combines Apache Spark for processing, Apache Parquet for a data storage format, and a Spray-based HTTP server.

http://mail-archives.apache.org/mod_mbox/spark-user/201408.mbox/%3C1408136797997-12203.post%40n3.nabble.com%3E

Mortar, the Hadoop/Pig as a Service system, has announced support for running jobs in local mode to improve development iteration.

http://blog.mortardata.com/post/95203524746/announcing-instant-mortar

Events

Curated by Mortar Data ( http://www.mortardata.com )

UNITED STATES

California

eHarmony’s Hadoop Program (Irvine) – Thursday, August 28
http://www.meetup.com/OC-HUG/

Cybersecurity & Big Data Analytics with Hadoop (Mountain View) – Thursday, August 28
http://www.meetup.com/BigDataGurus/events/202176882/

HBase Meetup @ Sift Science (San Francisco) – Thursday, August 28
http://www.meetup.com/hbaseusergroup/events/197773762/

Texas

MongoDB and Hadoop: Driving Business Insights (Austin) – Monday, August 25
http://www.meetup.com/Austin-MongoDB-User-Group/events/199019262/

Missouri

Enabling Advanced Analytics & From Sandbox to Production PA (Kansas City) – Monday, August 25
http://www.meetup.com/Data-Science-KC/events/196317212/

Wisconsin

Batch Data Processing at Spotify with Luigi (Madison) – Tuesday, August 26
http://www.meetup.com/BigDataMadison/events/194365712/

Ohio

Data Governance in Big Data – Cloudera/Gazzang (Dublin) – Tuesday, August 26
http://www.meetup.com/Columbus-HUG/events/200488822/

North Carolina

Tresata on Omnichannel Marketing Analytics in Hadoop (Charlotte) – Wednesday, August 27
http://www.meetup.com/CharlotteHUG/events/167351712/

RTP – Big Data Developer Day (Durham) – Thursday, August 28
http://www.meetup.com/Big-Data-Developers-in-Raleigh/events/197731892/

Virginia

Apache Spark Lessons Learned (McLean) – Tuesday, August 26
http://www.meetup.com/Washington-DC-Area-Spark-Interactive/events/200472002/

New Jersey

Storm: Real-Time Big Data Stream Processing at WebMD (Hamilton Township) – Tuesday, August 26
http://www.meetup.com/nj-hadoop/events/201348342/

CZECH REPUBLIC

Hadooping @ Prague (Prague) – Monday, August 25
http://www.meetup.com/CS-HUG/events/177483842/

SWITZERLAND

Database as a Service (CouchDB, MongoDB, Cassandra, DB2, Hadoop) in the Cloud (Zurich) – Tuesday, August 26
http://www.meetup.com/Big-Data-Developers-in-Zurich/events/199553642/

AUSTRALIA

PaaS and Big Data Tools (Melbourne) – Wednesday, August 27
http://www.meetup.com/Big-Data-in-Practice/events/194627392/

HDInsight: MapReduce and Beyond (Melbourne) – Thursday, August 28
http://www.meetup.com/MelbourneAzure/events/202007582/

UNITED KINGDOM

3rd Spark London Meetup (London) – Thursday, August 28
http://www.meetup.com/Spark-London/events/200361502/

INDIA

Apache Spark: In Memory Map-Reduce (Hyderabad) – Saturday, August 30
http://www.meetup.com/Hyderabad-Programming-Geeks-Group/events/197496752/

CHINA

Spark Meetup (Hangzhou) – Sunday, August 31
http://www.meetup.com/Hangzhou-Apache-Spark-Meetup/events/198707512/

[…]

Read More…

RDBMS vs Hadoop storage

RDBMS vs Hadoop storage is the topic that come to mind of new Hadoop seekers, of course this topic need to understand better before we dive in to other Hadoop area. Hadoop storage is useful for storing the unstructured data from various systems but RDBMS is used for structured data storage after the data got […]

Read More…

Exploring Life Without Compass

Exploring Life Without Compass:

Compass is a great thing. At Flickr, we’re actually quite smitten with it. But being conscious of your friends’ friends is important (you never know who they’ll invite to your barbecue), and we’re not so sure about this “Ruby” that Compass is always hanging out with. Then there’s Ruby’s friend Bundler who, every year at the Christmas Party, tells the same stupid story about the time the police confused him with a jewelry thief. Enough is enough! We’ve got history, Compass, but we just feel it might be time to try seeing other people. 

[…]

Read More…

Hadoop Weekly Issue #83

Hadoop Weekly Issue #83

17 August 2014

The big news this week was the Apache Hadoop 2.5.0 release. There are also a number of interesting technical articles covering the Apache Hadoop HDFS, Apache Drill, and several other ecosystem projects. Also, there’s an interesting post on profiling MapReduce jobs (which is typically quite challenging) with Reimann.

Technical

The Cloudera blog has a post on the motivation and design for HDFS caching, which was implemented as part of the Apache Hadoop 2.3.0 release. Cloudera recommends its use in CH 5.1 to speed up Impala and other applications. Data is stored in cache by sending a cache directive to the NameNode, which keeps track of which files are cached where. This design allows applications to take advantage of locality of cached data (and enable zero-copy reads).

http://blog.cloudera.com/blog/2014/08/new-in-cdh-5-1-hdfs-read-caching/

MapR is one of the biggest proponents of Apache Drill, so it’s interesting to hear their take on the recently 0.4.0 developer preview. This post talks about Drill’s agility (it can run queries directly over datasets without the need for a metastore), flexibility (its internal data model is JSON-like allow for nested data types), and familiarity (the query language is SQL). MapR also has pre-configured packages of Drill for their distribution.

https://www.mapr.com/blog/self-service-data-exploration-here-apache-drill-040-release-now-available-mapr-platform

IPython notebooks are a popular tool for data scientist, particularly when sharing data exploration tooling. Given that Spark has a Python API, it’s a natural (and powerful) idea to marry the two for data exploration and analysis. The Cloudera blog has a detailed tutorial on setting up IPython, pyspark, and a simple IPython notebook to interact with a spark cluster. There is some example code on github and the IPython viewer.

http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/

Several months back, the Apache Mahout community announced a migration from MapReduce to Spark for the backend of core algorithms. In addition, they’re developing a Scala DSL for representing data transformations. This post looks at the Scala DSL and the rewritten (for Spark) item-based recommendation system. It also describes the command-line tool that can be used to run this system against data stored in text-delimited files.

http://occamsmachete.com/ml/2014/08/11/mahout-on-spark-whats-new-in-recommenders/

Profiling distributed systems can be a complicated task. It’s particularly hard for MapReduce jobs where there is often a mix of user-code, library code (e.g. Hive, Cascading), and framework code. This post describes how factual uses Reimann to profile Hadoop jobs. It describes the system’s profiling strategy and how results collected at a central location for analysis. The post also describes several performance issues that the system helped to uncover and resolve.

http://blog.factual.com/profiling-hadoop-jobs-with-riemann

This post on the Hortonworks blog describes how to use Apache Knox as a secure gateway to HiveServer2. It’s a fairly complicated setup (Hive client -> JDBC over HTTPS -> Knox -> HTTP -> HiveServer2), but it can be used to achieve perimeter security for a Hadoop cluster (Knox can authenticate users). The post shows how to configure Hive with Apache Ambari and the required connection strings for Knox and the Hive client (beeline). There’s also a section on configuring another client, Simba, over ODBC.

http://hortonworks.com/blog/secure-jdbc-odbc-clients-access-hiveserver2/

This presentation, recently given at the Chicago Hadoop User Group, describes the Drill data model/architecture (namely, schema “on-the-fly”), the Drill execution engine (which does runtime byte-code generation/compilation), and a Drill demo. The video of presentation is available on vimeo at the second link below.

http://www.slideshare.net/ChicagoHUG/using-apache-drill-chug-august-2014-jim-scott
http://vimeo.com/103456131

This post describes an end-to-end solution for building a recommendation engine using Apache Spark’s MLlib. The system uses MLlib’s alternating least squares algorithm to build up predictions for each user of the website, which are stored in MongoDB. It features an application built with the Play framework to serve recommendations. The code for the project is on github.

http://chimpler.wordpress.com/2014/07/22/building-a-food-recommendation-engine-with-spark-mllib-and-play/

Apache Spark streaming and Apache Storm are often mentioned as tools solving similar problems. But this presentation makes the observation/point that Spark streaming is a (micro) batch processing framework while Storm is a stream processing framework. Trident, the abstraction atop of Storm, is more comparable to Spark Streaming. The rest of the presentation focusses on comparing Trident and Spark streaming, including considerations for fault tolerance and reliability.

http://www.slideshare.net/ptgoetz/apache-storm-vs-spark-streaming

The Tachyon project is trying to solve a similar problem to the HDFS file caching solution described in an earlier post. It takes a different approach, though, by implementing an in-memory FileSystem that also supports writing through to persistent storage on HDFS (or S3 or anything implementing the FileSystem API). This post has several more details about the project, which is currently in an early release.

http://www.datanami.com/2014/08/14/amplabs-tachyon-promises-solidify-memory-analytics/

News

The Qubole blog has a post summarizing a number of recent announcements in the Hadoop ecosystem. It focusses on the business and enterprise side of the Hadoop news in more depth than this newsletter typically does.

http://www.qubole.com/hadoop-matures/

Hortonworks announced that the code of the Hadoop security offering from XA Secure (which Hortonworks recently acquired) was submitted to the Apache incubator as the Argus podling. The post describes the project charter and invites developers to help build a community around the project.

http://hortonworks.com/blog/announcing-apache-argus-call-arms/

ScaleOut hServer is a drop-in replacement for the Hadoop MapReduce engine that executes on data stored in-memory. ScaleOut announced this week that they’ve attained Hortonworks Certification.

http://hortonworks.com/blog/scaleout-hserver-achieves-hortonworks-certification/

A lot of marketing and news coverage of Hadoop surrounds tech companies in the bay area and New York. This article takes a look at other areas where Hadoop and big data are having major impact—the agriculture, insurance, and automative industries.

http://www.techrepublic.com/article/big-datas-biggest-impact-is-not-on-silicon-valley/

Splice Machine, makers of a RDBMS backed by Apache HBase and Apache Derby, recently announced a $18M round of funding. This article has an interview with their CEO during which he explains more about their business plan and target customers. Rather than competing with existing Hadoop vendors, they’re hoping to grab users of Oracle, IBM, or other enterprise RDBMS products.

http://www.idgconnect.com/abstract/8673/splice-machine-ceo-wants-database-shake-up

Hadoop is a relatively young software project, and it’s lacking a number of important features. This article discusses some of those key features (e.g. security and ease of operation) and points out that folks are using Hadoop anyway. The conclusion seems to be that Hadoop is often used as a supplement to existing systems, so folks are willing to use it even given its warts.

http://readwrite.com/2014/08/13/hadoop-slow-security-issues-still-popular

Releases

Apache Hadoop 2.5.0 was released. The new version includes updates to HDFS (extended file attributes, an improved web UI) and improvements for YARN (better REST API support and security for the application timeline server). The release also contains a large number of improvements (including to documentation) and bug fixes.

http://mail-archives.apache.org/mod_mbox/hadoop-general/201408.mbox/%3CCALwhT94eHPwr65jX0swexC5n62yL0uwrBAgKKkz59_e4sq1yNQ%40mail.gmail.com%3E

MapR has announced support for new versions of AsyncHBase, HBase, Hive, Flume, and Oozie for their distribution. Flume is seeing the largest update, going from Flume 1.4 to 1.5 (which includes a disk-spillable channel and more).

https://www.mapr.com/blog/apache-open-source-software-release-august-update

Apache Sqoop 1.4.5 was released. The new version adds support for Apache Accumulo and a new high-performance Oracle connector. There are also a large number of bug fixes and improvements (covering HBase, Avro, Amazon S3, and MySQL support).

http://mail-archives.apache.org/mod_mbox/sqoop-user/201408.mbox/%3CCAOvM-chFPXjW-1YWXFUfrGHE-bw+E=KN534k1UgH-CrprDJcYQ@mail.gmail.com%3E

Mortar (full disclosure: they help with this newsletter and syndicate Hadoop Weekly) have open-sourced their StoreFunc for DynamoDB. The so-called DynamoDBStorage UDF allows for efficiently writing data to DynamoDB as part of a Pig job. It is customizable in its write throughput and retry behavior.

http://blog.mortardata.com/post/94633617697/the-best-way-to-use-dynamodb-with-hadoop-is-now

Events

Curated by Mortar Data ( http://www.mortardata.com )

UNITED STATES

California

Escape From Hadoop: Spark One-Liners for C* Ops (Milpitas) – Tuesday, August 19
http://www.meetup.com/DataStax-Cassandra-South-Bay-Users/events/197630412/

OC Big Data Monthly Meetup #4 (Irvine) – Wednesday, August 20
http://www.meetup.com/OCBigData/events/179381152/

Bay Area Hadoop User Group Monthly Meetup (Sunnyvale) – Wednesday, August 20
http://www.meetup.com/hadoop/events/136216522/

Network Design Challenges for Hadoop Environments (San Francisco) – Wednesday, August 20
http://www.meetup.com/San-Francisco-Arista-Networks-User-Group-Meetup/events/197512552/

Idaho

Boise BI User Group Summer Session (Boise) – Thursday, August 21
http://www.meetup.com/BoiseBI/events/199354212/

Utah

Hadoop Lunch at Adobe – Competition Rules/Details (Lehi) – Thursday, August 21
http://www.meetup.com/BigDataUtah/events/199388232/

Arizona

Genomic Sequencing & Hadoop (Scottsdale) – Tuesday, August 19
http://www.meetup.com/Phoenix-Biomedical-Informatics-Group/events/197062292/

A Detailed Look at Big R: R + IBM InfoSphere BigInsights (Scottsdale) – Wednesday, August 20
http://www.meetup.com/Big-Data-Developers-in-Phoenix/events/198736022/

Colorado

Getting Jiggy with Change Data Capture and Slowly Changing Dimensions (Boulder) – Wednesday, August 20
http://www.meetup.com/Boulder-Denver-Big-Data/events/200389372/

Nebraska

Apache Drill: Building Highly Flexible, High Performance Query Engines (Omaha) – Thursday, August 21
http://www.meetup.com/Heartland-Big-Data-Meetup/events/191993642/

Texas

Apache Samza: LinkedIn’s Real-Time Stream Processing Framework (Austin) – Wednesday, August 20
http://www.meetup.com/Distributed-Systems-Enthusiasts/events/200344292/

3rd Thursday Huddle! (Dallas) – Thursday, August 21
http://www.meetup.com/Big-Data-in-the-Big-D/events/172496362/

Illinois

Hybrid BI Solutions with Hadoop and Microsoft Toolsets (Oak Brook) – Thursday, August 21
http://www.meetup.com/Business-Intelligence-Chicagoland/events/195317252/

What’s New with Apache Spark? An Evening with Paco Nathan (Chicago) – Thursday, August 21
http://www.meetup.com/Chicago-Spark-Users/events/198384702/

Virginia

Building a Fully Functional Hadoop Cluster in 1 Hour for Less Than $1 (Richmond) – Tuesday, August 19
http://www.meetup.com/RVA-Data-Hackers/events/195677742/

North Carolina

Triad Hadoop Users Group (Winston Salem) – Thursday, August 21
http://www.meetup.com/Triad-Hadoop-Users-Group/events/187389672/

Pennsylvania

HUG Pittsburgh August Meeting (Pittsburgh) – Wednesday, August 20
http://www.meetup.com/HUG-Pittsburgh/events/195143712/

SQL on Hadoop (Philadelphia) – Wednesday, August 20
http://www.meetup.com/Big-Data-Developers-in-Philadelphia/events/199121852/

ROMANIA

Real-World Hadoop Applications, Built in Bucharest (Bucharest) – Thursday, August 21
http://www.meetup.com/Hadoop-User-Group-Bucharest/events/196677742/

INDIA

Hadoop Meetup (Bangalore) – Saturday, August 23
http://www.meetup.com/Bangalore-Baby-Hadoop-group/events/197924072/

[…]

Read More…

PIG vs HIVE

(PIG vs HIVE ), this is the most often question asked by new Hadoop information seekers .  Pig is a procedural data-flow language , programmers can execute programs step-by-step  defined by them self. Optimization  can be controlled for each and  every steps. Hive looks like SQL (structured query language) .Hive depends on its own optimizer and […]

Read More…

Pig to ease Hadoop programming

Pig is a Hadoop extension ease the programming  in MapReduce and other levels of  programming  by its  simple high-level data processing language . Pig will  automatically optimize  the pig scripts and make the  scripts  free from manual tuning. Pig has  two main components A high-level data processing language called Pig Latin A compiler that can compiles and execute Pig […]

Read More…

12 Hadoop myths

Hadoop has been touted as one of the newer– and perhaps one of the best—technologies designed to extract value out of “big data”. Hadoop technology has so frequently been linked to the concept of big data that the two often appear in lockstep at conferences, industry briefings and in media reports. But as Hadoop becomes a household name, it […]

Read More…

Hadoop Weekly Issue #82

Hadoop Weekly Issue #82

10 August 2014

We’re in the midst of a summer lull, so this week’s issue is shorter than usual. The lack of quantity is made up for in great quality, though. Technical posts cover YARN, HBase, Accumulo, and building an EMR-like local dev environment. There is also news on Actian, Adatao, Splice Machine, and the HP-Hortonworks strategic partnership. Hopefully there’s something for everyone!

Technical

The Hortonworks blog has a post on the ongoing work to improve the fault-tolerance of YARN’s ResourceManager (RM). This post describes phase two of the RM restart resiliency work, which aims to keep existing YARN application running during and after an RM reboot. The post covers the architecture of the solution, including which cluster state information is stored where.

http://hortonworks.com/blog/resilience-apache-yarn-applications-across-resourcemanager-restart-phase-2/

Hortonworks has another post in their series on curated Hadoop Summit content. This time, it focusses on Hadoop security. They highlight four sessions covering recent improvements in Hadoop security, security for the Apache Knox Gateway’s REST APIs, using Hadoop for threat detection, and the future of Hadoop security.

http://hortonworks.com/blog/hadoop-summit-curated-content-apache-hadoop-security/

The Apache blog has a post with updated performance evaluations of various HBase BlockCache configurations. They find that the on heap LruBlockCache performs best and the next best configuration is the CombinedBlockCache:OffHeap (a hybrid L1 LruBlockCache and a L2 BucketCache which stores data offheap). The post has details on the experimental setup and a deeper analysis of the results.

https://blogs.apache.org/hbase/entry/comparing_blockcache_deploys

An obstacle of adopting AWS’ Elastic MapReduce (EMR) can be building a local dev environment that matches EMR. While Amazon’s distribution isn’t open-source, this post describes how to setup an approximate local environment on a Mac. It shows you how to make configuration changes for s3 uris, sets up the AWS access keys, and add LZO compression support to Hadoop.

http://www.hakkalabs.co/articles/making-your-local-hadoop-more-like-aws-elastic-mapreduce

This is a good introduction to Apache Accumulo, the distributed key-value store built on HDFS. It describes the architecture at a high-level, contrasts it to Apache HBase, describes the data model (including column visibility), several use cases, and more.

http://www.slideshare.net/DonaldMiner/an-introduction-to-accumulo

This post looks at using Hadoop and new libraries for iterative computation, such as k-means clustering. It describes Iterative MapReduce, the Twister Programming Model, the Collective Model (the Harp project), and more. There are some experimental results of various frameworks for PageRank, K-means, and broadcast.

http://www.hpcc.unical.it/hpc2014/pdfs/qiu.pdf

News

Videos of the 2014 Accumulo Summit, which took place in June, have been posted online. There are presentations from folks at Sqrrl, Cloudera, Hortonworks, and more.

http://accumulosummit.com/program/talks/

The Hortonworks blog has a post from the HP team on the recently strategic partnership between Hortonworks and HP. It has some specifics on the partnership—Apache Ambari will be integrated with HP Operations Manager i (OMi).

http://hortonworks.com/blog/hp-hortonworks-apache-ambari-hadoop-operations/

Adatao announced a $13M Series A round of founding. The company makes pInsights, a predictive analytics and business intelligence solution built on Apache Spark. They also make pAnalytics, a system aimed at data scientists.

http://www.infoworld.com/t/hadoop/hadoop-meets-google-docs-analytics-made-easy-247862

Relational database on HBase startup, Splice Machine, announced that its Series B round was increased by $3m to $18M in total. The latest money comes from Correlation Ventures.

http://gigaom.com/2014/08/06/sql-on-hadoop-startup-splice-machine-adds-3m-in-funding/

Outspoken Hadoop skeptic and prolific DMBS researcher/creator Michael Stonebreaker has written a post with the provocative title “Hadoop at a Crossroads?” He argues that with the death of MapReduce (focussing in particular on next-generation SQL-on-Hadoop systems), Hadoop (and its vendors) are on a collision course with data warehouse systems. The post also questions the future of HDFS, which he predicts might fall victim to specialized storage layers.

http://cacm.acm.org/blogs/blog-cacm/177467-hadoop-at-a-crossroads/fulltext

Actian recently announced the Actian Vector Hadoop Edition, which is a SQL-on-Hadoop system. This post has more details on the integration, including how Actian uses HDFS (it has a proprietary file format) and YARN.

http://www.dbms2.com/2014/08/07/actian-vector-hadoop-edition/

Datanami has a post on Sinequa, makers of enterprise search software. The most recent version of their software adds support for analyzing data stored in HDFS and a handler for Apache Mahout to perform analysis using its algorithms.

http://www.datanami.com/2014/08/08/heres-another-option-hadoop-enterprise-search/

GigaOm has an article exploring some of the recent momentum of Apache HBase. While Cassandra and MongoDB have seen a lot of press coverage and adoption, HBase is gaining steam. Specifically, it has good integration with the Hadoop ecosystem and a number of companies are starting to build applications on top of it (e.g. Continuuity’s reactor and Splice Machines relational database).

http://gigaom.com/2014/08/10/is-hbases-slow-and-steady-approach-winning-the-nosql-race/

Releases

Apache Drill 0.4.0 was released this week. Drill is general purpose analytics software that strives to build a more general framework than existing systems (i.e. SQL-on-Hadoop) by supporting a wide variety of storage systems/formats and queries. The 0.4.0 release is a massive step forward with 100,000 lines of new code from a wide variety of contributors. The Apache Blog has the highlights of the new release.

https://blogs.apache.org/drill/entry/announcing_apache_drill_0_4

ZooKeeper 3.5.0-alpha was released this week. The release resolves over 500 Jira tickets, which include a large number of bug fixes and improvements. Among the improvements are the ability to dynamically reconfigure the ZooKeeper ensemble, improvements to recovery, better support for jdk7 and openjdk, and more.

http://zookeeper.apache.org/doc/r3.5.0-alpha/releasenotes.html

Events

Curated by Mortar Data ( http://www.mortardata.com )

UNITED STATES

California

August SF Hadoop Users Meetup (San Francisco) – Wednesday, August 13
http://www.meetup.com/hadoopsf/events/196685952/

Apache HBase: Understanding Where to Use It and How to Use It, with Subash DSouza (Los Angeles) – Wednesday, August 13
http://www.meetup.com/Los-Angeles-Big-Data-Users-Group/events/175710252/

Apache Solr (Irvine) – Thursday, August 14
http://www.meetup.com/Orange-County-Java-Users-Group-OCJUG/events/195103492/

Texas

Introduction to Spark Course: Spark Streaming (6 of 7) (Austin) – Wednesday, August 13
http://www.meetup.com/Austin-ACM-SIGKDD/events/187688872/

Illinois

Using Apache Drill (Chicago) – Wednesday, August 13
http://www.meetup.com/Chicago-area-Hadoop-User-Group-CHUG/

Pennsylvania

Distributed Data Storage: Comparing Cassandra, HBase, ElasticSearch and GridGain (Conshohocken) – Wednesday, August 13
http://www.meetup.com/Philadelphia-Cassandra-Users/events/196396302/

New York

Neo4j Intro Workshop (New York) – Tuesday, August 12
http://www.meetup.com/NYC-Open-Data/events/168621462/

ENGLAND

A Leap Forward for SQL on Hadoop (Manchester) – Tuesday, August 12
http://www.meetup.com/HadoopManchester/events/195672432/

RUSSIA

Workshop: SQL on Hadoop (Moscow) – Friday, August 15
http://www.meetup.com/Big-Data-Developers-in-Moscow/events/193667662/

[…]

Read More…