Hi I’m Jared Hillam,
If you’ve been accustomed to working with traditional SQL databases hearing somebody talk about Hadoop can sound like a crazy mess. . So in this video I’m going to try and simplify only 3 of many differences between Hadoop and traditional SQL sources which hopefully will provide context as to where each medium is used.
The first difference I want to point out is Schema on Write vs Schema on Read When we move data from SQL database A to SQL database B we need to have some information on hand before we write to database B. For example, we have to know what the structure of database B is and how to adapt the data from database A to fit that structure. Additionally, we have to ensure that the data being transferred meets the data types that the database is expecting. If we attempt to load something that does not meet what database B is expecting, then it will spit out errors and it will reject the data. This is what we call Schema on Write. Hadoop on the other hand has a schema on read approach. So when we write data into what’s called the Hadoop Distributed File System we just bring it in without dictating any gatekeeping rules. Then when we want to read the data, we apply rules rules to the code that reads the data rather than preconfiguring the structure of the data ahead of time. Now the concept of schema on write vs schema on read has profound implications on how the data is stored in Hadoop vs SQL which leads us to our second difference. In SQL the data is stored in a logical form with interrelated tables and defined columns. In Hadoop, the data is a compressed file of either text or any other data types. However, the moment data enters into Hadoop the file or data is replicated across multiple nodes in the Hadoop Distributed Filing System. So for example, let’s say we’re loading twitter data, and we have a large Hadoop cluster of 1000 servers. My twitter data might be replicated across 60 of them, along with all the other profiles of twitter users. Hadoop keeps track of where all the copies of my profile are. This seems like a waste of space, but it’s actually the secret sauce to the massive scalability magic in Hadoop, and this leads us to our 3rd difference. When thinking about Big Data solutions like Hadoop, think of something architected for an
unlimited number of servers. so let's stick with earlier example, and
will say we have one thousand servers in our Hadoop cluster. Now, imagine I'm searching
Twitter data, and I want to see all the sentences that have the word “unhappy”. The structure of the query in Hadoop will come in the form a Java program and that program defines
the request and distributes the calculation of that
search across all the 60 replicated copies of my profile in HDFS. However, instead of each copy conducting the exact same search, the Java code will break apart the workload so that each server is working on just a portion of my Twitter history. As each copy of my twitter data finishes its assigned segment of history, the answers are delivered to a reducer
program on the same node cluster, in this is responsible for adding up all the
tallies are producing a consolidated list. Now let’s say that 1 of those 60 servers servers breaks down or has some sort of issue while processing my request. The question now is, should I hold up the entire response to the user because I won’t have a complete data set yet? It’s the answer to this question is one of the primary differentiators between Hadoop and SQL. Hadoop would say no, and it would provide the user an immediate answer and eventually it would have a consistent answer. SQL would say yes, we must have complete consistency across all the nodes before we release anything to the user which is called a 2 phase commit. Now, neither approach is right nor wrong, as both have an important role to play based on the type of data being used. The eventual consistency methodology in Hadoop is a far more realistic method of reading continuously updating feeds of unstructured data across 1000’s of servers. While the 2 phase commit methodology for SQL databases is well suited for managing and rolling up transactions so we’re sure we get the right answer. However, with Hadoop we have a caveat. Because the query is literally a mapping and answer consolidation program which propagates to a flexible number of servers, the sky is the limit on how creative we can get with that program. But as you would guess, this Java based query also increases the complexity of talking to Hadoop. For this reason, you’ll find a lot of packaged distributions of Hadoop that provide some structure and focus to the Hadoop world. For example, Facebook created something called Hive that allowed their team members that didn’t know how to write Java code to write queries in standard SQL. Hadoop is so flexible that Facebook was able to build a program to essentially mimic SQL behavior on demand. This flexibility is also one of the reasons it’s so hard to nail down a single practice or even a “typical” deployment type.
The good news is that some of the leading tools for data management and analysis are now beginning to natively write Map Reduce programs and pre-packaged schemas on read so that organizations don’t have to hire expensive data scientists to get value from this powerful architecture. Intricity can help introduce you to some of these powerful platforms that allow for codeless access to your unstructured data. These are sources like your website’s log files, unstructured survey responses, or even using Hadoop as an Operational Data Store. I recommend you reach out to Intricity, and talk with one of our specialists. We can help you lower the cost of entry into this dynamic way of consuming data, while bringing value to your current infrastructure.
