The people & business behind Hadoop

Every decade has a key technology and a wave of new companies breed up around this technology. For the 90’s it was write once run anywhere Java, for the 2000’s the whole world was caught up in the web. For the 2010 decade, the software section is buzzing with the term Big Data along with a significant increase in consumer electronics such as tablets and phones. The key software framework which is most sought out for understanding Big Data is Hadoop. There are three main next gen companies which want customers to ask big questions and brave into new frontiers. In this blog I want to write about Hadoop and its allies.

Hadoop was created by Doug Cutting (used to work for Yahoo, currently Chief Architect at Cloudera & Director at Apache Software Foundation) and Mike Cafarella (currently a professor of computer science at University of Michigan) in 2005

Hadoop was quickly recognized as the go to solution for distributed computing using commodity hardware. This framework promised painless deployment, easy maintenance and was at the right time for handling the big data explosion (being a bit dramatic).  Ever since its release under the Apache federation foundation this project amassed lot of committers; soon an ecosystem was built around Hadoop. Variety of companies started providing Hadoop consulting services, amongst them the most popular are Cloudera, Hortonworks & MapR 

Cloudera

Three top engineers from Google, Yahoo and Facebook (Christophe Bisciglia, Amr Awadallah and Jeff Hammerbacher) foresaw a need for analyzing, managing and processing big data, so they teamed up with an Oracle employee and formed Cloudera in the early 2008.

Open Source Contribution

The employees of this company are active participants to many open source projects, some popular projects are:

Apache Avro, Apache Bigtop, Apache Crunch, Apache Flume, Apache Hadoop, Apache Hive, Apache Lucene, Apache MRUnit, Apache Oozie, Apache Sqoop, Apache Whirr, Apache ZooKeeper, Cloudera Development Kit, Crepo, Hue, Impala, Kitten, ML, Seism, CDH - 100% 
Open Source Hadoop Distribution

Other services provided by this company include a Cloudera Manager & Navigator- Centralized System Management which provide a rich set of features which include

Manager

Deployment & Configuration, Service Management, Service & Host Monitoring, Diagnostics API, Rolling Updates/Restarts, SNMP Support, LDAP Integration, Configuration History & Rollbacks, Operational Reports, Automated Disaster Recovery, BDR Add-on

Navigator

Data Audit – HDFS, HBase & Hive, Navigator Add-on, Access Management, Technical Support and Indemnity, Core Projects, Apache HBase, RTD Add-on, Cloudera Impala, RTQ Add-on, Cloudera Manager, Cloudera Navigator, Navigator Add-on

Partners & valuation

Over 160 active partners

Current net valuation of $700 million at least that the word on the street, along with an additional $65 million through Series E round of venture capital funding by Accel patners .

Developers Community

Cloudera offers training for only 5 days at a price of around $2500. This training includes Hadoop development, administration and data science. Also they provide certification as well.  

Hortonworks

This company was incubated by Yahoo. The employees of this company are regular committers to various Apache Hadoop eco systems projects. Primarily this company provides consulting services and offers an end to end solution with the Hadoop framework. They are emerging out with various data products with the newer version of Hadoop. 

Open Source Contribution

Yarn, Tez, Stinger, Ambari, HDP platform

Partners & Valuation

Microsoft, Teradata, Talend, Rackaspace are official partners. This company is currently valued at $200 million.

Developers Community

Hortonworks offers a similar training like Cloudera for only 5 days at a price of around $2500. This training includes Hadoop development, administration and data science. Also they provide certification as well. The advantage with the Hortonworks is they are committed with Microsoft. They provide training for Hadoop on Windows platform.

MapR

My personal opinion their business model is not viable, although backed up with EMC and Cisco, their online presence is dominated by Cloudera and Hortonworks. This company provides a commercial Hadoop solution by offering three products M3, M5 and M7. The employees of this firm actively participate with HBase, Pig, Apache Hive, ZooKeeper.

Partners & valuation
Google Compute Engine, EMC Cisco, valued at $30 million

What’s in it for us?

Well Hadoop is free, anyone can download it and use it, and the trick is to know what to use it for and how to manage it. There is a need for distributed computing platform and Hadoop by itself is in its nascent stages. It’s definitely not a threat to existing database platforms however investing time and money with Hadoop would be a safe bet for future.     

Insert into a table from stored proc

I never realized until today that the results from a stored proc can be inserted into a table. For example, I am selecting employeeID and ManagerID from Adventureworks

select EmployeeID, ManagerID
from HumanResources.Employee

Then create a stored procedure

create procedure dbo.EmpTest
as
select EmployeeID, ManagerID
from HumanResources.Employee

The results of the stored proc exec EmpTest can be used to insert into a table as follows.

declare @test table
(
    EmployeeID int,
    ManagerID int
)

insert into @test
exec EmpTest

select * from @test

How to read pdf in chrome in ubuntu 12.04

1. Download the latest chrome software from google
2. extract the libpdf.so from the archive and copy to the /usr/lib/chromium-browser, you need admin rights to copy do sudo
3. close chrome if opened, then check out chrome://plugins/, Chrome PDF Viewer  should be enabled

Ubiquitous Windows

Long ago, during Bill Gate times, every home should have a PC was the motto. As matter of fact it has been evident, now every home has a PC. Fortunately or unfortunately, time changes things, now every individual has a tablet, phone, laptop attached to his body. How this customer requirement direct the coming years of Microsoft?

If I remember correctly at some windows developer conference, our age old Mr. Gates vision has been modified to, "Every and any computing platform should be able to run Windows. Windows everywhere!!!” This according to me is a smart move from Microsoft. What has this got to do for developers and people who are aspiring for a career with Microsoft technologies?

This thought came to me while talking with my friend who is into device drivers. Microsoft has its arms spread out in Phones, Laptops. Tablets, Gaming Devices and Enterprise level software development. Making windows run on every platform is a herculean task which requires an in depth knowledge of device drivers, windows architecture and programming skills. To all those who are starting out a career like me with Microsoft my suggestion would be to keep Windows Device Drivers (WDD) and .NET Micro Framework(NETMF) up their ally. With more and more devices pouring into display, camera, various sensors and other cool things learning how to integrate them with windows would require an understanding of windows architecture itself. It’s a big area to ponder; I am starting out with these free tools given by MS. I think the day will come where I will say, “Hey, I want to port Windows on my new printer, get me the image.”

http://msdn.microsoft.com/en-us/windows/hardware/gg487428

http://msdn.microsoft.com/en-us/windows/hardware

Fixing the csc.exe issue

I had to write fairly small programs in C# of late and did not feel like using the IDE, however I ran into the path setting problem. My command line was not able to call the csc.exe to compile my code. The error:  'csc' is not recognized as an internal or external command, operable program or batch file. This is what I did to fix it:

1. Locate 'csc.exe' on your PC. Here is thing which I don't know. I have googled this issue and people suggested  to add Framework location and other folders. Also, on my box I have csc.exe on multiple locations none of them worked for me except for this one. c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC. Copy this address on a notepad.

2. Go to Control Panel\System and Security\System and click on Advanced system settings on your left, if you are admin you can access else call the admin. Once you click on Advanced System Settings click on environment variables which will open a dialog box look for System Variables under that you have multiple system variables. Select path and click edit

3. Carefully copy the address which is on notepad and add it to the end of the variable value list. click ok, twice and viola you are ready. Now you should be able to access csc from cmd prompt. 

Happy Coding!!

Inline table value function (tvf)

1. Produces a dynamic table at the time of execution

2. Unlike a view table valued function can be parameterized

3. Table valued function accepts a search criteria as function parameters; in a VIEW it is can be specified in a where clause 

VIEW:

create view view_person
as
select top 5 * from person.address

select * from view_person

TVF:

create function person.tvf_person
returns table
as
return
(
select * from person.address
);
go

select * from Person.tvf_person()

Importance of Software Requirement and Requirement Gathering

Business requirements for building an enterprise scale software apps are very crucial and critical for the business. Every IT professional is aware of the process of gathering requirements and its implications. Unfortunately, in my opinion, the cost of formulating an incorrect business rule and coding the rule is least understood by most IT professionals. The following is my experience of requirement fallacy.

My organization is providing health care services to disabled individuals. The IT division in my org is building apps to facilitate the other employees to handle the financial, medicare/medicaid, placement of the disabled. I am part of new project and we don't have a business analyst to gather the requirements (imho BA's are PRO for IT), so switched multiple hats (as usual) went straight to the stakeholder to get the required info. The stakeholder redirected to a specific employee to answer my questions. One of the main questions I had was

Can the benefit amount awarded to the client be split among entitled payees?
The reply was, yes the benefit amount comes in one check and is split among multiple payees.

Benefit Amount (ID  1) = $100
BenefitID 1 $X to Payee 1
BenefitID 1 $Y to Payee 2

So, took this reply and went to my team to discuss the implications because this was new information. The current database cannot handle the split amount of the benefit. For ex. the benefit could be $100 and is split $50 each if there were two payees.  We had about two months of intensive discussion on how to change the current asset table which houses all the benefit info of the client. Plan for how to change the front end screen to handle this new feature along with planned changes to the triggers and stored procs.

After 2 months we had a meeting again and I raised same to make sure we were on same page.  

Can the benefit amount awarded to the client be split among entitled payees?
The reply was, yes the benefit amount comes in one check and is split among multiple payees. The split is not on the entitled amount but it is as follows
Benefit ID1-a $X to Payee 1
Benefit ID1-b $Y to Payee 2
and its is not

This threw off the discussions to waste.  As most employees in IT are consultants, the IT management team successfully utilized the consultants time to design, build and  test an incorrect business requirement.

What can be the done to bridge the gap? Business communication is critical. Client may speak gibberish but they are always  correct because it makes sense to him/her. Its up to the translator to convert the gibberish  to proper sentences.   

Rule 1: Always double check the answers/business rules
Rule 2: Re ascertain yourself by asking  "is this what you are trying to say" or "Is my understanding of the situation correct?"
Rule 3: Try ask questions beginning with what/why/who/where/when and how

A part of software engineering is requirements gathering which form the basis for the software application.