Big Data Camp 27/10/2013

Big Data Camp was my first un-conference in big data. It’s really different compared to a regular conferences, the sessions are created on the fly, walk in and walk out are welcome. From what I have seen, it was non-technical, focused on the noob to intermediate skill market cap. The host Dave, managed the event very eloquently. Overall, it was good event. I complied small notes on what I liked.   

Talk1: Hadoop in consumer security products

• This organization implements hadoop technology to analyze the sales of their flagship consumer security products
• Existing data warehouse products and tools were OK, although they saw lot of customer’s showing up on their site, the sales reduced.
• The product was not at fault, but the customer analytics was.  The primary touch points or data points viz, server log, click stream etc were fed to hadoop, analytics was performed using sql, hive, mahout, used linear regression for modeling their data.

Talk 2: How to choose data analytics product

This presenter works at Datapad. “There is no one right way, there is a right way for your data, for your team” kinda made sense. Advocated the use of Python and Pandas library.

Talk 3: Block box vs transparent data modeling

I was trying to figure out what his point was, this is my understanding, the functionality of any google product is basically a black box and he was not cool with that or whatever.

Talk 4: Mainframe + Hadoop

This talk was pretty interesting, the presenter works at Syncsort company in CA.  

• 75% of data stored in mainframes (fact)
• Insurance, Retails and Banks store data in mainframes
• Offload mainframe data and batch process it in hadoop
• Mainframe store in ebcdic data type, (new info)
• ADP is hiring people for mainframes seriously?? (Maybe I should learn COBAL.. heck no!!)

Talk 5: Time series database – things happening in time

This was cool too, open source database called InfluxDB - Database based on events,

• HTTP native, show and do the analytics in a browser
• Read/write, manage and security with Http
• I asked the presenter, how is this product different form Storm, Spark & StreamInsight, could not answer my question 100%

Some other random tidbits, big data GUI tools, IBM Big sheets, Datameer &Talend

Python Tools for Data Science

I believe Python is increasingly becoming the de facto language for data science applications. Of the many good feature of this language, string manipulation is ridiculously easy and it is a very simple language to pick up. In this post I put together the tools and IDE’s related to Python which I found useful for data science applications.

Scientific Computing

• NumPy http://www.numpy.org/, scientific computing toolkit
• SciPy http://www.scipy.org/, scientific computing toolkit
• Matplotlib http://matplotlib.org/, graphics and plotting
• Scikit http://scikit-learn.org/, machine learning library  

IDE for Python

• Stani Editor,  http://pythonide.blogspot.com/, decent simple editor
• PythonXY https://code.google.com/p/pythonxy/, IDE built with Spyder
• Liclipse, http://brainwy.github.io/liclipse/ standalone python IDE based on eclipse
• PyDev, http://pydev.org/ plugin for Eclipse

Basic math skills for data science  

• Linear Algebra
• Statistics
• Probability
• Calculus

Handling M:M ring in Entity Modeling

Complex business process operations can be effectively visualized with Entity Relational ER modeling. ER modeling is proven to be the best way to design and model relational databases, this design choice is based of Dr.Peter Chan’s papers published in 1976. Before discussing the M:M ring problem in ER modeling, I will brief description the basics of ER modeling.

Nouns in an english sentence are usually the entities ex: Customers, Company, Teachers etc. The physical properties of these nouns are attributes for entities. ex height, weight, date, etc.

Entities can be related to each other, the relation is usually verbs/ adverbs ex: “based on”. “for”, “basis off”, “bought from”, “operator of”, “issued for”, “stored in”, “responsible for” etc

Simple ER example would be “Customers can place Orders ”, in this example, the relation is called 1:M, one customer can place more than one order (m for many). In some cases, like social and a person would be a 1:1 relation, one person can have 1 social

Modeling employment history in entities is a bit of a pickle. For instance, consider entities person, company and position. Person can work at a company holding a position  or, we can say a company hired a person for a position, or a position is held by a person employed at a company. Visually this can be as shown below.

It becomes increasingly tricky to answer the following question for each person we want to track the position held, for what company and how long. This configuration leads to what is known as M:M ring. This issue can be resolved by adding three other small entities as shown below.

TSQL Errors

Error 1: Cannot define PRIMARY KEY constraint on nullable column in table 

Cause: set ansi null default on, when you create a new col

Solution: explicitly state not null 


Error 2: The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object 

Cause: Duplicate value for the desired columns

Solution: Run the following query to check the dup key, assuming unique index is on col1

select Col1 , count(*)
from Customers 
group by col1
having count(*) > 1

How to use table name as a variable in stored procedure

Sometimes it would be nice to pass the table name as a variable in a select query, its possible by using dynamic query as shown below

create procedure
declare @tablename varchar(30)


create procedure dbo.Test
@tablename varchar(30),
@val int
as

DECLARE @cmd AS NVARCHAR(max)

SET @cmd = N'SELECT * FROM ' + @table_name +
' WHERE Column1 = ''' + @val +  ''''


EXEC sp_executesql @cmd


exec Test

How to series TSQL 2

6. How to find second highest

SELECT COL 
FROM TABLE 
ORDER BY COL DESC
OFFSET 1 ROWS
FETCH NEXT 1 ROWS ONLY;


7. How to use multiple cte

;with cte1
as
(select col1, null as col2 from Tab),

cte2
as 
(
select col2, null as col1 from Tab
)

select * from cte1
union
select * from cte2

8 How to rename column
EXEC sp_rename 'TABLENAME.OLD_COLUMNNAME', 'NEW_COLUMNAME', 'COLUMN';
.

How to series TSQL 1

1. How to change col datatype

-- TransDate was in datetime change to date

alter table Transactions 
alter column TransDate date


2.  How to create primary key on a col

alter table TransactionsArchive 
add constraint  PK_TransactionsArchive_KeyID primary key clustered (key_ID);
GO

3.  How to re-sequence an identity column


alter table List_NotesType
drop constraint PK_List_CommentsType

alter table List_NotesType
drop column NoteType

alter table List_NotesType
add NoteType tinyint identity(1,1)

alter table List_NotesType
add constraint PK_List_NoteTypeID primary key clustered (NoteType);

4. How to change a not null col to null

alter table [Notes] 
alter column Comments varchar(500) NULL

5. How to update existing NULL cols with a value

update Notes 
set Comments = t.Description
from Notes n join List_NotesType t on n.Type = t.NoteType
where Noteid in (select Noteid from Notes where Comments is null)