The big Hadoop news of the week is that Oracle has partnered with Cloudera to bring their Hadoop expertise to Oracle’s Big Data Appliance. As Computer World notes the prevailing wisdom had been that Oracle would put together their own distribution, and it may seem surprising that the world’s largest database vendor would use someone else’s database software. The next year will certainly be interesting – is this a try before you buy move for Oracle? Or is it rinse and repeat of what they did with RedHat; partner first, then try to take them out?
Anyway, the net of all of this for Quest and Toad for Cloud Databases is positive. Quest users want to be assured that Toad has them covered whatever the database landscape looks like and however it changes. We brought our product to market early, and have broad support for the Hadoop ecosystem with HBase and Hive support, as well as having partnered with Cloudera in 2010 on the Quest Data Connector for Oracle and Hadoop , a high-speed data connector to move data between Oracle and Hadoop that unlike Oracle’s Hadoop loader enables you to move data in both directions. We also have support for other systems – Cassandra, MongoDB, Amazon, and Microsoft SQL Azure and Azure Table Services. In 2012 we’ll be adding support for Oracle’s noSQL database and a couple of others that I’ll write about as our roadmap gets firmed up.

Here’s a run-down on my professional activities over the course of 2011:

• Articles: 3
• Conference Spoken: 14
• Customer Calls: 124
• Customer Visits: 7
• Magazine Columns: 14
• PASS Chapter Presentations: 12
• Pre-cons/Full-day Seminars: 7
• SQL Saturdays: 4
• SSWUG Sessions: 8
• Webcasts: 16

Plus, I got to got on an awesome SQLCruise and was featured on Richard Campbell’s RunAsRadio show at least once (Richard’s blog | twitter).  (I was thinking that I’d been on twice in 2011. But that other appearance may have been in late 2010. My records aren’t clear.)

I was also put in charge of the SQLServerPedia portion of the DBPedias sites.  Some statistics there:

• 133 contributing bloggers
• 4,500 blog posts added in 2011 (out of a total 9,000 blog posts)
• 57,000 content items added in 2011 (out of a total 138,000 content items)
• Monday-Thursday all Pedias average 8,700 visits combined
• 1.9 million visits in 2011 (out of a total 3.8 million visits to Pedia sites since SQLServerPedia was started in 2008)

2011 was also my year to jump into Twitter.  By years end, I had accumulated:

• 3,452 Tweets
• 531 Following
• 2,656 Followers
• 230 Listed

I think that my increase in tweets had a direct correlation on my decrease in blog posts.  Ironically, I have accumulated even more topics to blog about (I’ve somewhere around 630 nascent blog posts), but simply run out of time to put them into WordPress.  My blogging activity for 2011 was down to 77 entries, about half what I wrote in 2010.

I’ll talk a little about my plans for 2012 in another post.  I hope to see you following me on Twitter soon!  Thanks,

Kevin

Hive is a SQL-like language which compiles to Java map-reduce code, while Pig is a data flow language which allows you to specify your map-reduce data pipelines using high level abstractions. 

The way I like to think of it is that writing Java MapReduce is like programming in assembler:  you need to manually construct every low level operation you want to perform.  Hive allows people familiar with SQL to extract data from Hadoop with ease and – like SQL – you specify the data you want without having to worry too much about the way in which it is retrieved.  Writing a Pig script is like writing a SQL execution plan:  you specify the exact sequence of operations you want to undertake when retrieving the data.  Pig also allows you to specify more complex data flows than is possible using HQL alone.

As a crusty old RDBMS guy, I at first thought that Hive and HQL was the most attractive solution and I still think Hive is critical to enterprise adoption of Hadoop since it opens up Hadoop to the world of enterprise Business Intelligence.  But Pig really appeals to me as someone who has spent so much time tuning SQL.  The Hive optimizer is currently at the level of early rule-based RDBMS optimizers from the early 90s.  It will get better and get better quickly, but given the massive size of most Hadoop clusters, the cost of a poorly optimized HQL statement is really high.  Explicitly specifying the execution plan in Pig arguably gives the programmer more control and lessens the likelihood of the “HQL statement from Hell” brining a cluster to it’s knees.

So I’ve started learning Pig, using the familiar (to me) Oracle sample schema which I downloaded using SQOOP.   (Hint:  Pig likes tab separated  files, so use the –fields-terminated-by ‘\t’ flag in your SQOOP job). 

Here’s a diagram I created showing how some of the more familiar HQL idioms are implemented in Pig:

Note how using Pig we explicitly control the execution plan:  In HQL it’s up to the optimizer whether tables are joined before or after the “country_region=’Asia’” filter is applied.  In Pig I explicitly execute the filter before the join.    It turns out that the Hive optimizer does the same thing, but for complex data flows being able to explicitly control the sequence of events can be an advantage. 

Pig is only a little more wordy than HQL and while I definitely like the familiar syntax of HQL I really like the additional control of Pig.

These stories often turned out to be more interesting than the question or the answer in and of themselves.  For example, it’s a common public speaking best practice to repeat a question back to the attendee.  This helps ensure that you fully understood the question and, in case of a session recording that’s picked up only on the microphone, that the question is also recorded.  But when you’re immediate response, as the speaker, is “Your manager told you to do WHAT?!?”, you know you’ve hit a zinger, as in “Your manager told you that backups aren’t important?!?”

These stories came to be so fun, in the time-honored tradition of slowing down to carefully examine a car wreck on the highway to the point of clogging all other traffic, that I started to make IT Horror Stories a part of my regular presentation portfolio.  And I never have to repeat myself since something new and horrible aways seems to be happening and, in many situations, conference attendees specifically seek out these sessions just so they can air their grievances.

Want to share your IT Horror Story?  I’ll give you a free eBook for any that I post here!

In our first installment of IT Horror Stories, I bring you a little lesson from my friend and coworker, Richard Douglas (blog | twitter), a SQL Server enthusiast living in the Maidenhead UK region.  Richard writes:

The background story is that I was in a meeting with a few managers and they announced (as they tend to do) that in 20 minutes they were going to start UAT’ing on a machine I hadn’t heard of (let’s call it PC101) I asked what this
machine was as it wasn’t listed on my last estate audit using MAP (Ed: the Microsoft Assessment and Planning too, found here. I wrote it about on my SQL Server Pro magazine Tool Time column).

The manager told me that it was just a PC not a server with one spindle and only 2GB of RAM on Win 7 32bit OS to hold a suite of databases with a total size of 300GB with TDE enabled to boot – and they were going to be doing user testing on this!!!

I told them there was no way that this machine was going to be usable and the users would take a bad view of the new features because of the poor performance.  So I was given the challenge of doing what I could to improve performance - in 15 minutes.

Straight away, I rushed over to desktop support to see what spare machines they had lying about.  Luckily, they had some spare machines for new starters.  So I managed to grab a bit of extra RAM and a hard drive from another machine. We had trouble attaching the extra drive into the machine.  It just wasn’t going to fit.  So we ended up putting the drive on top and taped it on so it wouldn’t get knocked. All the log files were moved to the second drive to try to eliminate some of the disk contention and we also added a USB flash drive to make use of Readyboost.

Of course, the users still complained about performance.  But I like to think that we helped things a little and it’s a great story of British ingenuity!

Manager: Of course we can get top-of-the-line performance with a little PC under a desk somewhere with minimal RAM, CPU, and IO capabilities.

ITPro: Are you kidding me? We might’ve been able to make it fast if we’d done a little planning beforehand. But this is rolling out RIGHT NOW!

Manager: Well, see what you can do with it.

ITPro: Ok. What can I spend to upgrade components?

Manager: Nothing.

ITPro: Gurgle… < Makes clutching motion at throat as if dying>

Follow me on Twitter! Enjoy,

-Kev

Amazon Elastic Map Reduce is a cloud-based version of Hadoop hosted on Amazon Elastic Compute Cloud (EC2) instance.  Using EMR, you can quickly establish a cloud based Hadoop cluster to perform map reduce work flows. 

EMR support Hive of course, and Toad for Cloud Databases (TCD)  includes Hive support, so let’s look at using that to query EMR data.

Using the Toad direct Hive client

TCD direct Hive connection support is the quickest way to establish a connection to Hive.  It uses a bundled JDBC driver to establish the connection.

Below we create a new connection to a Hive server running on EMR:

1. Right click on Hive connections and choose “Connect to Hive” to create a new Hive connection.
2. The host address is the “Master” EC2 instance for your EMR cluster.  You’ll find that on the EMR Job flow management page within your Amazon AWS console.  The Hive 0.5 server is running on port 10000 by default.
3. Specifying a job tracker port allows us to track the execution of our Hive jobs in EMR.  The standard Hadoop jobtracker port is 50030, but in EMR it’s 9600.
4. It’s possible to open up port 10000 so you can directly connect with Hive clients, but it’s a bad idea usually.  Hive has negligible built-in security, so you’d be exposing your Hive data.   For that reason we support a SSH mode in which you can tunnel through to your hadoop server using the keypair file that you used to start the EMR job flow.  The key name is also shown in the EMR console page, though obviously you’ll need to have an actual keypair file.

The direct Hive client allows you to execute any legal Hive QL commands.  In the example below, we create a new Hive table based on data held in an S3 bucket (The data is some UN data on homicide rates I uploaded).

Connecting Hive to the Toad data hub

It’s great to be able to use Hive to exploit Map Reduce using familiar (to me) SQL-like syntax.  But the real advantage of TCD for Hive is that we link to data that might be held in other sources – like Oracle, Cassandra, SQL Server, MongoDB, etc.

Setting up a hub connection to EMR hive is very similar to setting up a direct connection.  Of course you need a data hub installed (see here for instructions), then right click on the hub node and select “map data source”:

Now that the hub knows about the EMR hive connection, we can issue queries that access Hive and – in the same SQL – other datasources. For instance, here’s a query that joins homicide data in Hive Elastic Map Reduce with population data stored in a Oracle database (running as Amazonn RDS:  Relational Database Service).  We can do these cross platform joins across a lot of different types of database sources, including any ODBC compliant databases, any Apache Hbase or Hive connections, Cassandra, MongoDB, SimpleDB, Azure table services:

In the version that we are just about to release, queries can be saved as views or snapshots, allowing easier access from external tools of for users who aren’t familiar with SQL.   In the example above, I’m saving my query as a view.

Using other hub-enabled clients

TCD isn’t the only product that can issue hub queries.  In beta today, the Quest Business Intelligence Studio can attach to the data hub, and allows you to graphically explore you data using drag and drop, click and drilldown paradigms:

It’s great to be living in Australia – one of the lowest homicide rates!

If you’re a hard core data scientist, you can even attach R through to the hub via the RODBC interface.  So for instance, in the screen shot below, I’m using R to investigate the correlation  between population density and homicide rate.  The data comes from Hive (EMR) and Oracle (RDS),  is joined in the hub, saved as a snapshot and then feed into R for analysis.  Pretty cool for a crusty old stats guy like me (My very first computer program was written in 1979 on SPSS).

Date: Thursday, December 15
Time: 10 a.m. PST / 1 p.m. EST
Duration: 1 hour
Register:  http://bit.ly/v90vfC

Ad-hoc queries have remained basically the same over the years, but query and analysis tools have changed significantly. In this educational webcast, we’ll explore the latest advancements in that area and the one tool that offers them all. You’ll see how Toad for Data Analysts will dramatically simplify your work with its ability to rapidly connect you to all your data sources. Our product expert will also show you how it accelerates reporting and analysis – and increases accuracy.

Plus, you’ll get in on some cool tricks that deliver significant time savings. We’ll reveal easy ways to:

• Connect to multiple heterogeneous databases, spreadsheets, and BI sources from a single tool
• Develop high-performance heterogeneous SQL using Toad’s powerful SQL editors
• Profile, cleanse, and stage data faster than ever for rapid analysis
• Automate and schedule reporting workflows to eliminate repetitive tasks

Registration date and time information:
Thursday, December 15, 2011 10 a.m. PST / 1 p.m. EST

Don’t have your own copy of Toad for Data Analysts? Here’s your chance to download the leading SQL query tool on the market today that is capable of everything from data profiling, data mining to predicitive analytics. Get it here for free for 30 days.

Jason Bovberg, senior editor at Windows IT Pro said, “Our Windows IT Pro, SQL Server Magazine and DevProConnections awards programs are unique in that the product-nomination process is open to readers. We don’t just present a predefined list of products and services, thereby limiting scope. Instead, the community both nominates and votes for the best products of the year, ensuring a nice breadth of inclusion in the surveys. The winners of these awards are truly the community favorites. Our winners have earned a unique honor to stand out among their peers as winners of our Community Choice Awards.” The article can be read at http://www.sqlmag.com/content1/topic/2011-sql-server-magazine-editors-community-choice-awards-140830/catpath/awards/showprivate/1/page/4

In addition to taking the gold, the editors at Windows IT Pro magazine also awarded Toad with a silver medal in the 2011 Editors’ Best Awards. Jason Boverg also said, “Our Editors’ Best Awards let us leverage our contributing editors’ expertise to provide well-earned recognition to products that exceed industry standards.”

And if that wasn’t enough, Toad was also named Runner Up for Best SQL Server tool at the Fall Connections 2011 show.

We are thrilled by the overwhelming support for Toad in the SQL Server community coming off the latest release of Toad for SQL Server 5.6. If you haven’t downloaded it yet, then I highly suggest you download the latest version for free for 30 days.

THANK YOU SQL SERVER COMMUNITY!

Toad for SQL Server version 5.6 provides several customer fixes and minor enhancements. You can learn more about this release in the Release Notes or just try the product for free for 30 days.

Toad for SQL Server 5.6 adds the following new features and enhancements :

All editions:

• Code Completion enhancements
• Enhanced Index Management functionality
• Ability to apply security settings across multiple servers
• Several Connection Manager enhancements including support for Central Management Servers.
• Support for XML in the editor and XML PATH in Results Set window
• Support for Snapshots
• User interface and performance improvements
• Improvements to Object Browser
• Support for TFS 2010

Xpert edition and Suites:

• SQL Optimizer for SQL Server version 7.1 is included.
• Virtual Indexes can now be created in User Defined Alternatives
• Support for Functional-Based Indexes

Development and DBA Suites:

• Benchmark Factory version 6.6.1 is included which provides bug fixes to version 6.6.0 (which introduced replay for SQL Server)

DBA Suite (Not included in N. America):

• Integrated single installer is now available.
• Includes Spotlight on SQL Server Desktop Edition version 8.0
• Includes Toad Data Modeler version 4.1.5

What’s New in Toad for IBM DB2 5.0

The latest release of Toad for DB2 offers powerful new features and enhancements that’ll make your database development and administration jobs easier than ever. This latest version of Toad was released at the IBM Information on Demand conference in Las Vegas and provides many new time-saving benefits for both z/OS and LUW users:

• Improved Support for DB2 LUW 9.7 and DB2 z/OS 9.1 and 10 – Get more support for new objects and new object attributes that were introduced in the latest DB2 versions. DB2 LUW pureScale (9.8) is now supported as well.

• DB2 z/OS Space Management – Gain the insight you need to efficiently manage your objects with our new space usage and historical space reports. The new real-time statistics advisor identifies objects in need of maintenance.

• Test Data Generation – Populate your LUW or z/OS tables with test data using the new data generator. This tool is extremely useful and makes testing easier than ever.

• HADR Administration (New for DB2 LUW) – Use the high availability/disaster recovery feature to get a view of your configuration and start or stop HADR servers.

• Other Enhancements – Take advantage of updates to Code Completion, Query Builder, Editor, Code Snippets, and the Dependency Tab. Use the new Histogram Data tool to analyze the distribution of your data values.

Check out all these great features and more when you download your free 30-day evaluation. Or visit us on ToadWorld.com to pick up your Toad for DB2 freeware edition or learn tips and tricks from our pros.

Don’t miss this live podcast with DM Radio on 11/10/11

Remember ‘ad hoc’ queries? They’re everywhere these days. One could even argue that Google revolutionized the concept of querying data sets. Enterprise data is obviously different from Web data, but many of the same principles do apply. Register for this episode of DM Radio to hear Hosts Eric Kavanagh and Jim Ericson as they interview veteran Analyst Wayne Eckerson of TechTarget along with guests John Joseph of Endeca, Ash Parikh of Informatica, and Peter Evans of Quest. We’ll talk about how to empower business people with the kinds of insights they need, and in a timely, cost-effective way. That includes some new and interesting techniques for achieving agility.

Quest offers a powerful tool for ad hoc queries. Download your 30 day trial and test drive all the new features like cross-platform queries, data cleansing, advanced analytics, data visualization, embedded database, visual query builder and more.