Last year, Google’s Felipe Hoffa uploaded complete data about Hacker News’ posts to Google’s BigQuery data engine. BigQuery is a giant clustered SQL engine in the cloud that can query enormous amounts of data very quickly. Felipe did some great initial analysis of the dataset writing SQL by hand and graphing in Matlab.

Others on Hacker News picked up the thread and added their own analyses on Hacker News (meta, right?). I loved the idea of exploring this public dataset and wanted to surface some new insights, and also to make it accessible to anyone who wanted to play with the data, whether or not they knew SQL.

So I wrote up a quick model in LookML (Looker‘s YAML-based modeling language) to describe how the BigQuery tables relate to each other. (It’s all of four files and fewer than 300 lines of code; you can view it here).

Below, I’ll walk through the process of building out the

 Data Hacking: Coding up a Recommendation Engine from Simple Playlist Data

I love Pandora. Type in an artist’s name and it starts playing similar stuff. Pandora’s recommendation engine feels like magic.

BigQuery provides a sample data set of some playlist data (Google’s @felipehoffa says the original data set was created by @apassant, awesome data!). The data is very simple: a single row for each track in the playlist. Track data contains playlist_id, artist (id and name), album (id and title) and track (id and title).

Using this simple data, we built a recommendation engine in Looker’s LookML that takes an artist, finds the most related artists and then recommends a playlist, all in about 300 lines of LookML.
View the code on Github

 First, Go Ahead, Play With It

Change the filter to your favorite artist and based on this data, we’ll recommend some songs.
Click on the artist

It’s more than halfway through January and we still have our Christmas tree up in the living room. The tree is dead. It doesn’t look dead, but it is. It has been cut off from its root system, but to all of us, it looks very much alive. It’s bright color and crisp smell deceive us–we are made to believe it is still alive when it is not. This is because trees live on a very different time frame than humans do. They live longer; they die slower. So when you kill one, it takes longer for it to look dead.

Dying, it turns out, has time scale.

 Engineering teams die too

In a sense, engineering teams die much the way that Christmas trees do. Young engineering teams grow fast and change the what they build even faster. As they get bigger, the rate of visible progress slows. Eventually, something happens–a valuable person leaves, someone takes over–and the team becomes stagnant. The team

In the Princess Bride, the hero Westley is captured by the Dread Pirate Roberts and put to work on his ship. Each day, Westley works hard and impresses the pirate. At the end of every day, the pirate says “Good night, Westley. Good work. Sleep well. I’ll most likely kill you in the morning.” This exchange goes on for years. Eventually, the Dread Pirate Roberts acknowledges Westley’s contribution, makes Westley the new Dread Pirate Roberts, and retires with a fortune to a small island.

At Looker, I’ve run into the Dread Pirate Roberts professionally.

Early at Looker, in order to win business, we built many proofs of concept. We encountered customers who were helpful and enthusiastic, but sometimes we’d encounter the Dread Pirate Roberts (DPR).

In the service world, the DPR expects you to fail. They’re giving you a shot because you’ve promised something they want, but they’re skeptical

(originally published on the [first round review]([http://firstround.com/article/What-making-partner-means-at-a-startup]))

“It’s easier to ask forgiveness than permission.”

— Grace Hopper

When he led the Netscape Browser team, Tom Paquin used to repeat this quote all the time. I was right there with him, on a large team of very smart people operating autonomously. This was Tom’s hands-off way of saying, “I trust you to figure stuff out, and if you blow it, I’ll forgive you.” And other leaders I’ve admired shared this confident, laissez-faire approach, too — and it worked.

When I left Netscape for LiveOps, then-CEO Bill Trenchard started introducing me as his ‘partner.’ And immediately that moniker felt right. Even though the company was organized as a traditional corporation, the driving force among management was partnership — partners who valued and trusted each other. Bill

As a kid, I used to read a lot of science fiction. Often these stories followed some design pattern or other, one of my favorites were stories patterned on The Monkey’s Paw. In the Monkey’s Paw, the main character in the story receives a magical object (the monkey’s paw) that enables him or her to make three wishes. The holder of the monkey’s paw makes a wish, the wish is granted, but the wish has unintended (and usually very bad) consequences. For example, in one story, the wisher asks for riches, and someone he loves dies and he’s left with the insurance money. Ultimately, the wisher regrets what he asked for.

 Focusing on a single metric is almost always a Monkey’s Paw.

A very common mistake is to pick a metric and wish for it to improve. It’s great to have a single number to rally around, but it’s almost always the exact wrong thing. Focusing on a single metric will often have

It turns out that having language to describe something in a nuanced way really does promote discovery and discourse. Creating good names for measures and patterns lets you describe things quickly and efficiently. The better your language, the better your conversations. The better your conversations, more likely you’ll be able to make an important discovery.

Take ‘conversion’ for example. It usually means the percentage of leads that turned into orders, but ‘conversion’ is a crappy name. Which leads? All the leads? Net of bounces? Leads where people spent more then 10 seconds on the site? Leads where someone actually had something in their shopping cart?

Measure them all, pick good names and then pare it down to the measures to the ones that actually tell you something interesting.

Don’t be afraid of doing a little inventing here, but don’t do it alone. It’s about

The Basic programming language is a really nice abstraction for a physical computer. As a language it has the notion of an instruction pointer (line numbers) and a stack pointer (using GOSUB). The entire language description fits on a page. A great way to teach it is using the javascript Apple II Basic emulator (which is frankly amazing).

http://www.calormen.com/jsbasic/

Good assignments are:

• Print “Hello World”
• Write a program that takes two numbers as INPUT and adds them
• Write a program that prints all the integers less then 100
• Write a program that prints all the integers less then 100 by powers of two.
• Write a program that takes a number as input and then counts up by that number (for example if you entered 5, it would count 5, 10, 15 …)
• Write a guess a number game (it picks a random number between 1 and 100, makes you guess until you get it telling you if you are too high

When I do something every day I don’t mind a little complexity.

When I do something rarely, I’d like that task to be a simple as possible.

I program every day. Sublime, my text editor, is pretty complex. Even something basic, like using spaces instead of tabs, is an effort for me to find. In order to figure it out, I might dig through the menus, the help and even the google. But I love Sublime anyway. Sublime’s complexity works for me because it can do all thing things I might want it to do (and much more). It does come a cost (my time to figure out out).

I’m not willing to go to something simple, like OSX Text Edit.app, because the features are simply too basic and can’t do the job for me.

On the other hand, system preferences on OSX are pretty simple (as opposed to say trying to do the same thing in the unix shell). For the most part, I can figure out how to do what