Your single source for new lessons on legal technology, e-discovery, and the people innovating behind the scenes.

How to Judge a Hackathon: 4 Criteria to Picking a Winner

Brendan Ryan

Judges for Relativity’s semi-annual hackathon, which took place earlier this month, were not left to their own devices and dispositions to determine who among their friends and colleagues would win.

In proud Relativity spirit, we developed a rubric.

Our panel of six judges—Relativity Developer Partner Co-Founder and former Relativian Charlie Connor as well as Relativians Nate Noonan, Rene Laurens, Ann Steimel, Alex Moy, and the humbled author of this blog post—based their decisions on four areas of measurement: business value, coolness, realistic capability, and level of innovation.

This post will explain each of these criteria, with insight on how you might use them to plan a hackathon for your own organization. But first, a bit on why you might plan a hackathon, and why they aren’t just for software engineers.

Unlock the Creativity of Your Team

“Get your creative juices flowing and talk to a lot of people. Don’t just stick with engineers. Go to sales folks, go to marketing folks. Heck, I went to a lawyer. Go wide.” — Sarmad Qutub, Relativity cloud operations technical product manager, hackathon participant

For this hackathon, we made an effort to include Relativians without backgrounds in software development. While the whole point is to build a functioning application or feature set on the Relativity platform, our hypothesis was that we’d see even stronger innovation by encouraging professionals from throughout the organization to participate. That turned out to be correct, and we’ll likely go even bigger next time.

In addition to swag prizes, hackathon winners are considered for a future spot on our product roadmap. For instance, Relativity’s first mobile application was the result of a hackathon project. Involving operations as well as sales and other customer-facing experts spreads the learning, and increases the odds that the solutions emerging after two caffeinated days of brainstorming and coding will be worth bringing to the market. It improves their chances of ranking high in our four key areas.

The Criteria

1. Business Value

“I’m getting to hear a lot of interpretations of things I’m coming across from people who have experienced them before. And I’m getting the chance to build a lot of unique relationships and just gain perspective on the way that we could be doing business differently.” — Al Weinstein, Relativity inside sales manager, hackathon participant

We rated “business value” and the other criteria on a scale from 1 to 5. How significant and relevant is the problem that the solution is trying to solve? Does that problem affect a market that we know and understand? Does it have a market?

2. Coolness

“I’m working on something that another team could, possibly, work on—but not my team. It’s really cool to try something else, for developers to get into a different code base, […] for our team to try things that make us feel somewhat uncomfortable.” — Peter Haller, Relativity Analytics technical product manager, hackathon participant

“Coolness” is an attempt to measure the important “wow” factor: the impression the hackathon entry makes when you see it, and the gestalt of the demo. Is it a reflection of a diverse team’s ingenuity? Is it something a human being would feel delighted, excited, empowered, or even relieved to use?

3. Realistic Capability

“Being able to work on something that’s not on the roadmap is actually really fun. And just trying to crank it out in two days is also enjoyable.” — Sam Solovy, Relativity Review advanced software engineer, hackathon participant

The two days we set aside for our teams to pause their normal course of work go by fast. Which teams built working solutions? Which bit off a little more than they could chew? What gets done in the space of a hackathon is a decent indication of the dependencies and level of effort that would be required to put the technology into production. This category is our way of quantifying that.

4. Level of Innovation

“It gives you perspective […] As a software engineer, you’re probably just coding. But when it comes to hackathon, you have to design everything—and you have to factor in the resources you have, and the time that you have—so that you can be a good manager of the time [and] an architect. So you have a lot of responsibility and roles at the same time.” — Srinivas Mothe, Relativity Analytics advanced software engineer, hackathon participant

Technology can be cool but not innovative—and vice versa. Apart from coolness, “level of innovation” is our take on participants’ awareness of other attempts to solve the same problem they are, and how the team positions their hackathon entry. Does it pull something off that hasn’t been pulled off before? Does it do something new and unique?

The Results

Judges used these criteria to anchor the discussion as we deliberated on who should win among about a dozen impressive participating teams. The winner: Relativity Lead DevOps Engineer Adam Kadzban’s “If You MSBuild It They Will Come”—a clever name for a team who built a way to simplify how the Relativity developer community builds applications on the platform to solve problems within and outside e-discovery. They integrated the process into MSBuild as a custom project type, saving our engineering team (and others) time and lowering the barrier even further to application development and distribution.

“The people” got to choose, too. We judges had our favorite, but from a closely watched vote emerged the most popular solution among civilian Relativians: “Hotlighting.”

When Peter Haller found out some customers’ case teams were printing out the names and information of various custodians, key players, and terminology to reference while reviewing documents, he envisioned a better way and enlisted the right help. Peter headed up a team to experiment with including contextual information on key terms, people, places—pretty much anything—in the platform’s persistent highlighting functionality. Such a feature would bring some of the value of Relativity Case Dynamics to the document viewer.

For more about our own hackathon story and the teams and technology described here, check out our coverage on Twitter.

Regardless of the criteria you use to judge the innovations of your team—technological or otherwise—remember that the spirit of a hackathon is a spirit of learning through collaboration and friendly competition. If you’re learning together, you’re doing it right.

“To learn a new technology or a new framework—the best way for me to learn something like that is to actually start making something with it.” — Josh Castor, Relativity Review software engineer intern, hackathon participant

Dive into our Data Discovery Law Year in Review for 2020

Brendan Ryan was Relativity’s senior manager of partner marketing.