Tech at Mintel: Creating a Development Pipeline for Looker

The Tech at Mintel series shines a spotlight on our global IT department and how some of the industry’s best and brightest build Mintel’s modern, agile, and scalable technology backbone.

At Mintel, our most valuable resource is the expertise of our team of analysts. We know anyone can provide you with data, but it’s the knowledge and insights that our experts provide that set us apart in our industry. So on the engineering team, we’re always looking for new and better ways to make that journey from raw data to expert insight in the hands of our clients as fast and seamless as possible.

That’s why an important piece of architectural guidance we follow is, “buy it, don’t build it.” If there’s a third party tool that can make what we’re trying to do easier, we’d rather use it than spend time and resources creating something from scratch, getting value to our clients that much faster.

One third party tool that we’ve been using to improve how we get expert analysis to our clients is the data visualization suite Looker. Compared to past projects we’ve built from scratch, creating data visualizations in Looker is not only faster and easier, but it lets our analysts participate more closely in the creation of the final deliverables that are included in client-facing products. It also eliminates the friction that comes from putting our web dev team in between the data experts and the end users. In other words, we avoid the game of telephone that comes from analysts explaining what they want to us and then having to translate that into a final client-facing product.

Looker does this, firstly, by providing analysts a direct and simple connection to our raw data. Our Looker instances are connected to our data warehouse and can query it directly without the need to replicate anything to another database that our web apps can query (a strategy we’ve used in the past). This gets the data in front of our experts quickly and easily, and lets them start playing with it without an engineering team even needing to be involved.

Secondly, Looker provides an easy-to-use interface for building visualizations without writing any code. This means our experts can build what they want clients to see, instead of describing what a visualization should include and leaving it to an engineering team to make their vision come to life. These visualizations created in Looker can then be embedded directly into a live web app, so what the analysts create can be exactly what we present to clients. 

The problem for the engineering team then becomes getting this work into production in a way that conforms to engineering standards for safety, accuracy and consistency. We need source control, a defined deployment process and automated checks all along the way.  While the ease of creating and editing Looker dashboards is a great feature, it can also be a drawback when it comes to keeping a client-facing app stable. We don’t want a few clicks in Looker’s UI to break an app that our clients are depending on.

To solve this problem we’ve created what we call our LookML Pipeline. At the start of this pipeline, our analysts can use Looker to easily create User Defined Dashboards (UDDs) right in Looker’s own UI. Then these UDDs are used by a set of processes and tools that allow us to treat them as though they were the source code of a traditional app. This means they can be edited and experimented on freely in their development environment without affecting the state of the application in production. When it is time to deploy those changes to production, it can be done in an easy but safe way that minimizes the possibility of mistakes.

The first step in this process is to create a Looker project using a tool we wrote called the LookML Cutter, which not only generates a repo for the new project based on a template, but also creates a project on our Looker development instance that’s linked to that repo, so any changes made within Looker immediately have a designated home in source control. The repo and the Looker project store their configuration, data models and dashboard definitions as LookML, Looker’s config file format (it’s very similar to .yaml). Also included within the newly created project repo are CLI tools and CI/CD configuration used by the rest of our pipeline.

These CLI tools come in handy once an analyst has created some UDDs for the new project and is ready to make those changes “official.” When we were first exploring using Looker, this required some cumbersome and tedious work from the analyst:  converting their dashboard into LookML using the Looker UI, and then copy-pasting that text into a project repo. Obviously this is not ideal; Ctrl+C and Ctrl+V are magical tools but also a great way to introduce mistakes. We also discovered that a big disadvantage of this approach was the potential for the state of UDDs and pasted LookML to drift away from each other. For example, if a change is made in the LookML that’s actually being used for the production project, and subsequently a change is made in the UDD, converting and pasting that UDD would wipe out the previous change that was made in the LookML.

Thankfully, our CLI tools now automate this process, and one command can safely get the LookML for a dashboard, create a merge request in GitLab for the changes and even run validation tests within Looker that ensure the changes are valid LookML. We’ve also solved the problem of drift by declaring that the original source UDDs should always be the source of truth for dashboard configuration, and by disallowing changes to be directly made to LookML by turning off Looker’s write access to its project repos. This means changes can only be made to the dashboard LookML files by running the command to generate LookML from the source UDD. 

We’ve also automated the deployment process that delivers these approved changes to their final home on the Looker production instance that’s used by our live apps. This is another task that can be done through the Looker UI, by clicking buttons to merge branches, validate changes and deploy — but we found doing this manually to be both inconvenient and error prone. So we lower the possibility of human error by integrating the Looker API with our CI/CD pipelines that run in Gitlab. When changes are merged to master, a pipeline job automatically deploys those changes to our Looker instance that’s designated for QA, so the changes are immediately visible on our beta apps. When a new release version is tagged, the pipeline job automatically deploys to the production looker instance, releasing the changes to our clients. The theme that you’re hopefully seeing here is that we’re doing our best to make the development of these dashboards as close to the standard process we’d use for deploying code changes to any production application.

It’s clear that using Looker makes development of data visualization applications fast and easy, and the LookML pipeline we’ve created to standardize and automate that development has only made it faster and easier, as well as safer. Using a tool out of the box is good, but making that tool a seamless part of your system is even better.