Driving Self-service and Bettering DataOps with Atlan
The Lively Metadata Pioneers sequence options Atlan prospects who’ve lately accomplished an intensive analysis of the Lively Metadata Administration market. Paying ahead what you’ve realized to the subsequent knowledge chief is the true spirit of the Atlan neighborhood! In order that they’re right here to share their hard-earned perspective on an evolving market, what makes up their trendy knowledge stack, modern use instances for metadata, and extra!
Within the first interview of this sequence, we meet Heidi Jones, knowledge evaluation and program administration extraordinaire, who explains the historical past of Docker’s knowledge workforce, how they evaluated the market, and the way they’ll use Atlan to assist their colleagues drive one of many world’s finest developer experiences.
This interview has been edited for brevity and readability.
Would you thoughts describing Docker and your knowledge workforce?
Docker is a platform designed to assist builders construct, share, and run trendy functions. We deal with the tedious setup, so builders can give attention to the code.
Knowledge professionals at Docker assist quite a lot of completely different departments. So now we have a core knowledge workforce with engineers and analysts, after which we even have knowledge engineers and analysts that assist the foremost capabilities of Docker, reminiscent of Advertising, Gross sales, the completely different merchandise at Docker, Finance, et cetera.
A number of skilled knowledge engineers and analysts who’ve joined Docker, have solely began within the final 9 months or so. So we’ve had fairly a little bit of progress on the information workforce, and at the moment are at that stage the place we’re making an attempt to spend money on good processes. That manner, our knowledge workforce can be certain that everybody at Docker has the information that they should do their jobs, and might finally assist builders do theirs.
And the way about you? May you inform us a bit about your self, your background, and what drew you to Knowledge & Analytics?
I believe the primary purpose I’ve been drawn to knowledge and analytics is as a result of I similar to having the ability to reply folks’s questions.
I got here into knowledge evaluation by a non-traditional route. I’ve been at Docker for about six months now, however I’ve been within the knowledge house for a couple of decade. It began with Excel and offering insights through spreadsheets, as much as PowerBI utilizing Snowflake, that sort of factor.
So I used to be all the time an information analyst, however then additionally a mission supervisor. And so what I do at Docker combines each of these. The information of information and the workflows required to get good knowledge and supply good insights, and likewise the mission administration and operations facet of it. All of it permits knowledge professionals to give attention to what they do finest, which is modeling knowledge and offering insights with out being blocked by something that has to do with workflow.
What does your stack seem like? Why did you want an lively metadata answer?
We ingest knowledge from quite a lot of sources in a number of other ways, relying on the supply. After which our knowledge warehouse degree is Snowflake. Our modeling layer is dbt, that’s the place we do modeling and transformation. After which our essential BI instrument is Looker, that’s the place we do visualization and evaluation.
We had been only a one-person workforce not too way back. So all of that knowledge work was on one individual’s plate, together with documentation and understanding knowledge sources. That’s fairly a bit for one individual.
A whole lot of that burden has been unfold out throughout a number of folks on the workforce by now. However we’re making an attempt to maneuver away from, “Oh, let me go ask my favourite knowledge individual,” towards, “I can go verify this instrument and I do know there’s an authorized knowledge asset.”
And so, due to our stack, we had been drawn to Atlan due to issues just like the Looker Chrome extension plugin, the dbt integration, that sort of factor. As a result of proper off the bat we had been capable of say, “Okay, any descriptions we put in our dbt layer will mechanically be uncovered in Atlan.”
So non-engineering customers who wish to know what the information means can go straight to Atlan and see what’s being performed within the modeling layer.
Did something stand out to you about Atlan throughout your analysis course of?
Atlan is a really cool instrument that has a very good suite of options that we had been on the lookout for, however the differentiator actually got here all the way down to the folks at Atlan.
You demonstrated very competent understanding of the issues within the knowledge house and likewise very mature buyer assist. We might inform that your assist was not simply one thing you had been promising for us, however one thing that you simply had been already actively doing with different prospects.
We knew that it could be an actual partnership and that the shopper assist org was ready to assist the wants of a company like ours. And that maturity stood out to us once we made our resolution.
However then once more, additionally the options like Playbooks, the integrations that I’ve already talked about with dbt, with Looker, and simply the fixed innovation as nicely that we had been capable of observe even through the analysis processes, which I imagine took us about two months.
There have been a number of improvements and releases that occurred throughout that point interval and we might see the cadence the Atlan was on to constantly enhance. All of these had been promoting factors to us.
What do you plan on creating with Atlan? Do you could have an concept of what use instances you’ll construct, and the worth you’ll drive?
Our largest worth that we’re making an attempt to drive with Atlan is to make it possible for professionals at Docker can get the knowledge they want in regards to the knowledge that they should do their jobs.
We wish to transfer in direction of self-serve analytics and permit each knowledge professionals, and those that simply need to have the ability to use knowledge extra freely of their work, to have the ability to accomplish that with out having to get into the entire SQL and technical particulars of the information.
They know they’ll belief the information set, they know they’ll belief the information that they’re taking a look at, and so they can go forward and make their choices. In the end, it ought to assist us assist our mission of delighting builders, and creating instruments that they take pleasure in utilizing.
We’ll be supporting that with Atlan, and likewise supporting our knowledge engineering and analytics groups. They should have extra supported and standardized workflows, in order that they’ll give attention to modeling, actually digging in and doing what they do finest with knowledge.
Did we miss something?
That’s a very good query. I believe how we found Atlan was fascinating. I’ve been following Prukalpa, truly, for a few years simply as an information skilled, simply form of watching Atlan.
And so once I joined Docker, they had been already taking a look at knowledge catalog instruments, however hadn’t been taking a look at Atlan but. And I stated, “Effectively, how about Atlan? Ought to we take a look at Atlan as nicely?”
So one of many first issues I did at Docker was to start out up that dialog, and the rationale why I did that’s as a result of I had appreciated studying what she stated in these areas. Concerning the causes we want knowledge catalog instruments, and past only a catalog, the way it may very well be a part of knowledge operations. And that piece of it actually had spoken to me over time.
And we noticed some spectacular instruments. It’s a burgeoning house. There’s some nice instruments on the market. However I’m glad that we additionally checked out Atlan as a result of finally it had a very good mixture of what we would have liked at Docker.
Picture by Annie Spratt on Unsplash