11 anos de data lake e seu impacto no mundo dos dados

Exactly 11 years ago, the concept of a data lake (DL) came to light in a text by entrepreneur and business intelligence (BI) expert James Dixon, originally posted on his website, James Dixon's Blog.

Since then, the idea of a data lake has been increasingly disseminated and applied in the big data processes of companies around the world.

In this article, you will understand:

• how the need to create a data lake initially arose;
• the repercussion of the concept created by Dixon;
• the impact of the data lake on the world of data;
• and how to know if your business needs a data lake.

Have a good read! 😉

The need for a data lake arises

In mid-2010, James Dixon and his research team at Pentaho gathered important information about the day-to-day difficulties of companies dealing with a large number of data in their processes.

Through the research, the team noticed that:

• 80%-90% of companies worked with structured or semi-structured data;
• The source of this information was usually a single application or system;
• the data was typically subtransactional or non-transactional;
• The daily volume of data did not fit technically or economically into a database management system.

The data analysis and reporting processes of the time were concerned with focusing on the most interesting attributes, causing the information to be aggregated in a data mart and making a deeper investigation impossible.

Faced with these questions, Dixon and his research team at Pentaho realized the need for an architecture that would be able to house more and more data from different states and sources.

"Based on the results of the research, we came up with a concept called a data lake to describe an optimal solution" (Dixon, 2010).

How was the idea of a data lake received?

In July 2011, Dan Woods wrote for Forbes a rather positive review of James Dixon's newly created DL concept.

In the text, the CTO and technology consultant explains why big data processes require larger storage architectures, and how the data lake can solve this issue.

This new concept allowed analysts to answer even more specific questions for their clients due to the large volume of information that became available.

But as not everything is rosy, there were not so positive reviews of DL at the time. In this sense, TechTarget's text became, let's say, famous, leading Dixon himself to answer all the questions and doubts raised there in this article published in 2014 on his blog, where he revisits the history of the data lake and demystifies problems of the term.

The Impact of the Data Lake on Data Analytics

Before the data lake, data storage was even more limited, which also limited analytics and reporting consequently. With his arrival, a world of possibilities has opened up.

DL makes it possible to carry out agile and flexible processes, in addition to assisting in unpredictable situations. In this way, it becomes a fundamental and necessary resource for companies that deal with data.

For example, its architecture adds to the generation of relevant insights due to the volume of information. In the same way, it allows the collection, storage, organization, and interpretation of complex data on a large scale.

The data lake has revolutionized data processes because of its ability to offer very versatile analytics due to the amount of information available.

Does your business need a data lake?

If it is a business project, not an IT project, with the aim of generating real results for your company, then yes.

This is because the data lake, in this case, will be part of a modern and complete data platform, architected to meet the company's needs within the Data Driven Journey.

Contact us right now by clicking here and count on our specialized team to know where and how to start implementing a data lake in your business.

‍