Data Lake develops and implements cutting-edge technologies to help advance ethical medical research, and improve access to treatment.

Data Lake

datalake

Welcome to HackerNoon’s interview series. This section is curated by our Editorial team to spotlight bold and disruptive startups across the globe. If you’ve been nominated, and answer these questions Startups-Of-The-Year create a brand profile here. Hey Hackers, EU-funded startup Data Lake has been nominated in HackerNoon's annual Startup of the Year awards in Warsaw, Poland! We would be beyond grateful if you could vote for us here: https://hackernoon.com/startups/europe/europe-warsaw-poland Read more about us below to understand the amazing things we’re working on, and why we believe we deserve your vote. Meet Data Lake Founded by two medical doctors, is an EU-funded startup working to solve two of the biggest issues in medical research today: access to - and bias in - medical data. Data Lake Humanity is on the brink of some incredible developments in medical technologies and treatments - from AI algorithms detecting cancer to nano-robotics and smart sensors, we are looking at a future in which patient outcomes are hugely impacted by medical technology. Having access to high-quality and representative data, however, is of vital importance to the design, training, and implementation of these technologies. Up until recently, however, there was no way for researchers to legally and ethically access the datasets needed to build these incredible developments. Thanks to new EU legislation and our tools, we have created a way for researchers to do just that! Based on ethically-gathered digital consents, Data Lake enables patients to provide access to their medical data to the research entities and startups who are building our medical future. Along with the and more than 40 patient and healthcare organizations, we have that will soon allow anyone in the world to donate their medical data to science. Donate Your Data Foundation launched a global medical data donation platform My Role My name is Wojciech Sierocki, co-founder and CEO of Data Lake. I am a medical doctor by training but have previously founded several startups including ‘Treatment with a Mission’, at the time one of Poland’s largest NGOs operating in Africa. From the start of the COVID pandemic, I worked as a physician in the Emergency Department of the HCP Hospital in Poznan, Poland. While being able to make an immediate impact in someone’s life as a doctor was something I found immensely satisfying, seeing the impact of data on COVID outcomes I soon realized that my greater potential resided in building systems that could impact hundreds of thousands - if not millions - of people. Thus, along with my co-founder Dr. Ligia Kornowska, the idea for an international medical data donation program and Data Lake was born. How We're Disrupting the Medical Research Industry We have built an ethical medical data access layer that allows medical researchers (who are performing pro-patient research) to study large amounts of medical data that has previously never been studied. Based on the informed consent of patients who have the right to revoke it at any time, we can connect researchers to multi-source data sets that can be incrementally and regularly updated. This can take various forms, including the creation of privacy-focused patient registries that allow researchers to connect with patients suffering from ultra-rare diseases, as well as enabling a new era of digital clinical trials. Standing Out from The Crowd Securing access to medical data has been attempted - and failed - before. This has primarily been due to two factors - first, previous attempts have required patients and the general public to gain access to and then upload their medical data. For anyone who has ever tried to get their medical files, they’ll know just how difficult a proposition that is. Additionally, previous attempts have not sufficiently taken into account the aspects of privacy and transparency, often asking people to trust them regarding how and when their data is being used. We have solved both of these issues, using technology that allows for direct integration with hospitals for the transfer of medical records based on the express consent of patients, given digitally and recorded with distributed ledger technology. In short, we do all the heavy lifting by taking on both all of the legal and operational burdens. What really makes our approach different, however, is the focus on openness and transparency. We don’t want patients to have to trust that their consent (or revocation of consent) has been received, and therefore we chose blockchain technology to ensure a verifiable and trust-less system. By recording operations of consent on a public blockchain - without personally-identifiable information of course - we have removed the need for trust from the system entirely, allowing patients to know beyond a shadow of a doubt what their consent status is, and whether their medical data is being used for research or not. Our system not only respects patient agency and privacy, but it reduces the risk for researchers who know absolutely whether they have permission to study a dataset or not. Our Predictions/Thoughts on the HealthTech Industry in 2023 As with many industries, the healthcare industry will become increasingly reliant upon and driven by data in the coming years. A.I. algorithms, machine learning, and smart sensors are already online and assisting in the diagnosis and treatment of diseases, and with continuous and parallel innovations in nanorobotics and 3D printing, we are looking at a Science Fiction level of healthcare being possible within the next few years. Our main prediction - and something we are working to ensure - is that the ethical sourcing, storage, and analysis of data will become an increasing trend. Before we started and before key EU legislation was put in place, the idea of “consent-based data donation” was foreign to many people in the healthcare industry. Thanks in part to our efforts, we are hearing it talked about more and more at conferences and in the media. We believe that engaging people in what happens with their medical data and ensuring they are stakeholders is key to unlocking a new wave of medical innovation. We are convinced that you can have both privacy and innovation, and we believe the coming year will prove us right. What word defines the state of HealthTech in 2023? It’s hard to define in just one word, but we’ll give you three: “exciting, but critical.” The technological innovations we’ve talked about previously are incredibly exciting when you consider their potential for improving quality of life, patient outcomes, and ultimately, saving lives. We’re not exaggerating when we say we’re on the brink of being able to cure and treat many diseases previously considered unbeatable. But how we build these technologies and algorithms - particularly when it comes to the data we are using to create and train them - is of crucial importance. It’s why we’re working not just on the technological side of things, but also engaging on the ethics and rules for practical implementation of these exciting developments, such as my co-founder being a leading contributor to the . Whitepaper on AI in Clinical Practice Why we decided to participate in HackerNoon's Startup of the Year awards To be honest, one of our community members must have signed us up because it was a surprise to us! But a very pleasant one, because we believe that our technology and the idea of medical data donation have the power to save lives, and we want as many people out there to be aware of this growing movement. Final Thoughts We’re very thankful for the nomination and would like to thank everyone who has voted for us, and we hope that if you haven’t already you will consider doing so! We would also love for you to check out as well and sign up for the waiting list to become a medical data donor! Best of luck to everyone! https://donateyourdata.io Click here: Vote for us today!

Startups of the Year 2023: Data Lake - The Startup That Is Improving Access to Medical Research

The idea of an AI-driven robot taking over our healthcare – one of the most “human” fields that we have – is not a new one. Until recently, however, the idea remained relatively firmly in the realm of science fiction books and movies. But with the viral success of ChatGPT, the discussion of what this new generation of AI will be doing for us in the future has started to come up in the healthcare field. Now we’re not saying that you’ll be talking to a robot doctor at your next annual checkup any time soon, but we have to ask the question… Could ChatGPT be your doctor in the future? How Does ChatGPT Work? Without getting too deep into AI and Machine Learning, let’s define ChatGPT. The initials “GPT” stand for “Generative Pre-training Transformer,” three fundamental properties of the world’s most famous (currently) Artificial Intelligence that already provide a description of how it works: : More specifically than an AI, ChatGPT is a generative language model, a program that has been trained to hold natural conversations with humans. Generative : ChatGPT was pre-trained using the largest text dataset ever used for this purpose. Furthermore, the company behind ChatGPT, OpenAI, also trained its product through human feedback. Pre-Training : ChatGPT is built on transformers, which are a type of artificial neural network architecture that allows it to elaborate an input with the information it has been trained with and produce an output. Transformer How is ChatGPT Similar to Doctors? Simply put, ChatGPT must be trained with datasets in order to process inputs and generate answers that are understandable and accurate to humans. Sounds familiar? This process is similar to the path our society has outlined for anyone seeking to become a professional – in our case, a medical doctor. Doctors are among the professions that require the most years of university education, studying from books and through lessons (datasets) before they can finally talk to patients (receive inputs), provide their diagnosis, and recommend specific treatments (generate outputs). Only in its infancy, ChatGPT is already a remarkable product, but it is still far from being sufficiently advanced to trust with healthcare decisions or diagnoses. AIs require training on specialized data sets in order to perform specialized functions, and ChatGPT just isn’t trained for healthcare purposes. In fact, when we asked ChatGPT if it could provide medical diagnoses, it responded: “ .” No, I’m sorry, but as an AI language model, I am not qualified to make medical diagnoses. My programming and training are focused on understanding and processing natural language to provide information, answer questions, and assist with a wide range of tasks. However, I can provide you with general information on medical conditions and their symptoms, as well as advice on seeking professional medical help. If you have a specific medical concern, it’s always best to consult with a licensed medical professional who can properly diagnose and treat your condition For now, doctors all over the world can breathe easily. However, if a new version of ChatGPT was developed – one fed with a dataset of scientific information about healthcare and medicine that is thousands of times larger than any amount of material humans can effectively learn – it could mean a drastic and possibly rapid shift in how we receive our healthcare. If functionality included voice recognition and medical imaging machine learning, such a tool could not only perform initial patient assessments and triage but could potentially become a primary tool - or at least a supplementary tool - for diagnosis in hospitals and clinics. Human doctors will always have their advantages. Being people, they always have the capacity for empathy and provide the crucial human, emotional and psychological support that patients need to recuperate. Humans are also better at innovative ideas and improvisation, both of which are crucial in ERs, operating theaters, and many other areas of healthcare. But AI models like ChatGPT may soon be able to provide critical support to their work and may even be a potential solution to over-stressed healthcare networks and national services. The Shift to AI May Not be Seamless Nevertheless, the implementation of AI in our daily lives may not be as smooth as one might expect. For example, in order to have a trustworthy product, AIs must be trained with reliable medical data. Furthermore, several paradoxes may emerge from a new collaboration between humans and robots. For example, the AI may use its sophisticated algorithms and data analysis to come up with a diagnosis, whereas the medical doctor may rely on their professional experience and intuition to make a different diagnosis. Which diagnosis should take precedence if the doctor and the AI arrive at opposing conclusions? What happens if a doctor and AI team disagree with the diagnosis of another doctor and AI team? Even without a deep dive into the ethics and implementation, it is obvious that - before any AI integration occurs - experts and governments must discuss a robust framework to regulate scenarios like this one. For these and other reasons, such a change may appear radical or even dangerous to many, but when one zooms out and considers the incredible advancements in medicine over the last century, as well as the incredibly fast rate at which things continue to change today, we think it’s safe to assume that future generations will have AI robots like ChatGPT covering some diagnoses and taking care of their health on a daily basis. In less than 100 years, we’ve gone from the discovery of penicillin to custom-printed drugs created through genetics-powered personalized healthcare. In the last few decades alone, we’ve doubled cancer survival rates for almost every type of cancer. Nearly every discipline of our healthcare is seeing benefits from technology, as well as a shift towards data-driven, patient-centric medicine. AI will play a major part in the fourth industrial revolution, and both medical practice and research are already beginning to be more and more driven by these advancements. Multnomah County Hospital, 1948 ( ) vs. Modern Medical Facility ( ) Source Source Conclusion History repeats itself: as it happened with the introduction of computers and of the internet, AI and similar other disruptive technologies are inevitably going to cause rapid and profound changes in every aspect of our society. AI technology is already being used in many healthcare sectors to diagnose and treat illnesses, automate medical processes, to discover and print new drugs, and further advancements are likely to have a major impact on the way medical care is provided. is an EU-funded startup aiming to solve the issues of access to and bias in data in order to ensure these technological developments are equitable and trustworthy, and board chair Dr. Ligia Kornowska is also the leader of the “AI in Healthcare” Coalition. She was instrumental in the creation of the , discussing the benefits, concerns, and ethical guidelines required when it comes to the use of AI in healthcare. It will take many similar efforts from governmental institutions and citizen organizations to ensure AI is properly and ethically implemented, with sufficient oversight and a patient-centric focus. Data Lake “AI in Clinical Practice” whitepaper This is an exciting time for healthcare, and as we move towards a digitalized future, it is our duty as a society to prepare for the impact of these technologies and begin a discussion on how they can be useful for humankind in a safe manner. Could ChatGPT replace your doctor? Well, it’s certainly not going to happen today or any time soon. However, as the introduction of computers and the internet brought profound changes to the way we live, work, and interact, the introduction of AI is certain to have a similarly dramatic impact on medicine and healthcare. This article was written by the staff of Data Lake, an EU-funded startup working to provide ethical access to medical data for pro-patient research and development purposes. Also published . here

ChatGPT Could Have a Profound Impact on the Healthcare of the Future

Presto SQL + S3 Data + Superset = Data Lake

Architecting a Modern Data Lake in a Post-Hadoop World

Mä Moderno Datos Quta Arquitectura ukax mä Post-Hadoop Mundo ukan luratawa

Post-Hadoop Dünyasında Müasir Məlumat Gölünün Memarlığı

Arhitektura modernog jezera podataka u post-Hadoop svijetu

Arkitektering af en moderne datasø i en post-hadoop verden

Arquitectura de un lago de datos moderno en un mundo post-Hadoop

ポストHadoopの世界における最新のデータレイクの設計

თანამედროვე მონაცემთა ტბის არქიტექტურა პოსტჰადოპის სამყაროში

Хадоопын дараах ертөнцөд орчин үеийн мэдээллийн нуурыг барих нь

Go aga Letsha la Datha la Sebjalebjale Lefaseng la Ka morago ga Hadoop

Kubaka Ikiyaga kigezweho muri nyuma yisi ya Hadoop

Arkitektimi i një liqeni modern të të dhënave në një botë post-Hadoop

Pag-arkitekto ng Modernong Data Lake sa isang Post-Hadoop World

An Architect’s Guide to the Top 10 Tools Needed to Build the Modern Data Lake

আধুনিক ডেটা লেক তৈরির জন্য প্রয়োজনীয় শীর্ষ 10টি সরঞ্জামের জন্য একজন স্থপতির গাইড

Ein Architektenhandbuch zu den 10 wichtigsten Tools zum Aufbau eines modernen Data Lake

Guía para arquitectos sobre las 10 herramientas principales necesarias para crear un lago de datos moderno

Guide de l’architecte sur les 10 principaux outils nécessaires à la création du lac de données moderne

आधुनिक डेटा लेक के निर्माण के लिए आवश्यक शीर्ष 10 उपकरणों के लिए एक आर्किटेक्ट गाइड

現代のデータレイクを構築するために必要なトップ 10 ツールに関するアーキテクト ガイド

Um guia de arquiteto para as 10 principais ferramentas necessárias para construir o moderno Data Lake

Руководство архитектора по 10 лучшим инструментам, необходимым для создания современного озера данных

Modern Veri Gölünü Oluşturmak İçin Gereken En İyi 10 Araca Yönelik Bir Mimarın Rehberi

Hướng dẫn của Kiến trúc sư về 10 Công cụ hàng đầu cần thiết để Xây dựng Hồ dữ liệu hiện đại

架构师指南：构建现代数据湖所需的十大工具

Build a Data Warehouse Not to Drown in Your Data Lake

An Architect’s Guide to Building Reference Architecture for an AI/ML Datalake

এআই/এমএল ডাটালেকের জন্য রেফারেন্স আর্কিটেকচার তৈরির জন্য একজন স্থপতির গাইড

Ein Architektenhandbuch zum Erstellen einer Referenzarchitektur für einen KI/ML-Datalake

Una guía del arquitecto para crear una arquitectura de referencia para un lago de datos de IA/ML

Guide de l'architecte pour créer une architecture de référence pour un datalake IA/ML

AI/ML डेटालेक के लिए संदर्भ आर्किटेक्चर बनाने के लिए आर्किटेक्ट की मार्गदर्शिका

AI/ML データレイクのリファレンス アーキテクチャを構築するためのアーキテクト ガイド

AI/ML 데이터레이크용 참조 아키텍처 구축을 위한 설계자 가이드

Guia do arquiteto para construir arquitetura de referência para um Datalake de IA/ML

Руководство архитектора по созданию эталонной архитектуры для озера данных AI/ML

AI/ML Datalake için Referans Mimarisi Oluşturmaya Yönelik Mimar Kılavuzu

Hướng dẫn của Kiến trúc sư về Xây dựng Kiến trúc Tham chiếu cho Datalake AI/ML

架构师指南：构建 AI/ML 数据湖参考架构

The Architect’s Guide to A Modern Datalake Reference Architecture

Developing Next-Gen Data Solutions: SingleStore, MinIO, and the Modern Datalake Stack

নেক্সট-জেন ডেটা সলিউশন তৈরি করা: একক স্টোর, মিনিও এবং আধুনিক ডেটালেক স্ট্যাক

Entwicklung von Datenlösungen der nächsten Generation: SingleStore, MinIO und der moderne Datalake-Stack

Desarrollo de soluciones de datos de próxima generación: SingleStore, MinIO y Modern Datalake Stack

Développer des solutions de données de nouvelle génération : SingleStore, MinIO et la pile Datalake moderne

अगली पीढ़ी के डेटा समाधान विकसित करना: सिंगलस्टोर, मिनियो और आधुनिक डेटालेक स्टैक

次世代データソリューションの開発: SingleStore、MinIO、最新のデータレイク スタック

차세대 데이터 솔루션 개발: SingleStore, MinIO 및 최신 Datalake 스택

Desenvolvendo soluções de dados de última geração: SingleStore, MinIO e Modern Datalake Stack

Разработка решений для обработки данных следующего поколения: SingleStore, MinIO и современный стек Datalake

Yeni Nesil Veri Çözümlerinin Geliştirilmesi: SingleStore, MinIO ve Modern Datalake Stack

Phát triển các giải pháp dữ liệu thế hệ tiếp theo: SingleStore, MinIO và Datalake Stack hiện đại

开发下一代数据解决方案：SingleStore、MinIO 和现代 Datalake 堆栈

Many of us have made changes to data based on requirements that later evolve. By the time we realize it, it becomes impossible to roll back. Change isn't just a constant; it's an essential part of managing data that demands a sophisticated approach. Enter the Git-like functionality of Dremio's data catalog Nessie. Just as Git has become fundamental to software development, data engineers need similar tools to work in parallel, compare data versions, promote changes to production and roll back data when needed. Nessie provides data engineers with a Git-like version control system for managing data versions, branches, merges, and commits. This can be very helpful when multiple data engineers are working with and transforming data at the same time. Nessie allows each engineer to work in separate branches while maintaining a single source of truth in the form of the main branch. This functionality empowers data engineering teams to maintain data quality collaboratively in the face of unrelenting change. This article provides a step-by-step guide demonstrating how Nessie, Dremio and MinIO work together to enhance data quality and collaboration in your data engineering workflows. Whether you're a data engineer, ML engineer, or just a modern data lake enthusiast, this blog equips you with the knowledge and tools needed to effectively enhance your data versioning practices. Understanding the Basics: Nessie Workloads Nessie allows for Git-like workloads that let you test, develop and push to production. Let's break down some key concepts: Just like in Git, branches in Nessie allow you to work on different data engineering tasks concurrently. For example, you might have a branch for feature development, data cleaning and data transformation. Each branch can have its own set of data changes. Branches: In Nessie, a commit represents a snapshot of the data at a specific point in time. When you make changes to your data, you create a new commit, which records those changes. Commits are linked to a specific branch. When you need to revert your data to a stable or known-good state, Nessie makes it easy to select a specific commit and roll back to that data version. This ensures data quality and consistency. Commits: : Nessie allows you to merge the changes from one branch into another. This is similar to merging code changes in Git. When you merge a branch, the data changes made in that branch become part of the target branch. Merges The Components is a distributed analytics engine that operates as an open-source platform, offering an intuitive self-service interface for data exploration, transformation and collaborative work. Its design is grounded in a high-speed columnar memory format. Dremio Apache Arrow, We’ve already explored how to deploy Dremio with Kubernetes, and also how to . For additional information about Dremio, consult . query Iceberg tables on MinIO with Dremio Dremio Resources is high-performance object storage. Renowned for its exceptional and , MinIO serves as a pivotal component in building and maintaining modern data lake infrastructures. MinIO empowers architects to efficiently manage and store massive volumes of data on-prem, on bare metal, on the edge, or on any of the public clouds. MinIO speed scalability is an open table format suited for managing large volumes of data in data lakes. Unique features like time travel, dynamic schema evolution, and Partition Evolution make it a game-changer, allowing query engines to work concurrently on the same data safely and efficiently. See for more information about Iceberg features. Apache Iceberg MinIO The Definitive Guide to Lakehouse Architecture with Iceberg and Getting Started You’ll need the and for this tutorial. The easiest way to get both if you don’t already have them is to install Docker Desktop. Docker Engine Docker Compose This part of the tutorial is based on Dremio’s . You can get the repo for this project . blog post here To begin, open a terminal and navigate to the folder where you cloned/downloaded the repo, and run the command below to start up Dremio. docker-compose up dremio Wait a few moments and then navigate to to access Dremio. Update the requested fields and then click Next. http://localhost:9047 Next, run the following command to start MinIO. docker-compose up minio The final command below will start up Nessie. docker-compose up docker-compose up nessie Navigate to to log in to MinIO with the username and password of . You’ll be prompted to create a bucket. http://localhost:9001 minioadmin:minioadmin Make one named . iceberg-datalake Then head back over to Dremio at and click on Add Source and select Nessie. http://localhost:9047 Set the Name to nessie Set the endpoint URL to http://nessie:19120/api/v2 Set the authentication to none Don’t click Save yet. Instead, in the navigation panel on the left, click Storage. MinIO is S3-API compatible object storage and can use the same connection pathways as AWS S3. For your access key, set minioadmin For your secret key, set minioadmin Set root path to /iceberg-datalake Scroll down for the next set of instructions. Click on the button under to create and configure the following properties. Add Property Connection Properties to fs.s3a.path.style.access true to fs.s3a.endpoint minio:9000 to dremio.s3.compat true Uncheck Encrypt connection Then click Save. You should now see Nessie Catalogs in your data sources. Create a Data Source In Dremio navigate to SQL Runner on the left-hand side. Make sure that in the upper right area of the text editor is set to our Nessie source. Otherwise, you will have to reference the context like instead of just to run this query. Copy and paste the SQL below and run. Context nessie.SalesData SalesData CREATE TABLE SalesData (
    id INT,
    product_name VARCHAR,
    sales_amount DECIMAL,
    transaction_date DATE
) PARTITION BY (transaction_date); Run the query below to insert data into the table you just created. INSERT INTO SalesData (id, product_name, sales_amount, transaction_date)
VALUES
    (1, 'ProductA', 1500.00, '2023-10-15'),
    (2, 'ProductB', 2000.00, '2023-10-15'),
    (3, 'ProductA', 1200.00, '2023-10-16'),
    (4, 'ProductC', 1800.00, '2023-10-16'),
    (5, 'ProductB', 2200.00, '2023-10-17'); Navigate back to MinIO to see that your data lake has been populated with the Iceberg tables. Branching and Merging with Nessie Return to Dremio at . Begin by querying a table on the main branch using the syntax: http://localhost:9047 AT BRANCH SELECT * FROM  nessie.SalesData AT BRANCH main; Create an ETL (Extract Transform and Load) branch to allow you to play around with and transform data without impacting production. CREATE BRANCH etl_06092023 in nessie Within the ETL branch, insert new data into the table: USE BRANCH etl_06092023 in nessie;
INSERT INTO nessie.SalesData (id, product_name, sales_amount, transaction_date) VALUES
(6, 'ProductC', 1400.00, '2023-10-18'); Confirm the immediate availability of the new data within the ETL branch: SELECT * FROM nessie.SalesData AT BRANCH etl_06092023; Note the isolation of changes from users on the main branch: SELECT * FROM nessie.SalesData AT BRANCH main; Merge the changes from the ETL branch back into the main branch: MERGE BRANCH etl_06092023 INTO main in nessie; Select the main branch again to see that the changes have indeed been merged. SELECT * FROM nessie.SalesData AT BRANCH main This branching strategy enables data engineers to independently handle numerous transactions across multiple tables. When they are ready, data engineers can merge these transactions into a single, comprehensive multi-table transaction within the main branch. Conclusion This blog post delved into the power of Git-like version control in data engineering, emphasizing how seamlessly manages data versions, branches and merges. This step-by-step guide demonstrates how Nessie, in collaboration with Dremio and MinIO, as the object storage foundation, enhances data quality and collaboration in data engineering workflows. Nessie Let us know what your data lakehouse looks like at or on . hello@minio.io our Slack channel Also appears . here

Data Lake Mysteries Revealed: Nessie, Dremio, and MinIO Make Waves

ডেটা লেক রহস্য প্রকাশিত হয়েছে: নেসি, ড্রেমিও এবং মিনিও মেক ওয়েভস

Data-Lake-Rätsel aufgedeckt: Nessie, Dremio und MinIO sorgen für Aufsehen

Misterios del lago de datos revelados: Nessie, Dremio y MinIO hacen olas

Les mystères du Data Lake révélés : Nessie, Dremio et MinIO font des vagues

डेटा लेक रहस्य का खुलासा: नेस्सी, ड्रेमियो और मिनिओ मेक वेव्स

データレイクの謎が明らかに: Nessie、Dremio、MinIO が波紋を起こす

데이터 레이크의 미스터리 공개: Nessie, Dremio 및 MinIO가 물결을 일으키다

Mistérios do Data Lake revelados: Nessie, Dremio e MinIO fazem ondas

Тайны озера данных раскрыты: Несси, Дремио и MinIO вызывают фурор

Data Lake'in Gizemleri Ortaya Çıktı: Nessie, Dremio ve MinIO Dalga Yaratıyor

Những bí ẩn về hồ dữ liệu được tiết lộ: Nessie, Dremio và MinIO tạo nên làn sóng

数据湖谜团揭开：Nessie、Dremio 和 MinIO 掀起波澜

Amid the fervor to adopt AI is a critical and often overlooked truth -  the success of any AI initiative is intrinsically tied to the quality, reliability and performance of the underlying data infrastructure. If you don't have the proper foundation, you are limited in what you can build and therefore what you can achieve. Your data infrastructure is the bedrock upon which your entire AI infrastructure is built. It's where data is collected, stored, processed, and transformed. Training models using supervised, unsupervised, and reinforcement learning require storage solutions that can handle structured data - like a Data Warehouse. On the other hand, if you are training Large Language Models (LLMs), you must manage unstructured data - documents in their raw and processed form. A Modern Datalake, or Lakehouse, is the foundation for both these different flavors of AI. A Modern Datalake is one-half Data Warehouse and one-half Data Lake and uses object storage for everything. More recently, we have seen the rise of open table formats. Open Table Formats (OTFs) like Apache Iceberg, Apache Hudi, and Delta Lake make it seamless for object storage to be used within a data warehouse. The rest of this article will examine how to leverage the characteristics of the Modern Datalake that differentiate it from conventional solutions like proprietary Data Warehouses and appliances. To build a foundation for AI infrastructure you need the following: Disaggregation of Compute and Storage Scale Out (Not Up) Software Defined Cloud Native Commodity Hardware If we agree on the above, a series of best practices emerge that focus on two areas of performance. If incorporated, the modern Datalake will be both fast and scalable. These best practices include: Optimizing Drive Price and Performance Incorporate a High-Speed Network Disaggregating Compute and Storage Disaggregating compute and storage within your data infrastructure means that separate resources are used for compute and storage. This contrasts conventional storage solutions, where everything comes packaged in a single server or, worse yet, an appliance. However, Modern Datalakes take disaggregation to another level. If the Data Lake and the Data Warehouse had completely different storage requirements, we could use two separate instances of an object store, as shown below. Additionally, If the Data Warehouse needs to support workloads that require conflicting configurations, then you could use more than one processing engine. This is shown below. A composable infrastructure allows you to scale your compute and storage resources independently. This means you can allocate more resources to the part of your infrastructure that needs it most rather than upgrading both compute and storage together. It results in cost-effective scaling as you only invest in the required resources. Scale-Out Not Up AI workloads are data intensive, are often distributed across multiple CPUs or GPUs, use copious compute power for training, and necessitate real-time inference. Scaling out, not up, helps optimize performance and accommodate high-speed networks. Scaling out and scaling up are two different approaches to increasing the capacity and performance of your data infrastructure. However, scaling out is proving to be the more viable approach as advances are made in clustering platforms like Kubernetes and more and more solutions strive to be cloud native. Scaling out in a disaggregated infrastructure provides: - If one node is busy, another node can take on a new request, reducing wait times and increasing throughput. If one node fails, the workload can be shifted to other nodes, reducing downtime and ensuring continuity. High Availability and Fault Tolerance - Scaling out can provide better performance by distributing workloads across multiple nodes or servers to handle larger volumes of data and more concurrent requests. Scaling out is also more flexible because you can add or remove nodes as needed, making it easier to adjust to fluctuating workloads or to accommodate seasonal variations. Performance and Flexibility - Maintenance and upgrades are simplified when you scale out. Instead of taking a critical system offline for upgrades, you can perform maintenance on individual storage or compute nodes without disrupting the entire infrastructure. Operationally and Resource Efficient Cloud Native + Software Defined The last component of leveraging the Modern Datalake to build a strong foundation for AI is taking a cloud-native, software-defined approach. Containers like Docker and container orchestration tools, like Kubernetes, make cloud-native architectures possible. All the components of a Modern Datalake run in containers that run in Kubernetes. Therefore, a Modern Datalake is cloud native. "Software-defined" refers to an approach in which software controls and manages the configuration, functionality, and behavior of hardware components, often in the context of computer systems and networking. This is the building block of the infrastructure as code movement where the emphasis is on smart software and dumb fast hardware. Software-defined storage abstracts and manages the storage resources through software, making allocating and managing storage capacity across different devices and storage media easier. Built for Speed: NVMe and 100GbE To take full advantage of your commodity hardware and software-defined architecture - you need two more key pieces. The first is NVMe drives. Modern, performance oriented workloads, the random nature of read/write, the rise of small objects and falling SSD pricing all favor a . Do that math, the upfront may be higher, the TCO will be lower. NVMe centric architecture The second component is 100GbE networking. In a software-defined world, the network turns out to be the bottleneck in many setups even at 100GbE. Here are some of those scenarios: - AI workloads often process massive datasets, such as images, videos, natural language text, and sensor data. High-speed networks can quickly transfer these large datasets between storage and processing units, reducing data transfer bottlenecks. Data Intensive - Many AI tasks involve distributed computing across multiple CPUs or GPUs. High-speed networks enable efficient communication and data exchange between these devices, ensuring the computing clusters work in parallel effectively. Distributed Computing - Training deep learning models, especially LLMs like transformers or convolutional neural networks, requires a lot of data and computational power. A high-speed network allows faster data loading and synchronization between distributed GPUs, which can significantly speed up training times. Model Training - Low-latency and high-throughput networks are essential for responsive applications that incorporate AI. A high-speed network ensures minimal delay between a user request and a response from a model. Real-time Inference Foundational Concepts By adhering to these principles: the disaggregation of compute and storage, scale-out, not up, dumb, fast hardware and smart cloud native software the enterprise can construct a Modern Datalake that has the right foundation for meeting these requirements and pushing your AI initiatives forward. You cannot construct a building on a poor foundation, just ask the ancient Egyptians. The AI game is about performance at scale, and this requires the right foundation. To skimp on the foundation is to accumulate technical debt that will topple your Jenga tower after a few minutes. Build smart, put the foundation in place.

The Principles to Keep In Mind When Building a Modern Datalake for Your AI Infrastructure

আপনার AI পরিকাঠামোর জন্য একটি আধুনিক ডেটালেক তৈরি করার সময় যে নীতিগুলি মাথায় রাখতে হবে

Die Grundsätze, die Sie beim Aufbau eines modernen Datalake für Ihre KI-Infrastruktur beachten sollten

Los principios a tener en cuenta al construir un lago de datos moderno para su infraestructura de IA

Les principes à garder à l'esprit lors de la création d'un datalake moderne pour votre infrastructure d'IA

आपके एआई इन्फ्रास्ट्रक्चर के लिए आधुनिक डेटालेक का निर्माण करते समय ध्यान में रखने योग्य सिद्धांत

AI インフラストラクチャ用の最新のデータレイクを構築する際に留意すべき原則

AI 인프라를 위한 최신 데이터레이크를 구축할 때 명심해야 할 원칙

Os princípios a serem considerados ao construir um Datalake moderno para sua infraestrutura de IA

Принципы, которые следует учитывать при создании современного озера данных для вашей инфраструктуры искусственного интеллекта

Yapay Zeka Altyapınız için Modern Bir Datalake Oluştururken Akılda Tutulması Gereken İlkeler

Những nguyên tắc cần ghi nhớ khi xây dựng Datalake hiện đại cho cơ sở hạ tầng AI của bạn

为人工智能基础设施构建现代数据湖时要牢记的原则

As much as we say is an all-in-one data platform that is capable of various analytics workloads, it is always compelling to demonstrate that by real use cases. That's why I would like to share this user story with you. It is about how they leverage the capabilities of Apache Doris in reporting, customer tagging, and data lake analytics and achieve high performance. Apache Doris This fintech service provider is a long-term user of Apache Doris. They have almost 10 clusters for production, hundreds of Doris backend nodes, and thousands of CPU Cores. The total data size is near 1 PB. Every day, they have hundreds of workflows running simultaneously, receive almost 10 billion new data records, and respond to millions of data queries. Before migrating to Apache Doris, they used ClickHouse, MySQL, and Elasticsearch. Then, frictions arise from their ever-enlarging data size. They found it hard to scale out the ClickHouse clusters because there were too many dependencies. As for MySQL, they had to switch between various MySQL instances because one MySQL instance had its limits, and cross-instance queries were not supported. Reporting From ClickHouse + MySQL to Apache Doris Data reporting is one of the major services they provide to their customers and they are bound by an SLA. They used to support such service with a combination of ClickHouse and MySQL, but they found significant fluctuations in their data synchronization duration, making it hard for them to meet the service levels outlined in their SLA. Diagnosis showed that it was because the multiple components add to the complexity and instability of data synchronization tasks. To fix that, they have used Apache Doris as a unified analytic engine to support data reporting. Performance improvements With Apache Doris, they ingest data via the method and reach an SLA compliance rate of over 99% in terms of data synchronization performance. Broker Load As for data queries, the Doris-based architecture maintains an of less than and a of less than . This is a 50% speedup compared to the old architecture. average query response time 10s P90 response time 30s Tagging Tagging is a common operation in customer analytics. You assign labels to customers based on their behaviors and characteristics so that you can divide them into groups and figure out targeted marketing strategies for each group of them. In the old processing architecture, where Elasticsearch was the processing engine, raw data was ingested and tagged properly. Then, it will be merged into JSON files and imported into Elasticsearch, which provides data services for analysts and marketers. In this process, the merging step was to reduce updates and relieve load for Elasticsearch, but it turned out to be a troublemaker: Any problematic data in any of the tags could spoil the entire merging operation and thus interrupt the data services. The merging operation was implemented based on Spark and MapReduce and took up to 4 hours. Such a long time frame could encroach on marketing opportunities and lead to unseen losses. Then Apache Doris takes this over. Apache Doris arranges tag data with its data models, which process data fast and smoothly. The aforementioned merging step can be done by the , which aggregates tag data based on the specified Aggregate Key upon data ingestion. The is handy for partial column updates. Again, all you need is to specify the Unique Key. This enables swift and flexible data updating and saves you from the trouble of replacing the entire flat table. You can also put your detailed data into a to speed up certain queries. . Aggregate Key model Unique Key model Duplicate model In practice, it took the user 1 hour to finish the data ingestion, compared to 4 hours with the old architecture In terms of query performance, Doris is equipped with well-developed bitmap indexes and techniques tailored to high-concurrency queries, so in this case, it can finish and reach over . customer segmentation within seconds 700 QPS in user-facing queries Data lake analytics In data lake scenarios, the data size you need to handle tends to be huge, but the data processing volume in each query tends to vary. To ensure fast data ingestion and high query performance of huge data sets, you need more resources. On the other hand, during non-peak time, you want to scale down your cluster for more efficient resource management. How do you handle this dilemma? Apache Doris has a few features that are designed for data lake analytics, including Multi-Catalog and Compute Node. The former shields you from the headache of data ingestion in data lake analytics while the latter enables elastic cluster scaling. The mechanism allows you to connect Doris to a variety of external data sources so you can use Doris as a unified query gateway without worrying about bulky data ingestion into Doris. Multi-Catalog The of Apache Doris is a backend role that is designed for remote federated query workloads, like those in data lake analytics. Normal Doris backend nodes are responsible for both SQL query execution and data management, while the Compute Nodes in Doris, as the name implies, only perform computation. Compute Nodes are stateless, making them elastic enough for cluster scaling. Compute Node The user introduces Compute Nodes into their cluster and deploys them with other components in a hybrid configuration. As a result, the cluster automatically scales down during the night, when there are fewer query requests, and scales out during the daytime to handle the massive query workload. This is more resource-efficient. For easier deployment, they have also optimized their Deploy on Yarn process via Skein. As is shown below, they define the number of Compute nodes and the required resources in the YAML file, and then pack the installation file, configuration file, and startup script into the distributed file system. In this way, they can start or stop the entire cluster of over 100 nodes within minutes using one simple line of code. Conclusion For data reporting and customer tagging, Apache Doris smoothens data ingestion and merging steps and delivers high query performance based on its own design and functionality. For data lake analytics, the user improves resource efficiency by elastic scaling of clusters using the Compute Node. Along their journey with Apache Doris, they have also developed a data ingestion task prioritizing mechanism and contributed it to the Doris project. A gesture to facilitate their use case ends up benefiting the whole . This is a great example of open-source products thriving on user involvement. open source community Also published . here

Data Analytics: Apache Doris' Impact in Reporting, Tagging, and Data Lake Operations

T3Go is China’s first platform for smart travel based on the Internet of Vehicles. In this article, Trevor Zhang and Vino Yang from T3Go describe the evolution of their data lake architecture, built on cloud-native or open-source technologies, including Alibaba OSS, Apache Hudi, and Alluxio. Today, their data lake stores petabytes of data, supporting hundreds of pipelines and tens of thousands of tasks daily. It is essential for business units at T3Go, including Data Warehouse, Internet of Vehicles, Order Dispatching, Machine Learning, and self-service query analysis. In this blog, you will see how we slashed data ingestion time by half using Hudi and Alluxio. Furthermore, data analysts using Presto, Hudi, and Alluxio saw the queries speed up by 10 times. We built our data lake based on data orchestration for multiple stages of our data pipeline, including ingestion and analytics. T3Go data lake Overview Prior to the data lake, different business units within T3Go managed their own data processing solutions, utilizing different storage systems, ETL tools, and data processing frameworks. Data for each became siloed from every other unit, significantly increasing cost and complexity. Due to the rapid business expansion of T3Go, this inefficiency became our engineering bottleneck. We moved to a unified data lake solution based on Alibaba OSS, an object store similar to AWS S3, to provide a centralized location to store structured and unstructured data, following the design principles of ; all the applications access OSS storage as the source of truth, as opposed to different data silos. This architecture allows us to store the data as-is without having to first structure the data and run different types of analytics to guide better decisions, building dashboards and visualizations from big data processing, real-time analytics, and machine learning. Multi-cluster Shared-data Architecture Efficient Near Real-time Analytics Using Hudi Our business in smart travel drives the need to process and analyze data in a near real-time manner. With a traditional data warehouse, we faced the following challenges: High overhead when updating due to long-tail latency. High cost of order analysis due to the long window of a business session Reduced query accuracy due to late or ad-hoc updates Unreliability in data ingestion pipeline. Data lost in the distributed data pipeline cannot be reconciled. High latency to access data storage. As a result, we adopted Apache Hudi on top of OSS to address these issues. The following diagram outlines the architecture: Enable near real-time data ingestion and analysis With Hudi, our data lake supports multiple data sources, including Kafka, MySQL binlog, GIS, and other business logs in near real-time. As a result, more than 60% of the company’s data is stored in the data lake, and this proportion continues to increase. We are also able to speed up the data ingestion time down to a few minutes by introducing Apache Hudi into the data pipeline. Combined with big data interactive query and analysis frameworks, such as Presto and SparkSQL, real-time data analysis and insights are achieved. Enable Incremental processing pipeline With the help of Hudi, it is possible to provide incremental changes to the downstream derived table when the upstream table updates frequently. Even with a large number of interdependent tables, we can quickly run partial data updates. This also effectively avoids updating the full partitions of cold tables in the traditional Hive data warehouse. Accessing Data using Hudi as a unified format Traditional data warehouses often deploy Hadoop to store data and provide batch analysis. Kafka is used separately to distribute Hadoop data to other data processing frameworks, resulting in duplicated data. Hudi helps effectively solve this problem; we always use Spark pipelines to insert new updates into the Hudi tables, then incrementally read the update of Hudi tables. In other words, Hudi tables are used as the unified storage format to access data. Efficient Data Caching Using Alluxio In the early version of our data lake without Alluxio, data received from Kafka in real-time is processed by Spark and then written to OSS data lake using Hudi DeltaStreamer tasks. With this architecture, Spark often suffered high network latency when writing to OSS directly. Since all data is in OSS storage, OLAP queries on Hudi data may also be slow due to a lack of data locality. To address the latency issue, we deployed Alluxio as a data orchestration layer, co-located with computing engines such as Spark and Presto, and used Alluxio to accelerate read and write on the data lake as shown in the following diagram: Data in formats such as Hudi, Parquet, ORC, and JSON are stored mostly on OSS, consisting of 95% of the data. Computing engines such as Flink, Spark, Kylin, and Presto are deployed in isolated clusters, respectively. When each engine accesses OSS, Alluxio acts as a virtual distributed storage system to accelerate data, being co-located with each of the computing clusters. Specifically, here are a few applications leveraging Alluxio in the T3Go data lake. Data lake ingestion We mount the corresponding OSS path to the Alluxio file system and set Hudi’s parameter value to use the scheme in place of scheme. Spark pipelines with Hudi continuously ingest data to Alluxio. After data is written to Alluxio, it is asynchronously persisted from the Alluxio cache to the remote OSS every minute. These modifications allow Spark to write to a local Alluxio node instead of writing to remote OSS, significantly reducing the time for the data to be available in the data lake after ingestion. target-base-path alluxio:// oss:// Data analysis on the lake We use Presto as an ad-hoc query engine to analyze the Hudi tables in the lake, co-locating Alluxio workers on each Presto worker node. When Presto and Alluxio services are co-located and running, Alluxio caches the input data locally in the Presto worker, which greatly benefits Presto for subsequent retrievals. On a cache hit, Presto can read from the local Alluxio worker storage at memory speed without any additional data transfer over the network. Concurrent accesses across multiple storage systems In order to ensure the accuracy of training samples, our machine learning team often synchronizes desensitized data in production to an offline machine learning environment. During synchronization, the data flows across multiple file systems, from production OSS to an offline HDFS followed by another offline Machine Learning HDFS. This data migration process is not only inefficient but also error-prone for modelers because multiple different storages with varying configurations are involved. Alluxio helps in this specific scenario by mounting the destination storage systems under the same filesystem to be accessed by their corresponding logical paths in Alluxio namespace. By decoupling the physical storage, allows applications with different APIs to access and transfer data seamlessly. This data access layout also improves performance. Microbenchmark Overall, we observed the following improvements with Alluxio: It supports a hierarchical and transparent caching mechanism. It supports cache promote mode when reading. It supports asynchronous writing mode. It supports LRU recycling strategy. It has pin and TTL features After comparison and verification, we choose to use Spark SQL as the query engine. Our performance testing queries the Hudi table, comparing Alluxio + OSS together against OSS directly as well as HDFS. In the stress test shown above, after the data volume is greater than a certain magnitude (2400W), the query speed using Alluxio+OSS surpasses the HDFS query speed of the hybrid deployment. After the data volume is greater than 1E, the query speed starts to double. After reaching 6E data, it is up to 12 times higher than querying native OSS and 8 times higher than querying native HDFS. The improvement depends on the machine configuration. Based on our performance benchmarking, we found that the performance can be improved by over 10 times with the help of Alluxio. Furthermore, the larger the data scale, the more prominent the performance improvement. Next Step As T3Go’s data lake ecosystem expands, we will continue facing the critical scenario of compute and storage segregation. With T3Go’s growing data processing needs, our team plans to deploy Alluxio on a larger scale to accelerate our data lake storage. In addition to the deployment of Alluxio on the data lake computing engine, which currently is mainly SparkSQL, we plan to add a layer of Alluxio to the OLAP cluster using Apache Kylin and an ad_hoc cluster using Presto. The goal is to have Alluxio cover all computing scenarios, with Alluxio interconnected between each scene to improve the read and write efficiency of the data lake and the surrounding lake ecology. Conclusion As mentioned earlier, Hudi and Alluxio cover all scenarios of Hudi’s near real-time ingestion, near real-time analysis, incremental processing, and data distribution on DFS, among many others, and play the role of a powerful accelerator on data ingestion and data analysis on the lake. With Hudi and Alluxio together, our R&D engineers shortened the time for data ingestion into the lake by up to a factor of 2. Data analysts using Presto, Hudi, and Alluxio in conjunction to query data on the lake saw their queries speed up by 10 times faster. Furthermore, the larger the data scale, the more prominent the performance improvement. Alluxio is an important part of T3Go’s plan to become a leading enterprise data lake in China. We look forward to seeing further integration with Alluxio in T3Go’s data lake ecosystem.

Building High-Performance Data Lake Using Apache Hudi and Alluxio 

Data lakes provide a myriad of benefits. They are data agnostic and don’t require you to define a schema upfront. However, without a proper structure, it may be challenging to find the data that you need. In this article, we’ll address this problem by creating a FastAPI abstraction allowing us to query the AWS Glue metadata catalog. Why FastAPI for a data discovery service? Photo by nine koepfer If we would implement it as a Python package instead, we would assume that everyone knows enough Python to use it. However, your company may have other developers (Java, C#, JavaScript, Go, …) that need access to data from a data lake. By building a REST-ful service for data discovery, you are providing a interface for everyone. This approach has further : programming-language agnostic API-first benefits If you later decide that you want to build a for business users, your task will become much easier since you already have a REST API that provides data as JSON. UI You can combine all the functionality for this purpose in a . single service If you need to add new methods or change the existing ones, you can deploy easier by creating a (ie, api_url/v3) without breaking any workloads that depend on the current version. changes new API version You can to include real-time data streams, unstructured data, as well as to allow querying access logs to identify which data is used most frequently and by which users. extend the functionality We can start by listing all the methods that can be potentially useful for data discovery: listing all databases or , schemas listing all for a specific schema, tables filtering for table names with a specific or , prefix suffix doing a on table names (say, all tables that have anything to do with orders or customers), full-text search doing a on table comments (ex., if we want to find any column related to ), full-text search logistics showing a ( ), full table definition all columns, data types, and column comments showing for a specific table, partitions showing from S3 below a specific partition, all files showing all columns with specific , comment showing all tables that contain a specific , column name finally, the best way for data discovery is to just of a full table. look at the first X rows We have three choices: either using lower-level AWS SDKs, doing queries on Athena’s information_schema, or leveraging awswrangler. In the end, we may combine all of them to satisfy the requirements from the previous section. #1. To list all databases ( ) and tables, we can use the package, especially the module wr.catalog. i.e., schemas awswrangler #2. To filter for table names with a specific or , we can also use wr.catalog.tables. The same is true for retrieving and doing a on those definitions: prefix suffix table definition full-text search #3. List all of a specific table: partitions #4. Show table’s DDL: #5. directory or show specific objects according to file type ( ) and last modified date: List S3 path ex. parquet files #6. Query to find the dataset that you need: column comments #7. Describe Athena : table definition #8. Search for specific : column names The above methods constitute a basic MVP for a data discovery service. It allows querying schemas, tables, columns, column comments ( ), showing a preview of the data, as well as exploring the underlying files, directories, and partitions. aka data dictionary By leveraging services such as AWS X-Ray and Lake Formation, we could add methods to query usage access patterns and identify the most and the least used datasets. If you don’t want to attach your boto3_session separately in each API method, you can attach a global session using: Side note: Implementation: FastAPI MVP The full code for this MVP can be found in . Let’s briefly discuss some of the details. this Github repository In the following demo, we can see all the endpoints applied on a . Brazilian E-commerce dataset from Kaggle The screencast above demonstrates how we can use this API to discover and query e-commerce data stored in an S3 data lake. The end users can identify tables or schemas that they need. By drilling down into the underlying files, data scientists and analysts can explore this data in an interactive notebook or other tools of their choice. Reading a specific parquet, CSV, Excel, or JSON file from S3 with awswrangler is as simple as: How can we serve this API at scale? There are several options to deploy a REST API. If you already use AWS, you may find it useful to leverage serverless services built specifically for building resilient and scalable APIs: and . AWS Lambda API Gateway With the excellent package called mangum, converting our FastAPI to a Lambda handler is as simple as importing this package ( ) and adding a single line of code: handler = Mangum(app=app). from mangum import Mangum The Dockerfile to deploy the API to AWS Lambda looks as follows: We use the official Python Lambda image with Python 3.8 and all required . Then, we copy our remaining API code and specify the that will be used as an entry point to our container. install packages Lambda handler Finally, we have to push the container to ECR — AWS container registry. To replicate this demo, replace 123456789 with your AWS account ID, and adjust your AWS region name. In case you’re wondering: dda is my abbreviation for data-discovery-api. AWS Lambda The container image is deployed to ECR. Now, we need to . We choose the container image option and select our ECR image. create a Lambda function Since querying some tables can take longer than Lambda’s default of just three seconds, we can extend it to three minutes. timeout Finally, we need to attach IAM policies to grant our Lambda function permissions to retrieve data from AWS Glue, Athena, and S3 ( ). of course, we should use more granular permissions for production API Gateway After configuring the Lambda function with our container image, we can create a new API in the API Gateway console. We need to choose REST API and specify a name. Then, we can add a method and resource to configure Lambda Proxy integration. Once all that is set up, we can deploy the API and start serving it. To see it step by step, here is a screencast that demonstrates API Gateway configuration, deployment, and testing the API: How can we monitor our data discovery service? Imagine that you deployed a similar service at your company and it turned out to be a resounding success. Now you need to ensure that it stays , can be easily , and runs . At this point, you can either: healthy maintained without errors Keep an eye on it manually ( ), not recommended Build an yourself by using AWS CloudWatch and related services, observability dashboard Make use of observability tools The implications of each option: With option #1, you will likely go mad. With option #2, you would have to spend extra time building CloudWatch dashboards, jump between several CloudWatch logs and multiple AWS console tabs to fix any potential bugs. With option #3, you only have to sign in to your account  to scans all serverless resources and provides you with visualizations you need to track the health of your APIs and microservices. You can then optionally ( ) build a by combining API Gateway and Lambda resources, and on failure. highly recommended! project dashboard set alerts In the image above, we can see how Dashbird pointed out the task which timed out after 3 seconds ( ). It also reveals when the function experienced a cold start and shows the duration for each API call. default Lambda timeout Lambda For , we can see the average latency, number of API requests, and different types of errors that we encountered in the initial testing of the API. API Gateway Conclusion In this article, we investigated how we can build a REST-ful service for data discovery. By leveraging Python libraries such as FastAPI, awswrangler, boto3, and Mangum, we can build useful APIs in just a few hours rather than weeks. Additionally, by deploying this service to API Gateway and Lambda, we can serve this API at scale with no operational headaches. Lastly, by leveraging Dashbird, you can add observability to your serverless resources without having to install any CloudWatch agents, pulling logs, or building any dashboards. Previously published at https://dashbird.io/blog/data-discovery-api-aws-data-lake/

How To Use a FastAPI Abstraction To Query AWS Data Lake

Databases store data in a structured form. The structure makes it possible to find and edit data. With their structured structure, databases are used for data management, data storage, data evaluation, and targeted processing of data. In this sense, data is all information that is to be saved and later reused in various contexts. These can be date and time values, texts, addresses, numbers, but also pictures. The data should be able to be evaluated and processed later. The amount of data the database could store is limited, so enterprise companies tend to use data warehouses, which are versions for huge streams of data. Source A data warehouse (DWH) is a prepared, structured (tabular) recorded data source. In the simplest case, an SQL database that contains at least one data record. At the enterprise level, such a warehouse naturally quickly takes on larger dimensions, so that there are entire business intelligence departments that only deal with the business warehouse. A data lake records both structured and unstructured data and store it in various processing steps (raw, processed, analyzed).The high possible variance of data sources (e.g. images, videos, text) and the raw data allows data scientists to carry out further analyzes that are not possible in a DWH. In abstract terms, the data warehouse can be considered as an Excel file on the computer, while the data lake represents a file folder. Like an Excel file, the DWH contains very structured data with named columns in a fixed schema. Adding new entries is not a problem, new columns are more difficult depending on the existing content. Once the data has been typed in and saved, the originals can no longer be found, so the file must be relied on. The content, in turn, is very easy to use for certain applications: visualizations or simple arithmetic operations for KPI calculation are very convenient. If you now take the data lake in the same example, it behaves like a file folder on the hard drive. You can store a large number of (original) data without having to format, type or structure them beforehand. However, if you want to continue working with the data, you have to prepare the data first and you cannot simply pull sums across columns. If this processing has taken place, however, the data can then also be saved in the folder again. Even many Excel files — i.e. data warehouses — can thus be generated and saved in the folder for further processing. Conclusion Hopefully, this simple example explains the difference between a database, data warehouse, and data lake. For companies, depending on the maturity of the data-driven business, it makes sense to use one or both of the infrastructures. A DWH allows a wide range of users quick access to structured data for analysis, while a data lake enables advanced users, for example, data engineers and data scientists, to apply machine learning and other advanced analytics methods. However, if you do need a deeper explanation of the differences between these 3 terms and a deep dive into Data Lake functioning ; read my previous article .

Database Vs Data Warehouse Vs Data Lake: A Simple Explanation

This is a hands-on course on how to deploy a fully Serverless web app using the . You will learn how to: AWS CDK Structure **CDK Stacks **to deploy an application from end-to-end Deploy a integrated with for dynamic requests processing Store data in a fast and cost-effective way with REST API AWS Lambda DynamoDB Use as a source for in an event-driven architecture DynamoDB streams Lambda Ingest and manipulate loads of data streams with Kinesis Firehose Deploy and query a with , and your entire application health in a single place using Data Lake Athena S3 GlueMonitor Dashbird You can use the resources declared in this demo application as a starting point to mix and adapt to your own architectures later, which should save you quite some time. The App and Architecture The demo app is a public blog where anyone can read, publish and like posts. It’s available . Go ahead and publish something in the top-left corner (yellow button) and also “like” articles already published. Check out the codebase on . on this link this repo Frontend Backend Data Lake and Analytical Querying What is the AWS CDK? CDK stands for Cloud Development Kit. Think of it as CloudFormation (CF) in your preferred language (Python, Typescript, C#, etc). Roughly speaking, it works like this: 1. You declare cloud resources using classes provided by the CDK libraries. Example: aws_cdk aws_s3
my_bucket = aws_s3.Bucket(self, ) from import 'MyBucket' 2. Run CDK deploy 3. CDK translates this to a CloudFormation template and deploy it on AWS for you. In case you would like to dig deeper, that will get your basics started. I also strongly recommend reading the . AWS also has a workshop official CDK documentation Advantages using the AWS CDK Use languages that are more expressive than YAML or JSON, for instance. Less - much less! - verbose than CloudFormation templates. Easier to apply reusability and inheritance principles to infrastructure code. Better integration with IDEs for code completion, IntelliSense, etc. Possible to test your infra code, just as any other software Portable: since it’s just a wrapper around CF, we can easily port it to JSON or YAML Disadvantages of using the AWS CDK Although released as a stable project by the AWS team, many parts (a lot of the good ones) are still experimental and APIs may change in backwards incompatible ways. It’s under constant development. During the preparation of this course, I had to upgrade my libraries three times. Documentation is still lacking at some parts and you will need to look at the CDK code, occasionally, to understand how to declare certain things. Deploy it yourself Although we have provided an online demo, you can also deploy this app in your own AWS account: Clone the repo: git clone git@github.com:byrro/serverless-website-demo.git sls-demo; cd sls-demo Setup your virtual environment: virtualenv -p /urs/bin/python3.8 .env; source .env/bin/activate; pip install -r requirements Specify an AWS account ID: ** export AWS_ACCOUNT_ID=1234567890 Deploy all three stacks: cdk deploy sls-blog; cdk deploy sls-blog-api; cdk deploy sls-blog-analytical ** You can also hard code your Account ID in the CDK project, as I’ll show in a minute; When starting a new project from scratch, you would run . This is not necessary for this demo, since the project is already created. cdk init --language [python|typescript|...] Monitoring Deploying this architecture in the cloud and blindly believing it will work flawlessly is not reasonable. We want to be the first one to know when something is not right to act upon it as quickly as possible. In this project, I used for its ease of use and seamless integration. Instead of having to deploy an agent inside my code, Dashbird plugs into my Stacks that I can deploy with the effort of one click. It not only monitors Lambda function errors, but also other resources that we’re using, such as DynamoDB tables. They even suggest cross-referenced against industry best practices. Dashbird through a CloudFormation template insights for architectural improvements Finally, Dashbird offers a free-forever plan. It’s a no-brainer to try it out by . registering for free How a CDK project is structured A CDK project creates an “Application”. This app may have one or more “Stacks”. A Stack is a group of cloud resources (Lambda functions, S3 buckets, etc) that are instantiated using CDK classes. It’s also possible to have multiple applications in a single CDK project. App Object Creating a CDK app is as simple as: app = core.App() When you run , an initial application with basic boilerplate code is created for you in the project root, under . cdk init --language [language] app.py The next thing we need is an environment, which is composed of an AWS Account ID and Region: env = env = core.Environment(
    account= ,
    region= ,
) 1234567890 'us-east-1' Declaring an environment is not required (CDK can infer from your AWS credentials), but is a good practice. Most of us work with multiple AWS accounts. It’s easy to mess around with several projects, accounts, credentials. When we explicitly set the environment in the CDK app, it’s locked and prevents mistaken deployments. Now we declare our stacks: my_project.my_project_stack MyStack

my_stack = MyStack(
    app, ,
    env=env,
) from import 'my-stack' This is how we our stacks for deployment. In the next section we’ll see how to those stacks. instantiate declare Stack Object The Stack object is where we our AWS resources. It inherits from the core.Stack CDK class and accepts a scope - which is our app object - a string identifier and an environment. declare ( . ): ( , : . , : , : . ,
            ** ,
            ) -> : (). ( , , ** )

        # class MyStack core Stack def __init__ self scope core Construct id str env core Environment kwargs None super __init__ scope id kwargs Declare AWS resources here Declaring AWS resources To declare AWS resources, we need a specific library for each service. Here’s a list of all and their counterparts. Other flavors are and . Python libraries Typescript Java .NET Let’s see how a basic REST API would be declared (typing expressions were removed for readability purposes): aws_cdk aws_apigateway, aws_lambda = aws_lambda.Function(
            self, ,
            runtime=aws_lambda.Runtime.PYTHON_3_8,
            code=aws_lambda.Code.asset( my_lambda.handler sls-blog-rest-api-gateway from import ( . ): ( , , , ): (). ( , , ** ) class MyStack core Stack def __init__ self scope id env super __init__ scope id kwargs my_lambda 'MyLambda' 'my_lambda_folder),
            handler=' ',
        )

        aws_apigateway.LambdaRestApi(
            self,
            ' ',
            handler=my_lambda,
        ) We first declare a Lambda function . We point its code to the . Inside this folder, there should be a file, containing a function called . This handler function should accept Lambda invocations normally (an and objects). my_lambda my_lambda_folder my_lambda.py handler event context Next, a is declared, using as the handler (not to confuse with the Lambda’s handler function). This will create a new API Gateway REST API integrated with using an . All HTTP requests will be routed to the Lambda function. LambdaRestApi my_lambda my_lambda AWS_PROXY integration type Our Project App & Stacks comprises one application with three Stacks. They’re all declared in the and files. This project app.py sls_website_stack.py Below we’ll walk through all Stacks in a high level. I encourage you to inspect the to learn how these resources are declared and also integrated. For example: a Kinesis Firehose is created in one Stack and referenced in another to include its name as an environment variable for the Lambda function that will interact with it. stacks file Except for the frontend static Stack - which is small - you will notice that resources are initialized with a (null) value in the beginning. The reason is that, even though the CDK is generally more succinct than CloudFormation, it can still be lengthy enough to clutter the view of the entire Stack. Having each resource declared first in one line, I can provide a short summary of everything that’s in the Stack and then instantiate the CDK classes in other methods. None = static_stack

        # SQS Queues
        self.queue_ddb_streams_dlq = None  # Dead-letter-queue DDB streams

        # DynamoDB Tables
        self.ddb_table_blog = None  # Single-table all blog content

        # DynamoDB Event Sources
        self.ddb_source_blog = None  # Blog table streams source

        # DynamoDB Indexes
        self.ddb_gsi_latest = None  # GSI ordering articles by timestamp

        # Lambda Functions
        self.lambda_blog = None  # Serves requests to the blog public API
        self.lambda_stream_reader = None  # Processes DynamoDB streams

        # Continues other resources... ( . ): ( , , , , ): (). ( , , ** ) . class SlsBlogApiStack core Stack def __init__ self scope id env static_stack super __init__ scope id kwargs self static_stack for for with Notice it takes another Stack object ( ) as an argument to its initialization. In the file, you can see that the is initialized passing the as an argument. static_stack app.py SlsBlogApiStack SlsBlogStack We use it to reference the CloudFront distribution domain ( ) in the Lambda environment variables. This variable can be used to customize the HTTP response header to comply with . This illustrates one way to easily integrate and reference information from one Stack into another within a CDK project. d1qmte5oc6ndq5.cloudfront.net Access-Control-Allow-Origin CORS standards At the end of the initialization, another method is called to instantiate the CDK classes for each resource and configure their parameters. self.create_cdk_resources() Next we’ll walk through each of our project’s Stacks. Stack 1: static website Our focus is on the Serverless backend, so the frontend here is terribly rough and simple. It’s stored in an S3 Bucket and distributed through a CloudFront CDN. CDK has a helpful class called BucketDeployment. It takes the contents of a directory and sync to an S3 bucket. In this case, we stored the frontend code in the folder. website_static aws_s3_deployment.BucketDeployment(
    self, ,
    sources=[aws_s3_deployment.Source.asset( )],
    destination_bucket=static_bucket,
    distribution=cdn,
) 'SlsBlogStaticS3Deployment' 'website_static' Stack 2: API/backend Our backend consists of an API Gateway (REST) connected to a . Microservices receives a lot of press, but you probably shouldn’t always break your applications in several functions. A Monolith is just fine - -, really. monolithic Lambda function and sometimes recommended This API & Lambda support a single endpoint (with GET and POST methods) with a queryString “action”, which takes three parameters: : populates the latest blog posts get-latest-articles : triggered when someone likes an article like-article : post a new blog article publish-article Here’s the power of the CDK model. We can create a REST API with 10 lines of code: rest_api_blog = aws_apigateway.LambdaRestApi(
    self, ,
    handler=lambda_blog,  # Previously declared Lambda 'sls-blog-rest-api-gateway' = . ( ),
    ),
) function deploy_options aws_apigateway StageOptions stage_name= ,
        throttling_rate_limit=lambda_param_max_concurrency,
        logging_level=aws_apigateway.MethodLoggingLevel( 'api' 'INFO' One nice little thing is that Lambda memory is used as a cache for the latest articles. We load the cache container outside the Lambda handler function. It remains in memory even after an invocation ends and is available for subsequent requests. Learn more here about . how to use Lambda as a cache mechanism MAX_CACHE_AGE: int = # In seconds
CACHE_LATEST_ARTICLES: Dict[str, Union[int, list]] = { : time.time(), : [],
} 120 'last_update' 'articles' All the data is stored in DynamoDB (DDB) using a , in . The site only shows the latest blog articles and items get after a few days by setting a attribute. single-table design on-demand mode auto-deleted by DDB time-to-live ddb_attr_time_to_live = ddb_param_max_parallel_streams = ddb_table_blog = aws_dynamodb.Table(
    self, ,
    partition_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.STRING,
    ),
    billing_mode=aws_dynamodb.BillingMode.PAY_PER_REQUEST,
    point_in_time_recovery=True,
    removal_policy=core.RemovalPolicy.DESTROY,
    time_to_live_attribute=self.ddb_attr_time_to_live,
    stream=aws_dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
) 'time-to-live' 5 'sls-blog-dynamo-table' 'id' The DDB table also has a that makes it easier to retrieve articles ordered by date for the site: GSI (Global Secondary Index) self.ddb_table_blog.add_global_secondary_index(
    index_name= ,
    partition_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.STRING,
    ),
    sort_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.NUMBER,
    ),
    projection_type=aws_dynamodb.ProjectionType.ALL,
) 'latest-blogs' 'item-type' 'publish-timestamp' Modifications to DDB items generate streams that are processed by a . These streams are then repackaged and sent to a Kinesis Firehose stream processor. second Lambda function DDB doesn’t provide the flexibility that SQL databases offer and many choose , for example. Although Aurora is a great service, personally I prefer DDB for its simplicity and reliable, consistent performance. But sometimes we do need to run analytical queries, those with aggregations and on-the-fly filters. For that, we’ll be using Athena (more in the next Stack). Aurora Serverless Stack 3: analytical querying A Kinesis Firehose Stream is responsible for batching data inserted/modified in DDB, converting them to Apache Parquet format and storing in dedicated S3 buckets. From S3, we create a Data Lake with AWS Glue (used to declare our data schemas) and Athena (used to query the data). Athena is extremely powerful. We can statements ( ) to query terabytes of data and pay on-demand ($0.005 per GB of data scanned). Using Parquet not only improves query speed, but also reduces cost by minimizing the amount of data Athena needs to scan for each query. use SQL SELECT with some limitations Queries that are impossible or expensive/slow in DynamoDB, such as aggregations and JOINs, are fast and cheap in Athena. The two services combine each other in a perfect way, so that your application has optimized transactional storage and flexible analytical querying capabilities. We can use Athena to query all articles ever published and cross-reference with likes and HTTP metadata (source IP address, country, device type, etc). Even articles that were already expired by DynamoDB TTL (time-to-live) would continue to be available in the Data Lake. For example, which countries are liking the most articles? In the AWS Console, we get something like this: Queries can also be executed programmatically with or (e.g. ) to integrate anywhere we need this data. Athena API AWS SDKs Python’s boto3 Athena also supports JOINs. Here’s an example joining articles and HTTP metadata to analyze the most popular authors among readers of a particular country: Deployment CDK can deploy one Stack at a time. Since we have three, it’s necessary to specify which one when running the command. We do that by passing in the Stack ID as a CLI argument. For example, the following command will deploy the SlsBlogApiStack (id: ): CDK deploy sls-blog-api cdk deploy sls-blog-api Since all Stacks involve some type of permission granting, CDK asks for confirmation before deploying those resources. You can review the permissions requested and hit y when it’s good to go. Wrapping up We’ve covered how to structure CDK apps and add a bunch of AWS Resources to deploy with a simple command. CDK deploy If you’re new to the CDK - and as suggested early in the article -, it’s strongly recommended to follow AWS and . CDK workshop documentation Keep an eye on future publications as well, as is releasing other examples and tutorials to reap the most out of AWS serverless services with the power of infra automation with CDK or else. Dashbird Previously published at https://dashbird.io/blog/crash-course-aws-cdk-serverless-rest-api-data-lake-analytical-querying/

Quick Guide to AWS CDK and Serverless with REST API and Data Lake Analytical Querying

The exact difference between the Data Warehouse and AWS Data Lake. Let me demonstrate what AWS Data Lake is all about and depict the reason for it’s popularity. In the world of mazon eb ervice ( ), Amazon S3 is an amazing object container. Like any bucket, you can put content in it in a neat and orderly fashion, or you can just dump it in. But no matter how the data gets there, once it’s there, you need a way to organize it in a meaningful way so you can find it when you need it. This is where data lakes come in. A W S AWS AWS Data Lake A data lake is a centralized repository that allows you to store structured, semi-structured, and unstructured data at any scale. A data lake is an architectural concept that helps you , and , through a single set of tools. A data lake takes Amazon S3 buckets and inside the buckets. It doesn’t matter how the data got there or what kind it is, you can store both structured and unstructured data effectively in an Amazon S3 data lake AWS offers a set of tools to manage the entire Data Lake without treating each bucket as separate, unassociated objects manage multiple data types from multiple sources both structured unstructured organizes them by categorizing the data On-Premises Data Movement Data lakes allow you to . Data is collected from multiple sources and moved into the data lake in it’s original format. The process allows you to scale the data of , while from defining data structures, schemas and transformation. import any amount of data any size saving time Real-time Data Movement Data lakes allow you to import . Data can be collected from multiple stream data sources and moved into the data lakes in its original format. any amount of data that can come in real time Machine Learning Data lakes enables organizations to including reporting on and implementing where are built to forecast likely outcomes and suggest a range of prescribed actions to achieve the optimal result. generate different types of insights historical data machine learning models Analytics Data lakes allow various roles in the organization, such as Data Scientists, Data Developers and Business Analysts, to access data with their choice of analytic tools and frameworks. This includes such as Hadoop, Presto, and Apache Spark and commercial offerings from data warehouse and BI vendors. open source frameworks Data lakes allow you to run analytics system. without the need to move your data to a separate analytics Benefits of a data lake on AWS Are a solution. You can durably store a nearly unlimited amount of data using Amazon S3. cost-effective data storage Implement industry-leading . AWS uses stringent data security, compliance, privacy, and protection mechanisms. security and compliance Allow you to take advantage of to ingest data into your data lake. many different data collection and ingestion tools Help you to simply and efficiently. Use AWS Glue to understand the data within your data lake, prepare it, and load it reliably into data stores. Once AWS Glue catalogs your data, it is immediately searchable, can be queried, and is available for ETL processing. categorize and manage your data Help you turn data into . Harness the power of purpose-built analytic services for a wide range of use cases, such as interactive analysis, data processing using Apache Spark and Apache Hadoop, data warehousing, real-time analytics, operational analytics, dashboards, and visualizations. meaningful insights Business Problem Many businesses end up grouping data together into numerous storage locations called silos. These silos are rarely managed and maintained by the same team, which can be problematic. Inconsistencies in the way data was written, collected, aggregated, or filtered can cause problems when it is compared or combined for processing and analysis. For example, one team may use the address field to store both the street number and street name, while another team might use separate fields for street number and street name. When these datasets are combined, there is now an inconsistency in the way the address is stored, and it will make analysis very difficult. AWS Solution But by using Data Lakes, you can (a repository that contains raw data that is accessible by one department but isolated from the rest of that organization) and that is managed by a single team. That gives you a single, . Because data can be stored in its raw format, you don’t need to convert it, aggregate it, or filter it before you store it. Instead, you can leave that pre-processing to the system that processes it, rather than the system that stores it. break down data silos bring data into a single, central repository consistent source of truth In other words, you to make it usable. You , however it got there, however it was written. When you’re talking exabytes of data, you can’t afford to pre-process this data in every conceivable way it may need to be presented in a useful state. don’t have to transform the data keep the data in its original form Let’s talk about having a . When we talk about truth in relation to data, we mean the . Is it what it should be? Has it been altered? Can we validate the chain of custody? When creating a single source of truth, we’re creating a dataset, in this case . The bonus is that we know it to be and . It’s trustworthy. single source of truth trustworthiness of the data the data lake, which can be used for all processing and analytics consistent reliable Amazon S3 for a solution meeting these requirements and tools for analyzing the data without requiring movement. Data Lakes provide a single storage backbone Why Data Lake is Popular? As the volume of data has increased, so have the options for storing data. Traditional storage methods such as However, . These new options can confuse businesses that are trying to be and . data warehouses are still very popular and relevant. data lakes have become more popular recently financially wise technically relevant : data warehouses or data lakes? . They are different solutions that can be used together while . So which is better Neither and both to maintain existing data warehouses taking full advantage of the benefits of data lakes Data Warehouse A data warehouse is a coming from one or more data sources. Data flows into a data warehouse from transactional systems, relational databases, and other sources. These data sources can include structured, semistructured, and unstructured data. central repository of information . Data is within the data warehouse . A schema defines how data is stored within tables, columns, and rows. The to ensure integrity of the data. The transformation process often involves the steps required to make the source data conform to the schema. These data sources are transformed into structured data before they are stored in the data warehouse stored using a schema schema enforces constraints on the data Following the first successful of data into the data warehouse, the process of ingesting and the data can continue at a regular cadence. Business analysts, data scientists, and decision makers access the data through business intelligence (BI) tools, SQL clients, and other analytics applications. Businesses use , , and tools to extract insights from their data, , and . These reports, dashboards, and analytics tools are powered by data warehouses, which store data efficiently to minimize I/O and deliver query results at blazing speeds to hundreds and thousands of users concurrently. ingestion transforming reports dashboards analytics monitor business performance support decision making Comparison of Data Warehouse and Data Lake Analyzing a Data Warehouse For analysis to be most effective, it should be performed on data that has been processed and cleansed. This often means This data is then placed in a data warehouse. It is very common for data from many different parts of the organization to be combined into a single data warehouse. implementing an ETL operation to collect, cleanse, and transform the data. Analyzing a Data Lake Data lakes provide customers a means for including unstructured and semistructured data in their analytics. . This beyond the confines of a single data warehouse. Analytic queries can be run over cataloged data within a data lake extends the reach of analytics Businesses can , with high , , at and at . Businesses can easily access and analyze data in a variety of ways using the tools and frameworks of their choice in a high-performance, cost-effective way without having to move large amounts of data between storage and analytics systems. securely store data coming from applications and devices in its native format availability durability low cost, any scale I hope the above content is knowledgeable and would have given you a glance about the topic. Do follow me on & to get updates regarding all my blogs. If you really enjoy this post, then do show your love by banging the because learning has no limits. Medium LinkedIn Claps Button below Thank you for reading…!!

Why Is Amazon AWS Data Lake Gaining Popularity?

If you are following the trends in data science, it is more likely that you have heard the words big data, analytics, and machine learning. These days everyone wants to jump into this area of data science. Many of the software giants like Google, Amazon, Microsoft & etc. are already leading the way. However, it’s not that easy for a new business to enter this area of expertise due to many reasons. One of the core problems is that the data is scattered everywhere in different systems and their own databases. It’s likely these datasets will live for many years, hardly providing any value for its businesses. Why Not Create a Data Warehouse Instead? Although it would be wonderful if we can create a data warehouse in the first place (Check my article on for more details). Things to consider before building a serverless data warehouse However, there are several practical challenges in creating a data warehouse at a very early stage for business. One of the main reason is that it is difficult to know exactly which data sets are important and how they should be cleaned, enriched, and transformed to solve different business problems. Just imagine how much effort you need to put to identify data sets upfront from all the different systems and extract these data, doing the cleaning, enriching and transforming them such that they could be pushed into the data warehouse at a very early stage. Unless you have the data science experts in-house who also understand the business domain in depth to identify the required data sets and do the data preparations, it’s highly likely to screw up the things. What is a Data Lake? A data lake is a centralized repository to store all the structured and unstructured data. The real advantage is of a data lake is, it is possible to store data as-is where you can immediately start pushing data from different systems. These data could be in CSV files, Excel, Database queries, Log files & etc. could be stored in the data lake with the associated metadata without having to first structure the data. Once the data is available in the data lake over a time period, it’s possible to process the data later to run different types of analytics and big data processing for data visualization. It is also possible to use the data from the data lake for and for better-guided decisions. machine learning deep learning tools Creating a Data Lake for your Business For a business, to start creating a data lake and making sure that different data sets are added consistently over long periods of time requires a process and automation. To move in this direction, the first thing is to select a data lake technology and relevant tools to set up the data lake solution. 1. Setup a Data Lake Solution If you plan to create a data lake in a cloud, you can deploy a which uses serverless services underneath without incurring a huge cost upfront and a significant portion of the cost of data lake solution is variable and increases mainly based on the amount of data you put in. data lake on AWS 2. Identify Data Sources Then it is important to identify the data sources and the frequency of data being added to the data lake. Once the data sources are identified, make sure that the decisions are taken to either add the data sets as-is or to do the required level of cleaning and transformation of the data. It is also important to identify the metadata for individual types of data sets. 3. Establish Processes and Automation Since the data sets are is coming from different systems which might even be belonging to different departments of the business, it’s important to establish processes for consistency. For example, the HR department could be informed to publish the employee satisfaction after each survey which is carried out annually to the data lake. Another example is that the Account department publishes data on payroll monthly to the data lake. For operations that require a higher frequency of data publishing or time-consuming work, it is possible to automate the data sourcing process. This could involve, automating the extraction, transformation, and publishing of data to the data lake or at least automate some of the individual steps. 4. Ensure Right Governance After setting up the data lake, it’s important to make sure, that the data lake is functioning properly. It’s not only about putting data into the data lake but also to allow or to facilitate the data retrieval for other systems to generate data-driven informed business decisions. Otherwise, the data lake will end up as a data swamp in the long run with little to no use. 5. Using the Data from Data Lake After the data lake is properly set up and functioning for a reasonable period, you will be already collecting data to your data lake with the right amount of associated metadata. It will require to implement different processes with ETL (Extract Transform and Load) operations before using them to drive different business decisions. This is where the importance of Data Warehouses and Data Visualization tools come in. You can either publish the data to a Data Warehouse if there are more processing needs to be done in correlation with different data sets from other systems or directly feed into Data Visualization and Analytic tools like and . Microsoft Power BI AWS QuickSight So What’s Next? The most important thing comes next is to ask the right business questions which could be answered based on the availability of the data. Although it seems like it is too obvious, this is one of the areas many businesses make things so complex. Although there is a fully functioning data lake which produces useful insights for the business, it is important not to stop from there. The strength of a data lake lies with continuous development and the evaluation of the solutions.

What is a Data Lake and How to Create One for Your Business

Stories, News & Mentions about DATALAKE

Datalake’s Stories on HackerNoon