MinIO is a high-performance, S3 compatible object store. It is built for large scale AI/ML, data lake and database workloads. It runs on-prem and on any cloud (public or private) and from the data center to the edge. MinIO is software-defined and open source under GNU AGPL v3.

MinIO

minio

The Noonification: Data Migration from Dell ECS to MinIO (9/22/2024)

Using MinIO to Build a Retrieval Augmented Generation Chat Application

Die gebruik van MinIO om 'n herwinningsaansoek vir uitgebreide generasie-klets te bou

استخدام MinIO لبناء تطبيق دردشة يعتمد على الاسترجاع المعزز

Použití MinIO k vytvoření aplikace pro chat s rozšířenou generací načítání

Verwenden von MinIO zum Erstellen einer Retrieval Augmented Generation Chat-Anwendung

Uso de MinIO para crear una aplicación de chat de generación aumentada de recuperación

MinIO を使用して検索拡張生成チャット アプリケーションを構築する

ការប្រើប្រាស់ MinIO ដើម្បីបង្កើតកម្មវិធីជជែកកំសាន្តដែលបង្កើតឡើងវិញ

Mampiasa ny MiniIO hananganana fampiharana amin'ny chat Generation Augmented Retrieval

د ترلاسه کولو وده شوي نسل چیٹ غوښتنلیک جوړولو لپاره MinIO کارول

Použitie MinIO na zostavenie aplikácie na rozhovor s rozšírenou generáciou načítania

MinIO-dan peýdalanyp, gözlegiň giňeldilen nesil söhbetdeşlik programmasyny gurmak

Paggamit ng MiniIO para Bumuo ng Retrieval Augmented Generation Chat Application

Data Migration from Dell ECS to MinIO

نقل البيانات من Dell ECS إلى MinIO

Migración de datos de Dell ECS a MinIO

Dell ECS から MinIO へのデータ移行

ការផ្ទេរទិន្នន័យពី Dell ECS ទៅ MinIO

Migration ya ba données depuis Dell ECS na MinIO

ການຍ້າຍຂໍ້ມູນຈາກ Dell ECS ໄປຫາ MinIO

Миграција на податоци од Dell ECS во MinIO

Migrasi Data daripada Dell ECS kepada MiniIO

Dell ECS kaqmanta MinIO kaqman Willayta Migracion

Migrácia dát z Dell ECS na MinIO

ዳታ ማይግሬሽን ካብ Dell ECS ናብ MinIO

Dell ECS-dan MiniIO-ga ma'lumotlarni ko'chirish

Hands-on with Apache Iceberg on Your Laptop: Deep Dive with Apache Spark, Nessie, Minio and more!

Transitioning from Hitachi Content Platform (HCP) to MinIO has never been easier, thanks to our HCP-to-MinIO tool. Developed to support our customers' evolving storage needs, this tool is freely available on GitHub and greatly simplifies the migration process. Many organizations are transitioning to leverage MinIO's modern, scalable, and high-performance object storage optimized for AI infrastructure. This tutorial provides a comprehensive step-by-step guide to ensure a smooth and efficient transition to MinIO. Why Transition to MinIO? Customers often migrate from HCP to MinIO for several compelling reasons, particularly when optimizing for AI-driven applications: Modern Application Requirements: MinIO is Kubernetes-native and built for the cloud operating model, which means that it inherently supports most modern application stacks for analytics, statistical analysis and AI/ML workloads.


Performance at Scale: MinIO efficiently handles mixed object sizes, from very small to very large, ensuring optimal performance for data-intensive AI operations.


Flexibility: Software-defined, S3-compatible object storage offers greater flexibility and cloud neutrality compared to dedicated storage appliances. HCP-to-MinIO Migration Process During migration, it's recommended to write exclusively to MinIO and not to HCP. This approach ensures you start as you plan to go on: using MinIO as your primary object store. Here’s the process: Initial Read Operations: While migrating, getobject calls are directed to MinIO. If an object isn't found, it's read from HCP. Once all objects are copied, this fallback can be removed.


Preserving Metadata: The tool retains the modified timestamps of objects during the migration. Step 1: Create a Temporary Directory After downloading the HCP-to-MinIO tool the first step is to create a temporary directory for the results of the list command in Step 2: $ mkdir /tmp/data Next, set the following environment variables: $ export MINIO_ACCESS_KEY=<Your-MinIO-Access-Key>
$ export MINIO_SECRET_KEY=<Your-MinIO-Secret-Key>
$ export MINIO_ENDPOINT=https://<Your-MinIO-IP-Address>:9000
$ export MINIO_BUCKET=newbucket  # optional, if unspecified HCP bucket name is used Step 2: List Objects in HCP The list command generates a list of objects stored in HCP and saves it to a local file. This file can be split into smaller lists and processed in parallel by multiple client machines to speed up the migration. Here is the command: hcp-to-minio list --auth-token --namespace-url --host-header --namespace-url --dir And an example of what it might look like: $ hcp-to-minio list --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "HOST:s3testbucket.tenant.hcp.example.com" --namespace-url "https://hcp-vip.example.com" --dir "/tmp/data" Step 3: Migrate Objects to MinIO The migrate command uses the list generated in the first step to transfer objects from HCP to MinIO. The command: hcp-to-minio migrate --namespace-url --auth-token --host-header --data-dir --bucket --input-file The example: $ hcp-to-minio migrate --namespace-url https://finance.europe.hcp.example.com --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "s3testbucket.sandbox.hcp.example.com" --data-dir /mnt/data --bucket s3testbucket --input-file /tmp/data/to-migrate.txt Get Started The HCP-to-MinIO tool is designed to help you move to a modern, scalable object storage solution. Download the tool from GitHub and start your migration today. For any questions or additional support, contact us at hello@min.io. Transitioning from Hitachi Content Platform (HCP) to MinIO has never been easier, thanks to our HCP-to-MinIO tool. Developed to support our customers' evolving storage needs, this tool is freely available on GitHub and greatly simplifies the migration process. Many organizations are transitioning to leverage MinIO's modern, scalable, and high-performance object storage optimized for AI infrastructure. This tutorial provides a comprehensive step-by-step guide to ensure a smooth and efficient transition to MinIO. GitHub GitHub Why Transition to MinIO? Customers often migrate from HCP to MinIO for several compelling reasons, particularly when optimizing for AI-driven applications: Modern Application Requirements: MinIO is Kubernetes-native and built for the cloud operating model, which means that it inherently supports most modern application stacks for analytics, statistical analysis and AI/ML workloads. Performance at Scale: MinIO efficiently handles mixed object sizes, from very small to very large, ensuring optimal performance for data-intensive AI operations. Flexibility: Software-defined, S3-compatible object storage offers greater flexibility and cloud neutrality compared to dedicated storage appliances. Modern Application Requirements: MinIO is Kubernetes-native and built for the cloud operating model, which means that it inherently supports most modern application stacks for analytics, statistical analysis and AI/ML workloads. Modern Application Requirements : MinIO is Kubernetes-native and built for the cloud operating model, which means that it inherently supports most modern application stacks for analytics, statistical analysis and AI/ML workloads. Modern Application Requirements Performance at Scale: MinIO efficiently handles mixed object sizes, from very small to very large, ensuring optimal performance for data-intensive AI operations. Performance at Scale : MinIO efficiently handles mixed object sizes, from very small to very large, ensuring optimal performance for data-intensive AI operations. Performance at Scale Flexibility: Software-defined, S3-compatible object storage offers greater flexibility and cloud neutrality compared to dedicated storage appliances. Flexibility : Software-defined, S3-compatible object storage offers greater flexibility and cloud neutrality compared to dedicated storage appliances. Flexibility HCP-to-MinIO Migration Process During migration, it's recommended to write exclusively to MinIO and not to HCP. This approach ensures you start as you plan to go on: using MinIO as your primary object store. Here’s the process: Initial Read Operations: While migrating, getobject calls are directed to MinIO. If an object isn't found, it's read from HCP. Once all objects are copied, this fallback can be removed. Preserving Metadata: The tool retains the modified timestamps of objects during the migration. Initial Read Operations: While migrating, getobject calls are directed to MinIO. If an object isn't found, it's read from HCP. Once all objects are copied, this fallback can be removed. Initial Read Operations : While migrating, getobject calls are directed to MinIO. If an object isn't found, it's read from HCP. Once all objects are copied, this fallback can be removed. Initial Read Operations getobject Preserving Metadata: The tool retains the modified timestamps of objects during the migration. Preserving Metadata : The tool retains the modified timestamps of objects during the migration. Preserving Metadata Step 1: Create a Temporary Directory After downloading the HCP-to-MinIO tool the first step is to create a temporary directory for the results of the list command in Step 2: $ mkdir /tmp/data $ mkdir /tmp/data Next, set the following environment variables: $ export MINIO_ACCESS_KEY=<Your-MinIO-Access-Key>
$ export MINIO_SECRET_KEY=<Your-MinIO-Secret-Key>
$ export MINIO_ENDPOINT=https://<Your-MinIO-IP-Address>:9000
$ export MINIO_BUCKET=newbucket  # optional, if unspecified HCP bucket name is used $ export MINIO_ACCESS_KEY=<Your-MinIO-Access-Key>
$ export MINIO_SECRET_KEY=<Your-MinIO-Secret-Key>
$ export MINIO_ENDPOINT=https://<Your-MinIO-IP-Address>:9000
$ export MINIO_BUCKET=newbucket  # optional, if unspecified HCP bucket name is used Step 2: List Objects in HCP The list command generates a list of objects stored in HCP and saves it to a local file. This file can be split into smaller lists and processed in parallel by multiple client machines to speed up the migration. list Here is the command: hcp-to-minio list --auth-token --namespace-url --host-header --namespace-url --dir hcp-to-minio list --auth-token --namespace-url --host-header --namespace-url --dir And an example of what it might look like: $ hcp-to-minio list --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "HOST:s3testbucket.tenant.hcp.example.com" --namespace-url "https://hcp-vip.example.com" --dir "/tmp/data" $ hcp-to-minio list --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "HOST:s3testbucket.tenant.hcp.example.com" --namespace-url "https://hcp-vip.example.com" --dir "/tmp/data" Step 3: Migrate Objects to MinIO The migrate command uses the list generated in the first step to transfer objects from HCP to MinIO. migrate The command: hcp-to-minio migrate --namespace-url --auth-token --host-header --data-dir --bucket --input-file hcp-to-minio migrate --namespace-url --auth-token --host-header --data-dir --bucket --input-file The example: $ hcp-to-minio migrate --namespace-url https://finance.europe.hcp.example.com --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "s3testbucket.sandbox.hcp.example.com" --data-dir /mnt/data --bucket s3testbucket --input-file /tmp/data/to-migrate.txt $ hcp-to-minio migrate --namespace-url https://finance.europe.hcp.example.com --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "s3testbucket.sandbox.hcp.example.com" --data-dir /mnt/data --bucket s3testbucket --input-file /tmp/data/to-migrate.txt Get Started The HCP-to-MinIO tool is designed to help you move to a modern, scalable object storage solution. Download the tool from GitHub and start your migration today. For any questions or additional support, contact us at hello@min.io. GitHub GitHub

Migrate to AI-Ready Infrastructure: Hitachi Content Platform to MinIO

Infraestructura AI-Ready ukar jaqukipaña: Plataforma de Contenido Hitachi ukar MinIO ukar sarxataña

Пераход на AI-Ready Infrastructure: Hitachi Content Platform на MinIO

Migrer til AI-Ready Infrastructure: Hitachi Content Platform til MinIO

Μεταφορά στην υποδομή AI-Ready: Πλατφόρμα περιεχομένου Hitachi στο MinIO

Migración a una infraestructura preparada para IA: Hitachi Content Platform a MinIO

مهاجرت به زیرساخت AI-Ready: پلتفرم محتوای هیتاچی به MinIO

AI-रेडी इंफ्रास्ट्रक्चर पर माइग्रेट करें: हिताची कंटेंट प्लेटफॉर्म से मिनियो तक

Migráció az AI-ready infrastruktúrára: a Hitachi Content Platform a MinIO-ra

AI対応インフラストラクチャへの移行：日立コンテンツプラットフォームからMinIOへ

Migrējiet uz AI gatavu infrastruktūru: Hitachi satura platforma uz MinIO

د AI چمتو زیربنا ته مهاجرت: MinIO ته د هیتاچي مینځپانګې پلیټ فارم

AI-Ready انفراسٹرکچر پر منتقل کریں: Hitachi Content Platform MinIO پر

The MinIO DataPod: A Reference Architecture for Exascale Computing

MinIO DataPod: Exascale Computing এর জন্য একটি রেফারেন্স আর্কিটেকচার

Der MinIO DataPod: Eine Referenzarchitektur für Exascale-Computing

MinIO DataPod: una arquitectura de referencia para la computación a escala de exaescala

Le MinIO DataPod : une architecture de référence pour le calcul exascale

मिनियो डेटापॉड: एक्सास्केल कंप्यूटिंग के लिए एक संदर्भ आर्किटेक्चर

MinIO DataPod: エクサスケール コンピューティングのリファレンス アーキテクチャ

MinIO DataPod: 엑사스케일 컴퓨팅을 위한 참조 아키텍처

O MinIO DataPod: Uma Arquitetura de Referência para Computação em Exaescala

MinIO DataPod: эталонная архитектура для эксафлопсных вычислений

MinIO DataPod: Exascale Hesaplama için Bir Referans Mimarisi

MinIO DataPod: Kiến trúc tham chiếu cho điện toán Exascale

MinIO DataPod：百亿亿次计算的参考架构

The Noonification: Limit Cloud Data Costs with MinIO on Equinix (6/30/2024)

In our previous discussion, we introduced the MinIO Object Store Firewall, a novel security component that goes beyond traditional network and application security layers. Neither IP-based firewalls nor application firewalls are designed for data. That is why we built the MinIO Enterprise Object Store Firewall – because in the modern enterprise, data is what must be protected and an S3-aware firewall doesn’t exist. This S3-aware data firewall is crucial for modern data protection, operating at the storage layer to safeguard your data comprehensively. The MinIO Enterprise Object Store Firewall is designed specifically to work with applications using MinIO object store and its API endpoints. The Enterprise Firewall is lightweight, powerful, flexible and extensible. Let’s delve into setting up this advanced firewall, designed to secure your data in today’s increasingly complex digital landscape. Enable and Configure Firewall Let's use the Enterprise Console to set up the firewall. Follow the below steps to enable and configure the Firewall. Configure TLS for Secure Communication As part of the MinIO Enterprise Suite, we’ve always recommended to ensure you enable TLS on MinIO Enterprise Object Store so that even inter cluster communications are encrypted. With that same spirit, we support TLS for the Enterprise Firewall as well. This ensures that any connection to the MinIO Object Store via the Firewall is encrypted end to end. For enhanced security, configure TLS settings when launching the firewall with Let’s Encrypt. Precedence of Anonymous Rule In our MinIO Enterprise Firewall configuration, you’ll encounter two distinct rules for anonymous access: Global Anonymous Setting: At the start when you enable firewall, we specify a global setting that allows anonymous access across all buckets unless a more specific rule denies it. Bucket Specific Anonymous: After configuring the global setting, under each individual rules, we can set a more specific rule that overrides the global setting, effectively denying anonymous access In this case, even though we initially toggled Global Anonymous to Allow for all buckets, because we set another more specific rule below it, which also applies to all buckets, the one we set later takes precedence over the global setting. In other words, anonymous bucket access will be denied for all buckets. Load Balance across MinIO nodes As an added bonus, because we need to define all the MinIO nodes as a backend for the Firewall, the Firewall doubles as a LoadBalancer, negating the need for a separate load balancer between the firewall and MinIO nodes. This eliminates further complexity and ensures there is no single point of failure in case one of the MinIO backend goes offline. Please note, you need to have another level of redundancy to ensure when connecting to Enterprise Firewall the incoming connections are also distributed to multiple Enterprise Firewall instances. This configuration is beyond the scope of this blog but its something to keep in mind. Health Checks and Monitoring The health check enabled you to see if the Firewall is in a healthy state. You can check the health and liveliness as follows. If everything is Green that will indicate that the firewall is functioning properly. Final Thoughts With the MinIO Enterprise Firewall, gone are the days of wrestling with complex IPtables and unclear access policies.Our firewall solution simplifies your security by focusing exclusively on the essential rules required for your object store and API interactions. It is optimized to ensure there are no latency or unforeseen rules, blocking access to your MinIO object store. Moreover, the Enterprise Firewall is fully supported by our awesome team at SUBNET where we can help you by architecting the Enterprise Firewall the right way to work with MinIO object store and troubleshoot any future issues. So what are you waiting for? If you have any questions on MinIO Enterprise Object Store Firewall be sure to reach out to us on Slack or hello@min.io! In our previous discussion , we introduced the MinIO Object Store Firewall, a novel security component that goes beyond traditional network and application security layers. Neither IP-based firewalls nor application firewalls are designed for data. That is why we built the MinIO Enterprise Object Store Firewall – because in the modern enterprise, data is what must be protected and an S3-aware firewall doesn’t exist. This S3-aware data firewall is crucial for modern data protection, operating at the storage layer to safeguard your data comprehensively. previous discussion previous discussion The MinIO Enterprise Object Store Firewall is designed specifically to work with applications using MinIO object store and its API endpoints. The Enterprise Firewall is lightweight, powerful, flexible and extensible. Let’s delve into setting up this advanced firewall, designed to secure your data in today’s increasingly complex digital landscape. Enable and Configure Firewall Let's use the Enterprise Console to set up the firewall. Follow the below steps to enable and configure the Firewall. Configure TLS for Secure Communication As part of the MinIO Enterprise Suite, we’ve always recommended to ensure you enable TLS on MinIO Enterprise Object Store so that even inter cluster communications are encrypted. With that same spirit, we support TLS for the Enterprise Firewall as well. This ensures that any connection to the MinIO Object Store via the Firewall is encrypted end to end. For enhanced security, configure TLS settings when launching the firewall with Let’s Encrypt. Precedence of Anonymous Rule In our MinIO Enterprise Firewall configuration, you’ll encounter two distinct rules for anonymous access: Global Anonymous Setting: At the start when you enable firewall, we specify a global setting that allows anonymous access across all buckets unless a more specific rule denies it. Global Anonymous Setting: Bucket Specific Anonymous: After configuring the global setting, under each individual rules, we can set a more specific rule that overrides the global setting, effectively denying anonymous access Bucket Specific Anonymous: In this case, even though we initially toggled Global Anonymous to Allow for all buckets, because we set another more specific rule below it, which also applies to all buckets, the one we set later takes precedence over the global setting. In other words, anonymous bucket access will be denied for all buckets. Load Balance across MinIO nodes As an added bonus, because we need to define all the MinIO nodes as a backend for the Firewall, the Firewall doubles as a LoadBalancer, negating the need for a separate load balancer between the firewall and MinIO nodes. This eliminates further complexity and ensures there is no single point of failure in case one of the MinIO backend goes offline. Please note, you need to have another level of redundancy to ensure when connecting to Enterprise Firewall the incoming connections are also distributed to multiple Enterprise Firewall instances. This configuration is beyond the scope of this blog but its something to keep in mind. Health Checks and Monitoring The health check enabled you to see if the Firewall is in a healthy state. You can check the health and liveliness as follows. If everything is Green that will indicate that the firewall is functioning properly. Final Thoughts With the MinIO Enterprise Firewall, gone are the days of wrestling with complex IPtables and unclear access policies.Our firewall solution simplifies your security by focusing exclusively on the essential rules required for your object store and API interactions. It is optimized to ensure there are no latency or unforeseen rules, blocking access to your MinIO object store. Moreover, the Enterprise Firewall is fully supported by our awesome team at SUBNET where we can help you by architecting the Enterprise Firewall the right way to work with MinIO object store and troubleshoot any future issues. SUBNET SUBNET So what are you waiting for? If you have any questions on MinIO Enterprise Object Store Firewall be sure to reach out to us on Slack or hello@min.io ! Slack Slack hello@min.io hello@min.io

A Closer Look Into the MinIO Enterprise Object Store Firewall

MinIO এন্টারপ্রাইজ অবজেক্ট স্টোর ফায়ারওয়ালের একটি ঘনিষ্ঠভাবে দেখুন

Ein genauerer Blick auf die MinIO Enterprise Object Store Firewall

Una mirada más cercana al firewall del almacén de objetos empresariales MinIO

Un examen plus approfondi du pare-feu MinIO Enterprise Object Store

मिनियो एंटरप्राइज़ ऑब्जेक्ट स्टोर फ़ायरवॉल पर एक नज़दीकी नज़र

MinIO エンタープライズ オブジェクト ストア ファイアウォールの詳細

MinIO Enterprise 개체 저장소 방화벽 자세히 살펴보기

Uma análise mais detalhada do firewall do MinIO Enterprise Object Store

Более пристальный взгляд на брандмауэр MinIO Enterprise Object Store

MinIO Kurumsal Nesne Deposu Güvenlik Duvarına Daha Yakından Bir Bakış

Xem xét kỹ hơn về Tường lửa lưu trữ đối tượng doanh nghiệp MinIO

深入了解 MinIO 企业对象存储防火墙

Effortlessly Launch LangChain APIs with LangServe and MinIO Integration

অনায়াসে LangServe এবং MinIO ইন্টিগ্রেশন সহ LangChain API চালু করুন

Müheloses Starten von LangChain-APIs mit LangServe- und MinIO-Integration

Lance las API de LangChain sin esfuerzo con la integración de LangServe y MinIO

Lancez sans effort les API LangChain avec l'intégration LangServe et MinIO

LangServe और MinIO एकीकरण के साथ LangChain API को आसानी से लॉन्च करें

LangServe と MinIO の統合により LangChain API を簡単に起動

LangServe 및 MinIO 통합을 통해 LangChain API를 손쉽게 실행

Inicie APIs LangChain sem esforço com integração LangServe e MinIO

Легко запускайте API-интерфейсы LangChain с интеграцией LangServe и MinIO.

LangServe ve MinIO Entegrasyonu ile LangChain API'lerini Zahmetsizce Başlatın

Dễ dàng khởi chạy API LangChain với tích hợp LangServe và MinIO

通过 LangServe 和 MinIO 集成轻松启动 LangChain API

Developing Next-Gen Data Solutions: SingleStore, MinIO, and the Modern Datalake Stack

নেক্সট-জেন ডেটা সলিউশন তৈরি করা: একক স্টোর, মিনিও এবং আধুনিক ডেটালেক স্ট্যাক

Entwicklung von Datenlösungen der nächsten Generation: SingleStore, MinIO und der moderne Datalake-Stack

Desarrollo de soluciones de datos de próxima generación: SingleStore, MinIO y Modern Datalake Stack

Développer des solutions de données de nouvelle génération : SingleStore, MinIO et la pile Datalake moderne

अगली पीढ़ी के डेटा समाधान विकसित करना: सिंगलस्टोर, मिनियो और आधुनिक डेटालेक स्टैक

次世代データソリューションの開発: SingleStore、MinIO、最新のデータレイク スタック

차세대 데이터 솔루션 개발: SingleStore, MinIO 및 최신 Datalake 스택

Desenvolvendo soluções de dados de última geração: SingleStore, MinIO e Modern Datalake Stack

Разработка решений для обработки данных следующего поколения: SingleStore, MinIO и современный стек Datalake

Yeni Nesil Veri Çözümlerinin Geliştirilmesi: SingleStore, MinIO ve Modern Datalake Stack

Phát triển các giải pháp dữ liệu thế hệ tiếp theo: SingleStore, MinIO và Datalake Stack hiện đại

开发下一代数据解决方案：SingleStore、MinIO 和现代 Datalake 堆栈

The public cloud changed the way companies build, deploy and manage their applications - mostly for the better. As you’re getting started, the public cloud supplies the infrastructure, services, enablement and maintenance to be up and running quickly. It provides ultimate scalability, in almost unlimited fashion, up and down, to provide you with the necessary resources no matter the load on your application. You could essentially go international on fully supported infrastructure at the flip of a switch. As time goes on, data from things like logs, customer input and application generated content can start to build up, sometimes into the petabyte and in very rare cases, the exabyte level. So, data that isn’t necessarily needed for your application’s active operation can suddenly bring unwanted surprises when your bill comes. It’s not just that either. Nearly 60% of public cloud costs are variable. In this blog, I’ll discuss how MinIO and Equinix have partnered to help combat these costs, how it helps you sleep at night from an operational perspective and offer a real-world example of a customer’s achievement of savings. About MinIO and Equinix Equinix is “the world’s digital infrastructure company.” It was founded in Silicon Valley in 1998 to be a vendor neutral data center provider. They securely interconnect some of the world’s largest networks with equality and neutrality in mind. In 2020, Equinix expanded on this great offering by acquiring Packet. In doing so, Equinix was able to add on-demand server provisioning capability to its extensive data center network. MinIO is the leading high performance object store in the market. It runs anywhere and on any hardware. Kubernetes-native and fully S3 API compatible it is the perfect pair for Equinix’s bare metal and colocation services. By combining MinIO and Equinix, developers can quickly and easily deploy S3 compatible persistent storage for any application, whether it’s hosted on Equinix, a public cloud provider, or even on-prem, allowing the flexibility for single, multi or hybrid cloud approaches. Why you can rely on MinIO and Equinix Flexibility Equinix provides multiple paths towards deploying your data and high intensity workloads within their 260+ global data centers across 71 metros in 33 countries. Equinix Colocation - Like any colocation service, this option involves buying, building, and managing your own hardware. So, you’d deploy MinIO to your server, either in a container or on bare metal, and then deploy that server setup to one or more Equinix data centers. This route is available at any Equinix data center around the world and provides the highest total cost savings when compared to the public cloud.


Equinix Metal - With this option, you’re taking full advantage of Equinix’s capabilities by using their bare metal server infrastructure as a service. There’s no need to supply, build, or manage your own hardware. Everything is done by Equinix’s Metal team. Equinix Metal also supports a range of deployment configurations from single-tenant, multi-user to multi-user and multi-tenant. The cost savings isn’t as high compared to the colocation approach, but it gives you a little bit of the best of both worlds. You don’t have to worry about the supply and maintenance of hardware while still saving money by offloading data and workloads that don’t need to run in the public cloud. While these are the main two options with Equinix, they are also open to custom approaches for very large deployments. Performance Not every use case requires ultimate performance. Sometimes you just need a less expensive but public cloud accessible place to store large amounts of logs and historical data. However, if you currently do need performance or might in the future, rest assured, MinIO and Equinix have you covered. With Equinix’s public cloud adjacency and interconnectivity, your data is only 1-2ms RTT from the major public clouds and <10ms RTT from your end customer’s device. MinIO will likely max out any network connectivity you can procure. It is the world's fastest object store with published GET and PUT results that exceed 325 GiB/sec and 165 GiB/sec, respectively, on 32 nodes of NVMe drives and a 100GbE network. Saving money with the cloud you control Pricing One of the unique aspects of the Equinix and MinIO collaboration is the operational simplicity and financial certainty they offer with fixed annual pricing. The pricing model for Equinix and MinIO is transparent, predictable and straightforward. Equinix offers a range of pricing options from hourly on demand to custom provisioning. In the case of MinIO, the pricing model is driven by capacity - there are no charges for egress and/or API calls. Everything is included. A real-world example While we don’t have the ability to share the identity of this customer, the following example will give a great representation of the cost savings that can be achieved via either Equinix deployment model as this customer reviewed both options. First, let’s look at the sizing and assumptions data used for the comparison. All of this is based on the customer’s last bill with one of the large public cloud providers. One point to note is that this comparison is not a knock on the public cloud providers. Both Equinix and MinIO partner with all the major cloud providers. This simply illustrates the modern cloud cycle of starting in the cloud and operationalize and create efficiencies with the private cloud. When we plug in the numbers on this based on the customer’s previous bill and Equinix and MinIO’s transparent pricing (using very conservative estimates), it starts to take shape. Finally, if we do all the math we end up with the following savings. Keep in mind, these savings are based on conservative figures on the MinIO and Equinix side. While the savings is quite substantial already, these figures could rise once real numbers come in from actual usage. Another thing that wasn’t even factored in that could have made this estimate even higher would have been factoring in any data movement from storage tiers. In the public cloud services, this can become very costly. In MinIO on Equinix, there is no charge for moving data where you want it. As you can see, the savings add up quickly. This is a very large use case, but it gives a clear look into what kind of savings can be had by repatriating your data into the cloud you control. Summary There is a great quote from Andressen Horowitz’s Martin Casado in his and Sarah Wang’s seminal piece The Paradox of the Cloud where he notes the key paradox “You’re crazy if you don’t start in the cloud; you’re crazy if you stay on it.” This is why companies who understand the cloud operating model and understand their workloads are voting with their feet. They can save anywhere from 20% up to 60% by moving to Equinix and running the same workloads. No performance impact, no operational impact and potentially better performance. That’s what’s called a win-win. If you’d like to look at how much you can save by repatriating your data, contact us, or hit up the Ask an Expert channel and start the conversation. For a deeper dive, see our blog on how to migrate from the public cloud to MinIO on Equinix. The public cloud changed the way companies build, deploy and manage their applications - mostly for the better. As you’re getting started, the public cloud supplies the infrastructure, services, enablement and maintenance to be up and running quickly. It provides ultimate scalability, in almost unlimited fashion, up and down, to provide you with the necessary resources no matter the load on your application. You could essentially go international on fully supported infrastructure at the flip of a switch. As time goes on, data from things like logs, customer input and application generated content can start to build up, sometimes into the petabyte and in very rare cases, the exabyte level. So, data that isn’t necessarily needed for your application’s active operation can suddenly bring unwanted surprises when your bill comes. It’s not just that either. Nearly 60% of public cloud costs are variable. In this blog, I’ll discuss how MinIO and Equinix have partnered to help combat these costs, how it helps you sleep at night from an operational perspective and offer a real-world example of a customer’s achievement of savings. About MinIO and Equinix Equinix is “the world’s digital infrastructure company.” It was founded in Silicon Valley in 1998 to be a vendor neutral data center provider. They securely interconnect some of the world’s largest networks with equality and neutrality in mind. In 2020, Equinix expanded on this great offering by acquiring Packet. In doing so, Equinix was able to add on-demand server provisioning capability to its extensive data center network. MinIO is the leading high performance object store in the market. It runs anywhere and on any hardware. Kubernetes-native and fully S3 API compatible it is the perfect pair for Equinix’s bare metal and colocation services. By combining MinIO and Equinix, developers can quickly and easily deploy S3 compatible persistent storage for any application, whether it’s hosted on Equinix, a public cloud provider, or even on-prem, allowing the flexibility for single, multi or hybrid cloud approaches. Why you can rely on MinIO and Equinix Flexibility Equinix provides multiple paths towards deploying your data and high intensity workloads within their 260+ global data centers across 71 metros in 33 countries. Equinix Colocation - Like any colocation service, this option involves buying, building, and managing your own hardware. So, you’d deploy MinIO to your server, either in a container or on bare metal, and then deploy that server setup to one or more Equinix data centers. This route is available at any Equinix data center around the world and provides the highest total cost savings when compared to the public cloud. Equinix Metal - With this option, you’re taking full advantage of Equinix’s capabilities by using their bare metal server infrastructure as a service. There’s no need to supply, build, or manage your own hardware. Everything is done by Equinix’s Metal team. Equinix Metal also supports a range of deployment configurations from single-tenant, multi-user to multi-user and multi-tenant. The cost savings isn’t as high compared to the colocation approach, but it gives you a little bit of the best of both worlds. You don’t have to worry about the supply and maintenance of hardware while still saving money by offloading data and workloads that don’t need to run in the public cloud. Equinix Colocation - Like any colocation service, this option involves buying, building, and managing your own hardware. So, you’d deploy MinIO to your server, either in a container or on bare metal, and then deploy that server setup to one or more Equinix data centers. This route is available at any Equinix data center around the world and provides the highest total cost savings when compared to the public cloud. Equinix Colocation - Like any colocation service, this option involves buying, building, and managing your own hardware. So, you’d deploy MinIO to your server, either in a container or on bare metal, and then deploy that server setup to one or more Equinix data centers. This route is available at any Equinix data center around the world and provides the highest total cost savings when compared to the public cloud. Equinix Colocation Equinix Metal - With this option, you’re taking full advantage of Equinix’s capabilities by using their bare metal server infrastructure as a service. There’s no need to supply, build, or manage your own hardware. Everything is done by Equinix’s Metal team. Equinix Metal also supports a range of deployment configurations from single-tenant, multi-user to multi-user and multi-tenant. The cost savings isn’t as high compared to the colocation approach, but it gives you a little bit of the best of both worlds. You don’t have to worry about the supply and maintenance of hardware while still saving money by offloading data and workloads that don’t need to run in the public cloud. Equinix Metal - With this option, you’re taking full advantage of Equinix’s capabilities by using their bare metal server infrastructure as a service. There’s no need to supply, build, or manage your own hardware. Everything is done by Equinix’s Metal team. Equinix Metal also supports a range of deployment configurations from single-tenant, multi-user to multi-user and multi-tenant. The cost savings isn’t as high compared to the colocation approach, but it gives you a little bit of the best of both worlds. You don’t have to worry about the supply and maintenance of hardware while still saving money by offloading data and workloads that don’t need to run in the public cloud. Equinix Metal While these are the main two options with Equinix, they are also open to custom approaches for very large deployments. Performance Not every use case requires ultimate performance. Sometimes you just need a less expensive but public cloud accessible place to store large amounts of logs and historical data. However, if you currently do need performance or might in the future, rest assured, MinIO and Equinix have you covered. With Equinix’s public cloud adjacency and interconnectivity, your data is only 1-2ms RTT from the major public clouds and <10ms RTT from your end customer’s device. MinIO will likely max out any network connectivity you can procure. It is the world's fastest object store with published GET and PUT results that exceed 325 GiB/sec and 165 GiB/sec, respectively, on 32 nodes of NVMe drives and a 100GbE network. Saving money with the cloud you control Pricing One of the unique aspects of the Equinix and MinIO collaboration is the operational simplicity and financial certainty they offer with fixed annual pricing. The pricing model for Equinix and MinIO is transparent, predictable and straightforward. Equinix offers a range of pricing options from hourly on demand to custom provisioning. In the case of MinIO, the pricing model is driven by capacity - there are no charges for egress and/or API calls. Everything is included. A real-world example While we don’t have the ability to share the identity of this customer, the following example will give a great representation of the cost savings that can be achieved via either Equinix deployment model as this customer reviewed both options. First, let’s look at the sizing and assumptions data used for the comparison. All of this is based on the customer’s last bill with one of the large public cloud providers. One point to note is that this comparison is not a knock on the public cloud providers. Both Equinix and MinIO partner with all the major cloud providers. This simply illustrates the modern cloud cycle of starting in the cloud and operationalize and create efficiencies with the private cloud. When we plug in the numbers on this based on the customer’s previous bill and Equinix and MinIO’s transparent pricing (using very conservative estimates), it starts to take shape. Finally, if we do all the math we end up with the following savings. Keep in mind, these savings are based on conservative figures on the MinIO and Equinix side. While the savings is quite substantial already, these figures could rise once real numbers come in from actual usage. Another thing that wasn’t even factored in that could have made this estimate even higher would have been factoring in any data movement from storage tiers. In the public cloud services, this can become very costly. In MinIO on Equinix, there is no charge for moving data where you want it. As you can see, the savings add up quickly. This is a very large use case, but it gives a clear look into what kind of savings can be had by repatriating your data into the cloud you control. Summary There is a great quote from Andressen Horowitz’s Martin Casado in his and Sarah Wang’s seminal piece The Paradox of the Cloud where he notes the key paradox “You’re crazy if you don’t start in the cloud; you’re crazy if you stay on it.” This is why companies who understand the cloud operating model and understand their workloads are voting with their feet. They can save anywhere from 20% up to 60% by moving to Equinix and running the same workloads. No performance impact, no operational impact and potentially better performance. That’s what’s called a win-win. The Paradox of the Cloud The Paradox of the Cloud If you’d like to look at how much you can save by repatriating your data, contact us , or hit up the Ask an Expert channel and start the conversation. For a deeper dive, see our blog on how to migrate from the public cloud to MinIO on Equinix . contact us Ask an Expert how to migrate from the public cloud to MinIO on Equinix

Limit Cloud Data Costs with MinIO on Equinix

Equinix এ MinIO এর সাথে ক্লাউড ডেটা খরচ সীমিত করুন

Begrenzen Sie die Cloud-Datenkosten mit MinIO auf Equinix

Limite los costos de datos en la nube con MinIO en Equinix

Limitez les coûts des données cloud avec MinIO sur Equinix

Equinix पर MinIO के साथ क्लाउड डेटा लागत को सीमित करें

Equinix の MinIO でクラウド データ コストを制限

Equinix의 MinIO로 클라우드 데이터 비용 제한

Limite os custos de dados na nuvem com MinIO na Equinix

Ограничьте затраты на облачные данные с помощью MinIO на Equinix

Equinix'te MinIO ile Bulut Veri Maliyetlerini Sınırlayın

Giới hạn chi phí dữ liệu đám mây với MinIO trên Equinix

使用 Equinix 上的 MinIO 限制云数据成本

How to Deploy MinIO and Trino with Kubernetes

কুবারনেটসের সাথে MinIO এবং Trino কীভাবে স্থাপন করবেন

So stellen Sie MinIO und Trino mit Kubernetes bereit

Cómo implementar MinIO y Trino con Kubernetes

Comment déployer MinIO et Trino avec Kubernetes

कुबेरनेट्स के साथ मिनिओ और ट्रिनो को कैसे तैनात करें

Kubernetes を使用して MinIO と Trino をデプロイする方法

Kubernetes를 사용하여 MinIO 및 Trino를 배포하는 방법

Como implantar MinIO e Trino com Kubernetes

Как развернуть MinIO и Trino с помощью Kubernetes

MinIO ve Trino'nun Kubernetes ile Dağıtılması

Cách triển khai MinIO và Trino với Kubernetes

如何使用 Kubernetes 部署 MinIO 和 Trino

We are back with another article for the HackerNoon Company of the Week series! Every week we feature an awesome tech brand from our tech company database, making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike.This week, we present our business blogging partnerMinIO - a high-performance, S3 compatible object store, built for large scale AI/ML, data lake and database workloads. It runs on-prem and on any cloud (public or private) and from the data center to the edge. MinIO also offers features like data replication, lifecycle management and high availability, making it a top choice for modern cloud-native applications. Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon! Meet MinIO: #FunFact Do you know that MinIO is open source under GNU AGPL v3? Open source is the key driver of technological innovation and here's how the company shares its support and belief in the role of open source in AI revolution:"Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." MinIO’s Top Stories on HackerNoon MinIO has published 43 stories via HackerNoon Business Blogging Program, and has generated 2 years 23 days and 29 minutes of reading time! Here are our top 3 favorite stories by MinIO: Open-Source: The Next Step in AI Revolution
Quantum State: How Two Things Can Be True at the Same Time
Analyzing the Pros, Cons, and Risks of LLMs We are back with another article for the HackerNoon Company of the Week series ! Every week we feature an awesome tech brand from our tech company database , making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. This week, we present our business blogging partner MinIO - a high-performance, S3 compatible object store, built for large scale AI/ML, data lake and database workloads. It runs on-prem and on any cloud (public or private) and from the data center to the edge. MinIO also offers features like data replication, lifecycle management and high availability, making it a top choice for modern cloud-native applications. Company of the Week series Company of the Week tech company database tech company database MinIO MinIO Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon! Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon ! Request Your Tech Company Page on HackerNoon Request Your Tech Company Page on HackerNoon Meet MinIO: #FunFact Do you know that MinIO is open source under GNU AGPL v3? Open source is the key driver of technological innovation and here's how the company shares its support and belief in the role of open source in AI revolution : "Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." role of open source in AI revolution role of open source in AI revolution "Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." MinIO’s Top Stories on HackerNoon MinIO has published 43 stories via HackerNoon Business Blogging Program , and has generated 2 years 23 days and 29 minutes of reading time! Here are our top 3 favorite stories by MinIO: MinIO MinIO HackerNoon Business Blogging Program HackerNoon Business Blogging Program Open-Source: The Next Step in AI Revolution Quantum State: How Two Things Can Be True at the Same Time Analyzing the Pros, Cons, and Risks of LLMs Open-Source: The Next Step in AI Revolution Open-Source: The Next Step in AI Revolution Quantum State: How Two Things Can Be True at the Same Time Quantum State: How Two Things Can Be True at the Same Time Analyzing the Pros, Cons, and Risks of LLMs Analyzing the Pros, Cons, and Risks of LLMs

Meet MinIO: The HackerNoon Company of the Week

মিট মিনিও: সপ্তাহের হ্যাকারনুন কোম্পানি

Lernen Sie MinIO kennen: Das HackerNoon-Unternehmen der Woche

Conozca MinIO: la empresa de la semana de HackerNoon

Rencontrez MinIO : l'entreprise HackerNoon de la semaine

मिलिए मिनियो से: इस सप्ताह की हैकरनून कंपनी

MinIO のご紹介: 今週の HackerNoon 企業

MinIO를 만나보세요: 금주의 HackerNoon 회사

Conheça MinIO: a empresa HackerNoon da semana

Встречайте MinIO: компания недели HackerNoon

MinIO ile Tanışın: Haftanın HackerNoon Şirketi

Gặp gỡ MinIO: Công ty HackerNoon của tuần

认识 MinIO：本周 HackerNoon 公司

Using MinIO and Python to Create Scalable and Efficient Event-driven Applications

স্কেলযোগ্য এবং দক্ষ ইভেন্ট-চালিত অ্যাপ্লিকেশন তৈরি করতে MinIO এবং Python ব্যবহার করে

Verwenden von MinIO und Python zum Erstellen skalierbarer und effizienter ereignisgesteuerter Anwendungen

Uso de MinIO y Python para crear aplicaciones basadas en eventos escalables y eficientes

Utiliser MinIO et Python pour créer des applications événementielles évolutives et efficaces

स्केलेबल और कुशल इवेंट-संचालित एप्लिकेशन बनाने के लिए मिनियो और पायथन का उपयोग करना

MinIO と Python を使用してスケーラブルで効率的なイベント駆動型アプリケーションを作成する

MinIO 및 Python을 사용하여 확장 가능하고 효율적인 이벤트 기반 애플리케이션 만들기

Usando MinIO e Python para criar aplicativos orientados a eventos escalonáveis e eficientes

Использование MinIO и Python для создания масштабируемых и эффективных приложений, управляемых событиями

Ölçeklenebilir ve Verimli Olay Odaklı Uygulamalar Oluşturmak için MinIO ve Python'u Kullanmak

Sử dụng MinIO và Python để tạo các ứng dụng hướng sự kiện có khả năng mở rộng và hiệu quả

使用 MinIO 和 Python 创建可扩展且高效的事件驱动应用程序

Managing Large Data Volumes With MinIO, Langchain and OpenAI

MinIO, Langchain এবং OpenAI এর মাধ্যমে বড় ডেটা ভলিউম পরিচালনা করা

Große Datenmengen verwalten mit MinIO, Langchain und OpenAI

Gestión de grandes volúmenes de datos con MinIO, Langchain y OpenAI

Gestion de gros volumes de données avec MinIO, Langchain et OpenAI

मिनियो, लैंगचेन और ओपनएआई के साथ बड़े डेटा वॉल्यूम का प्रबंधन

MinIO、Langchain、OpenAI による大容量データの管理

MinIO, Langchain 및 OpenAI를 사용하여 대용량 데이터 관리

Gerenciando grandes volumes de dados com MinIO, Langchain e OpenAI

Управление большими объемами данных с помощью MinIO, Langchain и OpenAI

MinIO, Langchain ve OpenAI ile Büyük Veri Hacimlerini Yönetmek

Quản lý khối lượng dữ liệu lớn với MiniIO, Langchain và OpenAI

使用 MinIO、Langchain 和 OpenAI 管理大量数据

The modern data landscape demands a new breed of infrastructure – one that seamlessly integrates structured and unstructured data, scales effortlessly, and empowers efficient AI/ML workloads. This is where come in, providing a central hub for all your data needs. However, building and managing an effective data lake can be complex. modern datalakes This blog post dives deep into three powerful tools that can optimize your current approach: , and MinIO. The steps below will walk you through how these services seamlessly combine to create a robust, cloud-native data lake architecture specifically optimized for AI/ML workloads. Apache Iceberg Tabular, What is Tabular? Tabular is a data platform created by the original creators of Apache Iceberg. It is designed to provide an independent, universal storage platform that connects to any compute layer, eliminating data vendor lock-in. This feature is critical to the modern data stack, it allows users to pick best-in-class compute and storage tools without being forced into a particular vendor's aged and or mismatched tool set. In an of MinIO and Iceberg and can be enhanced by Tabular. Tabular can be used to manage and query Iceberg data stored in MinIO, allowing for the storage and management of structured data in a scalable, high-performance, and cloud-native manner. These Kubernetes native, components work together smoothly with very little friction and build on each other’s capabilities to perform at scale. architecture Why S3FileIO instead of Hadoop's file-io? This implementation leverages Iceberg's S3FileIO. S3FileIO is considered better than Hadoop's file-io for several reasons. Some of which we’ve already discussed : elsewhere : Iceberg's S3FileIO is designed to work with cloud-native storage. Optimized for Cloud Storage Iceberg uses an to distribute files across multiple prefixes in a MinIO bucket, which helps to minimize throttling and maximize throughput for S3-related IO operations. Improved Throughput and Minimized Throttling: ObjectStoreLocationProvider Iceberg has been to fully leverage strict consistency by eliminating redundant consistency checks that could impact performance. Strict Consistency: updated Iceberg's S3FileIO implements a algorithm, which uploads data files parts in parallel as soon as each part is ready, reducing local disk usage and increasing upload speed. Progressive Multipart Upload: progressive multipart upload Iceberg allows for for S3 API writes to ensure the integrity of uploaded objects, which can be enabled by setting the appropriate catalog property. Checksum Verification: checksum validations Iceberg supports adding to objects during write and delete operations with the S3 API, which can be useful for cost tracking and management. Custom Tags: custom tags The FileIO interface in Iceberg does not require as strict guarantees as a Hadoop-compatible FileSystem, which allows it to that could otherwise degrade performance. Avoidance of Negative Caching: avoid negative caching In contrast, Hadoop's S3A FileSystem, which was used before S3FileIO, does not offer the same level of optimization for cloud storage. All this to say: Don’t hobble your future-facing data lake infrastructure with the trappings of the past. Prerequisites Before you begin, make sure your system meets the following requirements: Docker Docker Compose If you're starting from scratch, you can install both using the installer for your specific platform. It is often easier than downloading Docker and Docker Compose separately. Verify if Docker is installed by running the following command: Docker Desktop docker-compose --version Getting started To get started, clone or copy the YAML file in Tabular’s . You just need the YAML for this tutorial. Feel free to explore the rest of the repository at a later time. git repository Breaking it Down The provided YAML file is a Docker Compose configuration file. It defines a set of services and their configurations for a multi-container Docker application. In this case, there are two services: Spark-Iceberg and MinIO. Let's break down each section: 1. Spark-Iceberg Service: spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000 Specifies the Docker image to use for the spark-iceberg service. In this case, it uses the tabulario/spark-iceberg:latest image. image: Specifies that the spark-iceberg service depends on the rest and minio services. depends_on: Assigns a specific name (spark-iceberg) to the container. container_name: Sets environment variables for the container, including Spark and AWS credentials. environment: Mounts local directories (./warehouse and ./notebooks) as volumes inside the container. volumes: Maps container ports to host ports for accessing Spark UI and other services. ports: 2. Minio Service: minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"] Specifies the Docker image for the MinIO service. image: Assigns a specific name (MinIO) to the container. container_name: Sets environment variables for configuring MinIO, including root user credentials. environment: Maps container ports to host ports for accessing MinIO UI. ports: Specifies the command to start the MinIO server with specific parameters. command: Another aspect of the MinIO service is , MinIO’s command line tool. mc mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " Specifies that the mc service depends on the MinIO service. depends_on: Specifies the Docker image for the mc service. image: Assigns a specific name (mc) to the container. container_name: Sets environment variables for configuring the MinIO client. environment: Defines the entry point command for the container, including setup steps for the MinIO client. entrypoint: /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " This sequence of commands essentially performs the following tasks: Removes the existing warehouse directory and its contents from the MinIO server. Creates a new bucket named warehouse. Sets the access policy of the warehouse bucket to public. This Docker Compose file orchestrates a multi-container environment with services for Spark, PostgreSQL, MinIO. It sets up dependencies, environment variables, and commands necessary for running the services together. The services work in tandem to create a development environment for data processing using Spark and Iceberg with MinIO as the object storage backend. Starting Up In a terminal window, cd into the tabular-spark-setup directory in the repository and run the following command: docker-compose up Login into MinIO at with the credentials to see that the warehouse bucket has been created. http://127.0.0.1:9001 admin:password Once all the containers are up and running, you can access your Jupyter Notebook server by navigating to http://localhost:8888 Run one of the sample notebooks and return to MinIO at to see your warehouse populated with data. http://127.0.0.1:9001 Building Your Modern Datalake This tutorial on building a modern datalake with Iceberg, Tabular, and MinIO is just the beginning. This powerful trio opens doors to a world of possibilities. With these tools, you can seamlessly integrate and analyze all your data, structured and unstructured, to uncover hidden patterns and drive data-driven decisions that fuel innovation. Leverage the efficiency and flexibility of this architecture in production to expedite your AI/ML initiatives and unlock the true potential of your machine learning models, accelerating your path to groundbreaking discoveries. Reach out to us at or on our channel if you have any questions as you build. hello@min.io Slack

Building Effective Modern Data Architectures with Iceberg, Tabular and MinIO

আইসবার্গ, ট্যাবুলার এবং মিনিও সহ কার্যকরী আধুনিক ডেটা আর্কিটেকচার তৈরি করা

Erstellen effektiver moderner Datenarchitekturen mit Iceberg, Tabular und MinIO

Creación de arquitecturas de datos modernas y eficaces con Iceberg, Tabular y MinIO

Construire des architectures de données modernes et efficaces avec Iceberg, Tabular et MinIO

आइसबर्ग, टेबुलर और मिनियो के साथ प्रभावी आधुनिक डेटा आर्किटेक्चर का निर्माण

Iceberg、Tabular、MinIO を使用した効果的な最新データ アーキテクチャの構築

Iceberg, Tabular 및 MinIO를 사용하여 효과적인 최신 데이터 아키텍처 구축

Construindo arquiteturas de dados modernas e eficazes com Iceberg, Tabular e MinIO

Создание эффективных современных архитектур данных с помощью Iceberg, Tabular и MinIO

Iceberg, Tabular ve MinIO ile Etkili Modern Veri Mimarileri Oluşturma

Xây dựng kiến trúc dữ liệu hiện đại hiệu quả với Iceberg, Tabular và MinIO

使用 Iceberg、Tabular 和 MinIO 构建有效的现代数据架构

deployments come in all shapes and sizes. We support bare metal installs on any version of , containerized installs on any version of (including Red Hat OpenShift), and installs just about anywhere you can deploy a small lightweight single binary. But with flexibility comes the inevitability that edge case issues will require debugging. MinIO Linux Kubernetes In this blog post, we’ll show you how to debug a MinIO install running in Kubernetes and also some of the common issues you might encounter when doing bare metal installation and how to rectify them. Kubernetes Debugger Pod There are a few ways to access the MinIO API running inside a Kubernetes cluster. We can use or set up a listening on to be able to access the API. Both of these methods offer a way to access the service from outside the network, but they do come with one major downside: You can only access the Service that the NodePort or Port Forwarding references on an available port (not the usual configuration for the application). For example, you have to access the MinIO API, usually found on port , via a randomly assigned port. kubectl port-forwarding Service NodePort 9000 3xxxx What if I told you there was a better way – and it's not novel? When debugging applications you want to have full access to the native run-time environment so you can use various tools to troubleshoot and debug the cluster. One way to do that is launching a “busybox” style pod and installing all the required tools needed to debug the application. First launch a into the same namespace as your MinIO install. In order to do this create a yaml file called with the following yaml. Pod debugger-pod.yaml apiVersion: v1

kind: Pod

metadata:

  name: mc

  labels:

    app: mc

spec:

  containers:

  - image: minio/mc:latest

    command:

      - "sleep"

      - "604800"

    imagePullPolicy: IfNotPresent

    name: mc

  restartPolicy: Always The above Pod configuration is pulling the image for MinIO utility. In order to ensure the pod doesn’t just launch and then exit, we’ve added a command. mc sleep Once the yaml is saved apply the configuration to the Kubernetes namespace where the MinIO cluster is running kubectl apply -f debugger-pod.yaml Once the pod is up and running access it via shell $ kubectl exec -i -t -n default mc -c mc -- sh -c "(bash || ash || sh)"

[root@mc /]# Then with you can access the MinIO cluster mc [root@mc /]# mc alias set myminio    --insecure

Added `myminio` successfully. Now that we have a debugger pod up and running, you can perform action on the cluster directly within the same network. For example, if the replication was broken due to an site going offline or a hardware failure, you can resync any pending objects to be replicated using the following command [root@mc /]# mc admin replicate resync start minio1 minio2


[root@mc /]# mc admin replicate resync status minio1 minio2


✔ ✔ ✔

ResyncID: 		 2248d1d1-633f-4d61-b938-d8ea0b9b2d31

Status:   		 Completed

Objects:  		 2225

Versions: 		 2225

FailedObjects:     0

Throughput:   	 5.3 MiB/s

IOPs:     		 124.23 objs/s

Transferred:  	 94 MiB

Elapsed:  		 17.909833202s

CurrObjName:  	 testbucket/web-app/tsconfig.json Another reason you would run the debugger pod is if there are some file system permissions or invalid groups configurations in your pod, you can update them using the debugger pod [root@mc /]# chgrp -R 1000780050 .minio.sys/ The above debugging method can also be used in bare metal environments. For instance you can launch a busybox or bastion node with installed and follow the same instructions as above. mc Debugging Bare Metal Bare metal Linux installs are the most straightforward. In fact it just takes a few commands to get MinIO installed and running with SystemD. For details, please see . Configuring MinIO with SystemD Once in a great while, bare metal installs go awry. Here are some of the (not-so-common) pitfalls that we are asked about  in or . These pitfalls are not hardware or operating specific but can be useful to know in any kind of environment. SUBNET Slack File Permission One of the most common pitfalls is the file permissions of the MinIO binary and the configuration file. If this occurs, when you start MinIO using SystemD you will see and here is the full stack trace Assertion failed for MinIO. # systemctl status minio.service

● minio.service - MinIO

     Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: enabled)

     Active: inactive (dead)

     Assert: start assertion failed at Tue 2023-12-26 18:21:38 PST; 8s ago

             AssertFileIsExecutable=/usr/local/bin/minio was not met

       Docs: https://docs.min.io


Dec 26 18:13:37 minio1 systemd[1]: minio.service: Starting requested but asserts failed.

Dec 26 18:17:50 minio1 systemd[1]: Assertion failed for MinIO.

Dec 26 18:21:38 minio1 systemd[1]: minio.service: Starting requested but asserts failed.

Dec 26 18:21:38 minio1 systemd[1]: Assertion failed for MinIO. This could be caused by a number of reasons, let's go down the list and check for each of them. : The binary, in this example located at needs to have permission for user and group, respectively. MinIO Binary /usr/local/bin/minio root:root # ll /usr/local/bin/minio

total 93804

-rwxr-xr-x 1 root    root    96018432 Nov 15 16:35  minio* : The MinIO service needs to run under a unique Linux user and group for security purposes, never run as a user. By default we use for the user and group names. In the SystemD service config file you should see something like this MinIO Service User and Group root minio-user User=minio-user

Group=minio-user : The directory where MinIO data will be stored needs to be owned by or whichever user you decide to run the MinIO service as above. MinIO Data Dir minio-user:minio-user # ls -l /mnt

total 4

drwxrwxr-x 2 minio-user minio-user 4096 Dec 27 09:58 data : Both the config files should have permissions for user and group like so SystemD and MinIO config root:root # ls -l /etc/default/minio

-rw-r--r-- 1 root root 1330 Dec 27 09:52 /etc/default/minio


# ls -l /etc/systemd/system/minio.service

-rw-r--r-- 1 root root 941 Dec 26 17:13 /etc/systemd/system/minio.service : The entire install process should be run as . You can also try if your user has permissions but the recommendation is to run as root as the install needs to place files in a bunch of places that only the user can access. Your bash prompt should have a and not a like so Run as Root root sudo root # $ vs # $ If none of the above work, the best approach is to remove the app, directories and configs and start a fresh install as a root user. Port conflict Another common issue related to deleted files which still hold on to the process, which causes port conflicts. Even when a service is not running, you may be unable to start a new service on the existing port or the service that is running will misbehave (such as not allowing you to login). # lsof -n | grep (deleted)

COMMAND PID USER FD TYPE DEVICE   SIZE 	NODE NAME

nginx   	13423 root 5u  	REG 253,3   42949672960 	17 (deleted)

minio 	13423 minio 6u    REG 253,3 	0         	      18 (deleted)

minio       13423 minio 7u    REG 253,3 	0         	      19 (deleted) You might see errors such as those below on a MinIO install Login Failed net::ERR_FAILED 500 Internal Server Error 401 Unauthorized The screenshot above shows an internal server error and an unauthorized error. While looking at the surface it looks unclear what has caused this error, we can debug with a little linux knowledge what to look for that could cause this, let's take a gander. There are several ways to debug this issue, first lets check to see if multiple MinIO processes are running on the same node # ps aux | grep -i minio

minio-u+    5048  0.3  1.7 1594008 144384 ?      Ssl  11:03   0:01 /usr/local/bin/minio server --console-address :9001 /mnt/data/disk1/minio


minio-u+    9276  0.3  1.7 1594208 144301 ?      Ssl  11:25   0:01 /usr/local/bin/minio server --console-address :9001 /mnt/data/disk1/minio As we can see above there are 2 MinIO processes running. Start by killing the process that is older or has been running the longest, in this case it seems to be the process ID . 5048 kill -9 5048 Sometimes even after killing the process the service might still not start or might still get hung up because it has reserved a process number but not let it go. This can be caused by files that have been deleted but are still being tracked by the operating system. You can find the deleted files via LSOF lsof -n | grep '(deleted)' Last but not least, if there are no deleted files left over or hung processes and if everything looks absolutely clean, the last resort is to quickly restart the node. This is a no-nonsense method that  shuts down and clears any pending files and processes so you start a fresh install. SUBNET to the rescue Although rare, installation edge cases will always exist. MinIO customers know that they have nothing to worry about because they can quickly message our engineers – who have written the code – via the portal. We've seen almost everything under the sun, so while the issue might look cryptic or mind boggling at first glance, we'll put our years of expertise debugging installations in many varied environments to work and help you in a jiffy. SUBNET If you have any questions on troubleshooting and debugging installs be sure to reach out to us on ! MinIO Slack

Pro Tips for Debugging MinIO: Overcoming Common Issues in Kubernetes and Bare Metal Environments

MinIO ডিবাগ করার জন্য প্রো টিপস: কুবারনেটস এবং বেয়ার মেটাল পরিবেশে সাধারণ সমস্যাগুলি অতিক্রম করা

Profi-Tipps zum Debuggen von MinIO: Überwindung häufiger Probleme in Kubernetes- und Bare-Metal-Umgebungen

Consejos profesionales para depurar MinIO: superar problemas comunes en Kubernetes y entornos bare metal

Conseils de pro pour déboguer MinIO : surmonter les problèmes courants dans les environnements Kubernetes et Bare Metal

मिनियो डिबगिंग के लिए प्रो टिप्स: कुबेरनेट्स और बेयर मेटल वातावरण में आम समस्याओं पर काबू पाना

MinIO デバッグのプロのヒント: Kubernetes とベアメタル環境での一般的な問題の克服

MinIO 디버깅을 위한 전문가 팁: Kubernetes 및 베어메탈 환경의 일반적인 문제 극복

Dicas profissionais para depuração do MinIO: superando problemas comuns em ambientes Kubernetes e Bare Metal

Советы для профессионалов по отладке MinIO: решение распространенных проблем в Kubernetes и средах с голым железом

MinIO'da Hata Ayıklamaya Yönelik Profesyonel İpuçları: Kubernetes ve Bare Metal Ortamlarında Yaygın Sorunların Aşılması

Mẹo chuyên nghiệp để gỡ lỗi MinIO: Khắc phục các sự cố thường gặp trong môi trường Kubernetes và Bare Metal

调试 MinIO 的专业技巧：解决 Kubernetes 和裸机环境中的常见问题

Leveraging MinIO and Apache Tika for Automated Text Extraction and Analysis

স্বয়ংক্রিয় পাঠ্য নিষ্কাশন এবং বিশ্লেষণের জন্য MinIO এবং Apache Tika ব্যবহার করা

Nutzung von MinIO und Apache Tika zur automatischen Textextraktion und -analyse

Aprovechando MinIO y Apache Tika para la extracción y el análisis de texto automatizados

Tirer parti de MinIO et Apache Tika pour l’extraction et l’analyse automatisées de texte

स्वचालित पाठ निष्कर्षण और विश्लेषण के लिए मिनियो और अपाचे टिका का लाभ उठाना

MinIO と Apache Tika を活用した自動テキスト抽出と分析

자동화된 텍스트 추출 및 분석을 위해 MinIO 및 Apache Tika 활용

Aproveitando MinIO e Apache Tika para extração e análise automatizada de texto

Использование MinIO и Apache Tika для автоматического извлечения и анализа текста

Otomatik Metin Çıkarma ve Analizi için MinIO ve Apache Tika'dan Yararlanma

Tận dụng MinIO và Apache Tika để trích xuất và phân tích văn bản tự động

利用 MinIO 和 Apache Tika 进行自动文本提取和分析

Stories, News & Mentions about MINIO

Minio’s Stories on HackerNoon