scale

The Hidden Cost of Hypergrowth: Scale Without Sacrificing Stability

If you’re a marketing department and you want to reach the widest audience possible, text messaging is the best channel to do that with. Most people check their texts more often than email, and text messages have a 90%+ open rate within three minutes of reception. To send SMS and MMS messages at scale you need a cloud communications platform — a.k.a. a communications platform as a service (CPaaS). Peer-to-peer review site G2 lists more than 100 of them, with Plivo ranked number one for customer satisfaction. Throughout this article we’ll use Plivo and Twilio, the CPaaS market share leader, as examples. What’s Your Number? Once you’ve settled on a CPaaS, you have another decision to make: What number type should you use to send your messages? g follows different rules from person-to-person (P2P) texting. Businesses have three options: Commercial application-to-person (A2P) messagin 10-digit long codes (10DLC), toll-free numbers, and short codes. 10DLC Numbers Long codes are normal 10-digit phone numbers — the kind that individuals use every day to engage in P2P conversational messaging. is a service offered by major US carriers to explicitly allow commercial A2P SMS traffic over long code phone numbers. 10DLC is available only in the US. 10DLC You can turn long codes into 10DLC numbers by registering them with a carrier, which you can do through your CPaaS provider. Provisioning a 10DLC number takes less than a week. A vetting process determines the maximum throughput a carrier will allow for a 10DLC number. Throughput is the number of messages they can send in a given time period — messages per second, for example. Toll-Free are used by businesses that want their customers to reach them at no cost. In North America, 800, 888, 877, 866, 855, 844, and 833 denote toll-free numbers. Toll-free numbers Businesses provide toll-free numbers so that incoming calls are free of charge for the calling party, but they can also be enabled for SMS messaging through your carrier or CPaaS, generally in about one business day. Short Codes are five- or six-digit numbers used to send and receive SMS messages. Short codes Unlike long code numbers, shortcodes are not tied to an area code, so they can be used across a country to provide a nationwide brand identity under one easy-to-remember number. Short codes take the longest to acquire — up to 16 weeks. Throughput and Cost Two major differences between number types are their maximum throughput and cost. Throughput Limits Carriers impose varying maximum throughput on different number types. Generally speaking, short codes support the highest throughput and toll-free numbers the lowest. 10DLC numbers sit in the middle. Carrier-imposed throughput for 10DLC numbers depends on vetting scores assigned by an independent organization called The Carrier Registry. Cost Considerations There are three kinds of costs involved in messaging: one-time setup costs, number rental costs, and cost per message. All pricing numbers that we quote below were taken from the two representative vendors’ pricing pages. Short codes incur the highest setup costs: $1,500 for Plivo and $500 for Twilio. 10DLC numbers incur $44 charges for brand and campaign registration and vetting. There are no one-time charges for toll-free numbers. Number rental costs for long codes and toll-free numbers are negligible compared to usage costs for organizations that send a lot of messages. Short code rental costs, however, are significant. Cost per message depends on several factors: whether you’re sending or receiving messages, whether you’re using SMS or MMS, and which carrier you use. Your CPaaS vendor’s pricing page can break it down for you. For comparison purposes, we put together a table showing pricing for sending SMS messages, which for many businesses is the most common use case, and arbitrarily chose T-Mobile as the carrier for which to show surcharges in this table; the “plus” numbers are the T-Mobile carrier surcharges. Run the Numbers All of the different options make it difficult to figure out which number type makes the most economic sense. The best way to decide is to create a spreadsheet that includes the number of SMS and MMS messages you plan to send and receive per month, and factor in your carrier’s surcharges. Regardless of which number type you use, message senders must obtain opt-in consent before texting consumers — and be aware of other considerations. Plivo’s SMS best practices guide lays out those considerations for businesses that want to take advantage of the best channel for business communications — text messaging.

How to Send SMS and MMS Messages at Scale: What is Your Number?

Cómo enviar mensajes SMS y MMS a escala: ¿Cuál es su número?

Comment envoyer des messages SMS et MMS à grande échelle : quel est votre numéro ?

बड़े पैमाने पर एसएमएस और एमएमएस संदेश कैसे भेजें: आपका नंबर क्या है?

SMS および MMS メッセージを大規模に送信する方法: あなたの番号は何ですか?

Como enviar mensagens SMS e MMS em escala: Qual é o seu número?

Cách gửi tin nhắn SMS và MMS ở quy mô lớn: Số của bạn là gì?

如何大规模发送 SMS 和 MMS 消息：您的电话号码是多少？

Building a food delivery fleet of hundreds of robots is like teaching math to third graders. Their ability to learn and interact with their surroundings emotionally is a work in progress, but the sky is the limit. If carried out well, growth and success are very likely. Since starting in 2017, we’ve been scaling robotic technology to create smart cities while demonstrating to the general public and investors the extraordinary potential of the latest technological advances. Not only was and automation important, but setting boundaries and acknowledging the difficulties of the competitive tech space are key to success. company culture Most importantly, some of the most progressive companies deliberately take on employees with track records reflecting both as failing is an essential part of growth. failure and success So if your fintech launch doesn’t go to plan or SaaS doesn’t gain traction, however painful it may be to let go and move on, you’re not alone. Almost of startups pivot by changing their business plan. 60% With plans to grow tenfold next year and expand our team, this is what I’ve learned about scaling operations. Assess Your Company’s Capability to Take on Large Projects There are key performance indicators (KPIs) and quantitative metrics to measure a company´s success and help calculate the mid to long-term predictions to achieve profitability. I call them complex or compounded metrics – they are formed by input variables and governed by cost and time. For example, at Kiwibot, we look at how much it costs to operate our robots per kilometer. Any journey in a pilot program that costs over $2 per km is not considered a progression from regular delivery fees. The project would need far more testing to be able to scale up. What robotic delivery fleets are doing is similar to what Amazon did when eCommerce was less prevalent a few years ago: Exploring uncharted territory and working their way into the spotlight. There were plenty of attempts to set up eCommerce sites and sidewalk robotic delivery services but only those that leverage existing technological capabilities, and apply them in new ways, can succeed. Amazon didn’t invent anything new, they just leveraged the infrastructure and technology that already existed to take shipping speed to the next level. is a vital aspect of your company’s capability to scale too – and is an indicator of this. Employees must show flexibility, persistence, resilience, and be prepared for challenges. Human capital culture Startups are the kings of ambiguity, and the people you want by your side when times get tough are those that have equal levels of passion and persistence, otherwise known as . grit Search for individuals who have and have irreplaceable knowledge from overcoming hardships. Most importantly, fail as fast as possible. survived failure According to Jeff Bezos, the defining attribute of Amazon is its : “Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.” appreciation of failure And as soon as you see something that doesn't work, say goodbye. It doesn’t matter to an idea or innovation, there’s just no point going over old territory. how attached you felt Start Small to Control the Risk of Growing Too Fast Startups must be optimistic, but they often tend to overshoot with overly hopeful objections. Always make sure to underpromise and overdeliver. Take calculated risks as you assess opportunities, start with small operations, and set up shop in one major city, allowing you to sprawl. This is what would be referred to as a “cluster” – where concentrations of interconnected companies and institutions from a specific field are situated – a generic concept used by many years ago that still serves today. Michael Porter As a company scales, it must cluster around points of interest and use cases that deliver traction. It’s vital to partner with organizations in a chosen region and improve relationships with city authorities. Once you have one client signed or a city in the bag, there will be proof of your product gaining customers, and the company will grow swiftly. For example, university campuses or cities like Los Angeles and Miami are significant markets for food delivery, so that’s where Kiwibot has active operations. We have the capacity as a company to expand to Scotland and Wales, for example, but would that be good for business? Probably not. My advice would be: Always choose those “second cities,” know your limits, and set boundaries. Pushbacks: Why It’s Harder to Scale Tech Companies Scaling a tech company is notoriously difficult because of ongoing public antitrust sentiment and the sheer cost. Andy Grove, one of Intel’s founders, once warned that it can be difficult for smaller US tech companies to expand while , and this can mean missed opportunities. competing with giant rivals Delivery robots are part of the green revolution and have a set speed governed by laws in cities across the US. Indeed, technology needs regulation, but with that comes added taxes or licenses which can impact tech companies’ budgets. However, these taxes could help generate revenue for city authorities and cover the costs associated with robotic delivery service operations. Tax implications are what transport technology companies should watch out for – it’s a repetition of the Uber and Lyft stories all over again. Now, consider this example. Buildings have not been designed for robots, but there is a high demand for delivery robots to have self-autonomy indoors. Imagine getting your food order delivered to your desk or a dress from Bloomingdales brought to your apartment before a date. In theory, it’s an incredible invention. Still, it involves APIs to facilitate communication between robots and elevators, the merging of technologies, and a lift company to actually have the appropriate software. It poses a costly venture which can be a blocker to innovation. Malls have the potential to drastically innovate with robots – like Westfield or Halfords – as they search for a leading edge. But will they take that leap? This challenge is similar to what Google struggled with twenty years ago – the development of the cloud, servers, mapping data, as well as physical infrastructure. All the foundations had to be laid before the infrastructure could be built. Over time, robotic technology will become cheaper as demand skyrockets. Think about Yelp’s algorithm, for example, which allows you to find a restaurant that has a view or allows pets – something that is now viewed as indispensable. That opportunity came from mapping technology, and billions were spent on that by Google who now own about 80 percent of the market. Learning to Manage a Lack of Resources US Cities are now busy planning to revive the economy and help local businesses grow post-pandemic. One initiative that was introduced was food ordering apps in restaurants. But when 80% of deliveries are made by cars powered by gas, cities are searching for green opportunities for local businesses. Diverging demand to delivery robots could be the answer, but there aren’t sufficient resources. Covid-19 exposed the lack of tech infrastructure at a local level. From a technology standpoint, there are several major obstacles for robotic fleets to work with local businesses: The cost, legal issues, logistical aspects, and the fact that each has a different app. The solution is to integrate with their apps and offer an alternative tool for delivery. It is vital that technology providers work together to offer free IT support and advice to those local businesses who need it, in an effort to protect jobs and the wider economy. When more and more tasks are becoming digitized or automated, but workers do not possess the skills that the technology requires, the outcome will be greater inequality. Overall, it’s vital for tech companies wanting to scale to do market research, spend time creating demand for their product, assess their qualitative and quantitative metrics, and strive for continuous growth.  The best way to move forward is by searching for innovative solutions to stop a lack of resources or high costs from being a barrier to innovation.

What Tech Companies Should Watch Out for When Scaling Operations 

This will be the first article on a series dedicated to the micro-frontends topic. This one will focus on and an suitable for an enterprise application at scale while keeping the advantages that a Single Page Application ( provides to our end users. analysing designing architecture SPA) By the end, you’ll have a working platform with a scalable micro-frontends approach. Why Micro-Frontends? First of all, you might be wondering why do you even need this, right? Well, there are several answers to this question, and I’m definitely not an advocate of using this pattern/architecture “just because”. I’ll provide a concrete example where this makes sense: You’re in a big enterprise project, it’s a big monolithic application encompassing several different “products” inside one single project/product. The company most likely chose this initial model to start fast and get to market soon. Now, they are established and new engineers come to the team. The team grows a lot, and dedicated teams for each sub-product starts to feel like a good and realistic possibility. Maybe each sub-product can have its own product manager, to extract the most out of each product and to enable the teams to focus on smaller parts of the platform. Up until now, the teams have been working horizontally, meaning that they are working focused on the technology itself (front-end, back-end), instead of focused on the Product. The company decides that vertical teams will be more productive on the long term and also more scalable as new engineers get into the company. These vertical teams will hold all the knowledge inside a specific product and will be composed by the 3 related horizontal layers (front-end, back-end, infrastructure), which will also allow growing each product individually. With this approach, the company can also get more independent teams and deployments, by not being dependent on a single gigantic release to get a new version out to the clients. Given these changes, how will the front-end reflect them? Solutions After a lot of investigation, I concluded that micro-frontends would be the solution to the scenario presented above. But micro-frontends has many faces, many technologies possible, many opinion articles (just like this one) and it’s not easy to know where and what to bet on. I’ll be presenting an that is but that still leverages the and performance that we love to rely on to deliver a great experience to our customers. architecture suitable for scale SPA feel The imaginary business that we are going to use for these articles is a store of clothing items (dresses, shirts, pants, etc). This business is a case of the scenario presented above, working in a horizontal structure on a monolithic application. Now, to increase team independence, they want to create 3 vertical teams for the different sub-products of their platform: , and the . Items Checkout, Blog Architecture To achieve the objective of the clothing company presented in the previous section, we need to design an architecture that enables all the teams to be independent while connecting all these individual products into a coherent platform. The following technologies will allow us to achieve what we need: ReactWebpack 5 with Module Federation Yarn Workspaces with lerna The following architectural diagram represents an overview of the final solution. This is what we’ll achieve at the end of this series of articles on ( ). micro-frontends MFEs All of the components presented in the architecture are , and as such, each is , and but all of them tie together to deliver a consistent experience to the customer. MFEs independently developed tested deployed, App Shell This MFE will be the into the platform. single entry point It will be responsible for rendering the first page to our user and also for invoking the respective micro-frontends as the user navigates through our platform. Using , we will be able to asynchronously load the necessary micro-frontends , keeping the solution memory and performance efficient. client-side navigation at run-time It’s pretty common to find this App Shell approach in micro-frontend projects, but some of the approaches are not exactly like this. They follow more of a where each MFE can be the entry point for the user, and then that MFE will import the “setup” code from the App Shell (e.g. Base Layout with the Header and Footer). Although this approach is perfectly valid, it will most definitely not feel like an SPA to the end user while switching between different entry points for the platform, since it will force a refresh each time the user navigates to a domain from a different MFE. micro-services approach Shared Inevitably, there are components that will be shared between our different MFEs, and for that reason the Shared MFE will be their home. Regarding this MFE, in my opinion, there is not a team specifically responsible for this, instead, there should be a “caretaker” of the MFE that would review any pull request created regarding shared components. This person would be responsible for keeping order in Shared and also for keeping a critical thinking on the following question: “Will this change affect other MFEs?”. Items / Checkout / Blog These MFEs are the actual products with dedicated teams working on them. They’re all in development and deployment. None of them is directly dependent on the other. Naturally, since the platform works as a whole, we need to cover the possibility of some MFE being broken and handle that gracefully. However, this approach will not let any MFE break the whole platform, as long as we handle the remote imports properly. independent Repository Structure We’re going to use a for this solution. monorepo The option to use a is arguable, and for all the cons it may have, the I’m after by using it are: monorepo pros Easily shareable configurations (linters, tasks, setup jobs, code styles)Better / easier overview of the whole systemEasier to make a gradual migration from an existing monolith Architecture (in detail) Let’s see in more detail how these components interact with each other and how exactly we enable the features mentioned in the previous section. The key to making this whole architecture work is . Module Federation Webpack 5 Module Federation aims to solve the sharing of modules in a distributed system, by shipping those critical shared pieces as macro or as micro as you would like. It does this by pulling them out of the build pipeline and out of your apps. The excerpt is from the official a new feature shipped officially on . If you want to learn more you can read the information on the website and also check out their section. There is also a whole with examples using this new technology. Module Federation page , Webpack 5 videos GitHub project Very briefly, Module Federation provides us the chance to code (components, functions, variables) at , while efficiently managing dependencies between different remotes. remotely import javascript run-time Basically, any application can be classified as a (in case it’s the entry point) or a (in case it’s imported by a host). An application can be both a host and a remote at the same time, in which case it’s called . host remote bi-directional Next, let’s analyse some common micro-frontends pitfalls and see how we can get around them using the presented architecture. Common pitfalls Dependency Duplication Using a micro-frontend approach may lead to importing dependencies multiple times. When the components are remotely imported from another MFE, they need certain dependencies to work. Chances are that the MFE importing that piece of code already has some (or all) of those dependencies. Without highly customised caching for dependencies, the imported components will most likely bring along their own dependencies, causing the host to have multiple instances of the same libraries (e.g. having different versions of React in scope may lead to unexpected problems). With the presented architecture we’ll take advantage of to manage these dependencies. When a remote module is imported at run-time, Webpack will actually check if it already has the dependencies required in the . If they are there, then the only thing imported from the remote is the actual code. Module Federation host Dependency Mismatches In micro-frontends, sometimes different versions of the same library are loaded in the same scope. To avoid this, we’ll be using a combination of to manage our monorepo and to manage the dependencies of our different MFEs. lerna yarn workspaces By using yarn workspaces we’ll make sure that the versions of critical libraries are locked for all the micro-frontends, preventing this whole class of problems. This approach reduces independence, but it allows a more robust overall solution. UI / UX In a micro-frontends solution, it often happens that the and diverge between different MFEs. It’s a natural process when different independent teams are each developing their products. At the same time, we need to keep in mind that the different products form the platform as a whole and that platform should provide a consistent experience to the user. UI UX This should happen both in the UI components and in the flows / user experience provided. In the solution presented, by using a Shared MFE, we are keeping these same visual components and reusable behaviours in the same package. This will guarantee that the UI experience is unique and consistent. At the same time, we’re giving this Shared MFE the independence it needs to release frequently. This will help fix small problems in the common components without needing to redeploy any other product MFE (like Items and Blog). Local Development Local development can be taxing in a micro-frontends solution, sometimes needing to run the whole solution in order to test what you are developing in a small part of that platform. The way we’re going to solve this is by providing the chance for all the different MFEs to be run in mode. This will be done by providing 2 separate entry points to our MFE, one for the integration with the whole MFE solution and another for the stand-alone entry point. stand-alone Conclusion The architecture presented in this article was achieved after a lot of investigation on the topic and you can check a lot of material in the section below. References In the next articles, we’ll be deep-diving on how to implement this architecture by following along a github repository with the example project for our imaginary business use case. We’ll also talk about common challenges that occur when using and what can we do to solve them. Module Federation Stay tuned 😉 If you find this article interesting, please share it, because you know — Sharing is caring! If you enjoy working at a large scale in projects with global impact and if you like a real challenge, feel free to reach out to us at xgeeks ! We are growing our team and you might be the next one to join this group of talented people 😉 Check out our social media channels if you want to get a sneak peek of life at xgeeks! See you soon! References https://martinfowler.com/articles/micro-frontends.html https://dev.to/marais/webpack-5-and-module-federation-4j1i https://blog.nrwl.io/monorepos-and-react-microfrontends-a-perfect-match-d49dca64489a https://dev.to/brandonvilla21/micro-frontends-module-federation-with-webpack-5-426 https://www.nicolasdelfino.com/blog/micro-frontends-module-federation-webpack https://indepth.dev/webpack-5-module-federation-a-game-changer-in-javascript-architecture https://blog.bitsrc.io/revolutionizing-micro-frontends-with-webpack-5-module-federation-and-bit-99ff81ceb0 https://medium.com/dazn-tech/orchestrating-micro-frontends-a5d2674cbf33 https://medium.com/hacking-talent/two-years-of-micro-frontends-a-retrospective-522526f76df4 https://single-spa.js.org/docs/recommended-setup/ https://medium.com/swlh/webpack-5-module-federation-a-game-changer-to-javascript-architecture-bcdd30e02669 https://module-federation.github.io/videos Also read behind a paywall at https://medium.com/xgeeks/micro-frontends-at-scale-part-1-a8ab67bfb773

An Introduction to Micro-Frontends in Enterprise Applications (part 1)

I’ve come to talk with you again Photo by on Shubham Dhage Unsplash I’ve written a whole bunch of blogs about running Cloud Foundry on Kubernetes. They all work, to varying degrees of success and have the support of a smattering of communities (read: companies with large budgets + engineers + open source ideals). Sometimes you just want BOSH. That sweet, sweet, familiar BOSH. Enter . bosh-lite What is ? bosh-lite works by spinning up a VM and placing BOSH on it. From there, you can use it to spin , , or anything else that has a BOSH manifest. bosh-lite cf-deployment zookeeper The BOSH director is configured with runtime configs, cloud configs, stemcells, releases, and all the other BOSH-like functionality you’ve come to expect. Installation The tool has been around forever, with a very good tutorial on installing and configuring located at . The rest of this blog assumes that you have followed these instructions up to step #7. bosh-lite https://bosh.io/docs/bosh-lite/ I had an issue with a newer version of VirtualBox (6.1.28), what worked for me is version 6.1.26 which is available . here It ends with an example deployment of which I guess is cool, but I’m guessing most BOSH directors are associated with a Cloud Foundry deployment. Read on for tips that maybe aren’t obvious from the documentation! zookeeper Deploying Cloud Foundry In -land there is documentation for using BBL to deploy and CF to . I point these instructions out in case you have an unlimited IAAS budget, however, I’ll show you how to deploy this to your Mac. cf-deployment bosh-lite GCP or AWS There is also a Readme within . I’ll be using this as the basis for the scripting below. cf-deployment For Operators Deploying CF to local bosh-lite Clone cf-deployment For these examples, I’ve assumed you will clone the CF repo to your home folder: cd ~

git clone https://github.com/cloudfoundry/cf-deployment.git Setup BOSH Environment variables This is spread across a few places in the documentation, I’ve gathered them all together: export CREDHUB_SERVER=https://192.168.56.6:8844
export CREDHUB_CLIENT=credhub-admin
export CREDHUB_SECRET=$(bosh interpolate ~/deployments/vbox/creds.yml --path=/credhub_admin_client_secret)
export CREDHUB_CA_CERT="$(bosh interpolate ~/deployments/vbox/creds.yml --path=/credhub_tls/ca )"$'\n'"$( bosh interpolate ~/deployments/vbox/creds.yml --path=/uaa_ssl/ca)"

export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET="$(bosh int ~/deployments/vbox/creds.yml --path /admin_password)"
export BOSH_CA_CERT="$(bosh interpolate ~/deployments/vbox/creds.yml --path /director_ssl/ca)"
export BOSH_ENVIRONMENT=vbox

bosh alias-env vbox -e 192.168.56.6 --ca-cert <(bosh int ~/deployments/vbox/creds.yml --path /director_ssl/ca) You might consider adding this to your bash/zsh profile. Upload Stemcell, Cloud Config and Runtime Config bosh upload-stemcell --sha1 f399044d2ebe3351f0f1b0b3f97ef11464d283b4 "https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-xenial-go_agent?v=621.125"
bosh update-runtime-config ~/workspace/bosh-deployment/runtime-configs/dns.yml --name dns
bosh update-cloud-config ~/cf-deployment/iaas-support/bosh-lite/cloud-config.yml As time goes by, CF may complain of wanting a newer stemcell version, update the command with the requested version. The version and are listed , note that uses stemcells. bosh upload-stemcell sha1 here bosh-lite warden Go Get a Drink This part will take around 45 minutes to complete, unless you’ve changed the IP address of the BOSH director, the following command will deploy CF. cd ~/cf-deployment
bosh -e 192.168.56.6 -d cf deploy \
  cf-deployment.yml \
  -o operations/bosh-lite.yml \
  -v system_domain=bosh-lite.com Log Into CF This is done in two steps, targeting the API and then logging in, pulling the admin password out of credhub without printing it to the screen: cf api https://api.bosh-lite.com --skip-ssl-validation
cf login -u admin -p $(credhub get -n $(credhub find -n admin | grep cf_admin | cut -d: -f2) | grep value | cut -d: -f2) -o system -s test Tada! Create a space and start pushing apps. (Optional) Setup SSH to BOSH Director Why? I like to be able to SSH to my BOSH Director to poke around. I’m weird. This is not required for the CF deployment, but nice to have: bosh int ~/deployments/vbox/creds.yml --path /jumpbox_ssh/private_key > ~/deployments/vbox/jumpbox.key
chmod 600 ~/deployments/vbox/jumpbox.key
ssh jumpbox@192.168.56.6 -i ~/deployments/vbox/jumpbox.key Beware of Dragons Issue #1 – Reboot Rebooting your Mac causes to misbehave, badly, unless you follow a couple of easy steps. bosh-lite Before shutdown: In Oracle VM VirtualBox Manager select the VM, right click and navigate to > Close Save State After reboot: In Oracle VM VirtualBox Manager select the VM, right click and navigate to > Start Headless Start Even performing these steps I can’t always bring the BOSH Director back online. If you have accidentally rebooted the host you can recreate the BOSH director using the state file and use to recover the broken CF deployment: bosh cck cd ~/deployments/vbox

bosh create-env ~/workspace/bosh-deployment/bosh.yml \
  --state ./state.json \
  -o ~/workspace/bosh-deployment/virtualbox/cpi.yml \
  -o ~/workspace/bosh-deployment/virtualbox/outbound-network.yml \
  -o ~/workspace/bosh-deployment/bosh-lite.yml \
  -o ~/workspace/bosh-deployment/bosh-lite-runc.yml \
  -o ~/workspace/bosh-deployment/uaa.yml \
  -o ~/workspace/bosh-deployment/credhub.yml \
  -o ~/workspace/bosh-deployment/jumpbox-user.yml \
  --vars-store ./creds.yml \
  -v director_name=bosh-lite \
  -v internal_ip=192.168.56.6 \
  -v internal_gw=192.168.56.1 \
  -v internal_cidr=192.168.50.0/24 \
  -v outbound_network_name=NatNetwork --recreate

bosh cck If that doesn’t work, you can completely wipe and recreate the BOSH and CF deployment by: Logging into Oracle VM VirtualBox Manager, select the VM, right click and select > Remove Delete all files Delete the state file at ~/deployments/vbox/state.json Rerun the command then redeploy CF. bosh create-env Issue 2 – Networking issues If you get: Deploying:
  Creating instance 'bosh/0':
    Waiting until instance is ready:
      Post "https://mbus:<redacted>@192.168.56.6:6868/agent": dial tcp 192.168.56.6:6868: connect: connection refused

Exit code 1 You probably skipped the step to add the host routing: sudo route add -net 10.244.0.0/16     192.168.56.6 Final Thought We all deserve nice things. is one of those nice things for folks who enjoy BOSH and want to use it on their own computer. bosh-lite Enjoy! Also published . here

Spin Up a BOSH VM Faster Using BoshLite

When I started working on a start-up nine months ago, this statement didn’t satisfy me or my co-founders: “There is no way to build a Slack bot more reliable than a Slack itself.” — common sense. Our product is an incident management tool called . It’s a Slack app and yes, it should be super reliable. We didn’t leave our SE jobs in Silicon Valley and London to give up when faced with our first technical challenge. Amixr.IO What’s a reliable web service? And what’s not? In terms of reliability, Slack is pretty good. They do a lot of good things. For example they keep users posted about problems through and . status page twitter Unfortunately, Slack has also experienced serious periods of downtime, and users remember that. That’s why users don’t expect an uptime-critical service, such as the our tool, to rely on it. We’ve found a way to achieve that goal. Over the last few months, our service has proved much more reliable than Slack, and we’ve even unveiled an SLA because we’ve built something that customer can rely on. Six simple principles described below helped us here. 1. Draw a red line Any Slack app, as well as any other bot for Facebook or Telegram, uses a web server. Slack is just an interface that allows users to interact with a system. That means we can draw a red line across all of our applications. Parts that are connected to Slack are in danger; other parts depend only on us. Imagine you’re creating a small Slack app. Its purpose is to post a message about new cards on your Trello board. The bot could be a simple web server that will wait for a from Trello in order to post a new message to Slack. It could be as simple as: webhook slackclient SlackClient rest_framework.views APIViewё sc = SlackClient(<bot_access_token>)
        sc.api_call( ,
           channel=<channel_id>,
           text= ,
        ) from import from import : class TrelloForwarderAPIView (APIView) : def get (self, request) "chat.postMessage" "New Cards in Trello detected" Let’s imagine that sc.api_call raises an Exception because the Slack API doesn’t work. I’m sure it won’t last long, so there’s no need to panic. However, you won’t be able to publish a message about this particular Trello card. It will be missed, forever, and your business could be damaged. Now, let’s draw a line between the Trello part and the Slack part and sandbox everything where Slack could cause data loss. To do that, we’ll receive a message from Trello, write it to a queue, and then only remove it from the queue when the message has been successfully posted. celery shared_task slackclient SlackClient rest_framework.views APIView sc = SlackClient(<bot_access_token>)
    sc.api_call( ,
        channel=<channel_id>,
        text= ,
    ) notify_slack.apply_async() from import from import from import @shared_task(autoretry_for=(Exception,), retry_backoff=True, max_retries=None) : def notify_slack () "chat.postMessage" "New Cards in Trello detected" : class TrelloForwarderAPIView (APIView) : def get (self, request) We use Celery, RabitMQ, and Django. Celery’s apply_async method publishes a task to RabbitMQ. In our case, it’s a highly available cluster. Celery workers look out for new tasks and try to execute them until they’re executed, without exception. Now, we can be sure that we won’t lose data if the Slack API causes exceptions on our side. 2. “Broken” doesn’t always mean that “everything is broken” Slack is huge. It has millions of users online and uses multiple servers in regions across the world. If something goes wrong in one place, it doesn’t mean that everything is broken at another location. Even if 99% of servers aren’t able to proceed with a request, there’s another 1% that can. That’s exactly what to Slack in June when its servers weren’t able to process a small fraction of our requests. However, our server didn’t give up, and sent everything successfully. Just nine messages got stuck at one moment, but we’ve sent them successfully since then. happened 3. Be notified early We love it when Slack works, but we need to know when it doesn’t. That’s why we monitor Slack ourselves. We check message delivery, verify API responses, and so forth. We proactively monitor Slack and sometimes detect issues, such as the one detailed in this message regarding the , before anyone else. blocks API If you’re building your service on top of another service, then write scripts and automated tests that proactively monitor the third party. That’s the only way to be the first to learn about a problem. 4. Prepare a Plan B OK, so you’ve built an awesome Slack app that works even when Slack doesn’t. How can you be sure that everything is ready? . It’s just a small example of a fascinating subject called chaos engineering ( ). Simulate a problem, even in production, then check your monitoring and your backup systems. Try simulate Slack downtime yourself https://principlesofchaos.org/ A huge part of this trick is known as “graceful degradation.” It’s the ability to deactivate non-critical parts if they don’t work as expected. In the case of total Slack downtime, our app will switch to a back-up delivery channels, such as e-mail, phone, or another messaging service. It reduces functionality and delivers business value anyway. 5. People should be close We don’t let a critical situation evolve without qualified people watching the process carefully. Even if a case-critical situation is 100% what we expected and recently tested for, there’s still a huge chance of the unexpected occurring. Sometimes, engineers consider how to fix problems temporarily while causing as little damage as possible. For instance, during the issue with the blocks API, we were able to find a workaround pretty quickly. We dealt with the issue before Slack fixed the problem on its side. 6. Talk with clients When discussing outages, we’ve chosen a proactive strategy. We actively inform our users about outages before they notice anything suspicious. If that happens, they won’t consider our service unreliable — they’ll be confident that we know about the problem and are doing our best to fix it. Fun fact: At the very beginning ,we did not have reserve notification channels about a downtime. For the first time, I had to look for users on social networks, write on Facebook, introduce ourselves, and make sure that we didn’t let anyone down. It was really creepy, but, in the end, people reacted positively and became regular users. The six principles described above are universal. They can help to build a reliable service on top of any other third party, and they’ve already helped us with Slack. Each product has different requirements so it's up to you which practices to implement.

How to build a Slack App more reliable then Slack

Pitfalls to Avoid in High-Scale Cloud Applications

​​Easily Scale Real-Time Updates with Firestore: A Maintenance-Free Alternative to WebSockets

The One Tool You Absolutely Need to Efficiently Scale Retrieval-Augmented Generation

Behind Every Question-Answer AI Is a Data Pipeline Built for Scale — Here's How to Build Your Own

Authors:
(1) Zhaoqing Wang, The University of Sydney and AI2Robotics;
(2) Xiaobo Xia, The University of Sydney;
(3) Ziye Chen, The University of Melbourne;
(4) Xiao He, AI2Robotics;
(5) Yandong Guo, AI2Robotics;
(6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence;
(7) Tongliang Liu, The University of Sydney. Table of Links Abstract and 1. Introduction 2. Related works 3. Method and 3.1. Problem definition 3.2. Baseline and 3.3. Uni-OVSeg framework 4. Experiments 4.1. Implementation details 4.2. Main results 4.3. Ablation study 5. Conclusion 6. Broader impacts and References A. Framework details B. Promptable segmentation C. Visualisation 4.3. Ablation study We conduct an extensive ablation study to demonstrate the contribution of each component of our framework. Mask-text alignment. Compared to the straightforward baseline, as shown in Tab. 3, our proposed Uni-OVSeg achieves significant gains of 4.8% PQ and 9.5% mIoU on the COCO dataset, and 11.2% mIoU on the PASCAL Context-59 dataset. This demonstrates our method effectively align objects in images and entities in text descriptions, generalising the CLIP embedding space from the image level to pixel level. By resorting to the refinement of text descriptions, new texts are more correlated with the corresponding images, improving the mIoU from 34.5% to 37.3% on the COCO dataset. Compared to the traditional NLP toolkit (NLTK) [3], ChatGPT-based parser extracts more reliable entities from text descriptions, which achieves obvious improvements of 3.1% and 3.7% mIoU on the COCO and PASCAL Context-59 datasets, respectively. Finally, the proposed multi-scale ensemble strategy that leverages the multi-scale information of objects within the images, stabilise the mask-text matching, which achieves a performance gain of 1.8% PQ on the COCO datasets. Multi-scale ensemble in mask-text matching. The quality of correspondence between masks and entities is an essential part of mask-text matching. To investigate the impact of multi-scale information on this correspondence, as illustrated in Tab. 4, we use masks and semantic classes from the ADE20K and COCO datasets, reporting the Top1 accuracy and forward time per sample. We first resize input images to multiple resolutions and extract visual features via the clip visual encoder. Given ground-truth masks, regional features are pooled from CLIP visual features and projected into the clip embedding space. Each regional embedding is classified by text embeddings. Taking into account the trade-off between performance and latency, we adopt the sizes of 869 × 896 and 1024 × 1024 as default. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Zhaoqing Wang, The University of Sydney and AI2Robotics; (2) Xiaobo Xia, The University of Sydney; (3) Ziye Chen, The University of Melbourne; (4) Xiao He, AI2Robotics; (5) Yandong Guo, AI2Robotics; (6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence; (7) Tongliang Liu, The University of Sydney. Authors: Authors: (1) Zhaoqing Wang, The University of Sydney and AI2Robotics; (2) Xiaobo Xia, The University of Sydney; (3) Ziye Chen, The University of Melbourne; (4) Xiao He, AI2Robotics; (5) Yandong Guo, AI2Robotics; (6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence; (7) Tongliang Liu, The University of Sydney. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related works 2. Related works 3. Method and 3.1. Problem definition 3. Method and 3.1. Problem definition 3.2. Baseline and 3.3. Uni-OVSeg framework 3.2. Baseline and 3.3. Uni-OVSeg framework 4. Experiments 4.1. Implementation details 4.1. Implementation details 4.2. Main results 4.2. Main results 4.3. Ablation study 4.3. Ablation study 5. Conclusion 5. Conclusion 6. Broader impacts and References 6. Broader impacts and References A. Framework details A. Framework details B. Promptable segmentation B. Promptable segmentation C. Visualisation C. Visualisation 4.3. Ablation study We conduct an extensive ablation study to demonstrate the contribution of each component of our framework. Mask-text alignment. Compared to the straightforward baseline, as shown in Tab. 3, our proposed Uni-OVSeg achieves significant gains of 4.8% PQ and 9.5% mIoU on the COCO dataset, and 11.2% mIoU on the PASCAL Context-59 dataset. This demonstrates our method effectively align objects in images and entities in text descriptions, generalising the CLIP embedding space from the image level to pixel level. By resorting to the refinement of text descriptions, new texts are more correlated with the corresponding images, improving the mIoU from 34.5% to 37.3% on the COCO dataset. Compared to the traditional NLP toolkit (NLTK) [3], ChatGPT-based parser extracts more reliable entities from text descriptions, which achieves obvious improvements of 3.1% and 3.7% mIoU on the COCO and PASCAL Context-59 datasets, respectively. Finally, the proposed multi-scale ensemble strategy that leverages the multi-scale information of objects within the images, stabilise the mask-text matching, which achieves a performance gain of 1.8% PQ on the COCO datasets. Mask-text alignment. Multi-scale ensemble in mask-text matching. The quality of correspondence between masks and entities is an essential part of mask-text matching. To investigate the impact of multi-scale information on this correspondence, as illustrated in Tab. 4, we use masks and semantic classes from Multi-scale ensemble in mask-text matching. the ADE20K and COCO datasets, reporting the Top1 accuracy and forward time per sample. We first resize input images to multiple resolutions and extract visual features via the clip visual encoder. Given ground-truth masks, regional features are pooled from CLIP visual features and projected into the clip embedding space. Each regional embedding is classified by text embeddings. Taking into account the trade-off between performance and latency, we adopt the sizes of 869 × 896 and 1024 × 1024 as default. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

The Impact of Mask-Text Alignment and Multi-Scale Ensemble on Uni-OVSeg’s Segmentation Accuracy

To tackle the booming counterfeit market, Ennoventure Inc., an AI-driven brand protection leader, has raised $8.9 million in its Series A funding round, led by Tanglin Venture Partners. The funding, which also includes investment from Fenice Investment Group, will fuel Ennoventure’s ambitious expansion plans in the U.S., UAE, and India. This development comes at a critical time when industries across the globe are facing an unprecedented surge in counterfeit products and intellectual property theft. Ennoventure’s platform is built on a combination of AI and cryptography, allowing brands to embed an invisible digital signature within product packaging. This invisible signature is a game-changer for brands because it enables instant authentication without disrupting the manufacturing process or requiring additional capital investments. The company’s patented technology has already gained traction across sectors such as FMCG, automotive, and pharmaceuticals, where counterfeiting poses severe financial and reputational risks. By harnessing AI for real-time product authentication, Ennoventure’s solution is designed to protect both brands and consumers from fraudulent goods. “Ennoventure is addressing a critical need for effective, scalable brand protection with its innovative, AI-based approach,” said Ahmad Chatila, Chairman of Ennoventure. “This funding round empowers us to expand our footprint and enhance our product offerings to safeguard more brands globally.” The Rising Demand for Anti-Counterfeit Solutions The size of the global trade in counterfeit goods could reach $1.79 trillion by 2030. Industries worldwide are urgently seeking effective anti-counterfeit solutions, with Asia-Pacific emerging as a significant market for product authentication technologies. Ennoventure’s technology is poised to meet this demand with a platform that is both efficient and scalable for large enterprises. Sankalp Gupta, Partner at Tanglin Venture Partners, emphasized Ennoventure’s potential in addressing this market need. “Ennoventure’s AI-driven technology stands out for its seamless integration and scalability. We believe they’re well-positioned to redefine brand protection across multiple sectors.” How Ennoventure’s AI-Based Technology Works Ennoventure’s solution leverages AI and cryptography to create a digital “fingerprint” on product packaging. This invisible marker enables brands to authenticate products in real-time, protecting against counterfeit goods that threaten both revenue and customer trust. The AI system is process-agnostic, meaning it can be applied without changes to existing packaging lines –a critical feature for brands aiming to streamline operations without compromising security. The company’s technology has already proven effective in securing billions of product units, and its platform continues to evolve with advanced AI capabilities. The New Beginning With the Series A investment, Ennoventure plans to enhance its R&D capabilities and drive adoption across high-risk markets. The company’s expansion in the U.S., UAE, and India marks a strategic step forward as it positions itself as a leader in the anti-counterfeit industry. “We’re committed to building innovative solutions that empower brands to tackle counterfeiting,” said Padmakumar Nair, CEO and co-founder of Ennoventure. “With the support of our investors, we’re positioned to drive meaningful change in brand protection.” Key Takeaways Series A Funding Round: Ennoventure raises $8.9 million led by Tanglin Venture Partners, with contributions from Fenice Investment Group.
Innovative Technology: Uses AI and cryptography to create invisible digital markers for product authentication.
Global Expansion: Funding will support growth in key markets, including the U.S., UAE, and India, in industries where counterfeiting is a significant risk.
Industry Impact: Ennoventure’s solution provides real-time authentication, making it a sought-after technology for brands facing counterfeiting threats. As the global fight against counterfeiting intensifies, Ennoventure’s AI-powered platform is poised to play a pivotal role in safeguarding brand integrity and consumer safety. This latest funding round not only accelerates Ennoventure’s mission but also reinforces the company’s standing as a leader in AI-based anti-counterfeit solutions. To tackle the booming counterfeit market, Ennoventure Inc. , an AI-driven brand protection leader, has raised $8.9 million in its Series A funding round, led by Tanglin Venture Partners. The funding, which also includes investment from Fenice Investment Group, will fuel Ennoventure’s ambitious expansion plans in the U.S., UAE, and India. This development comes at a critical time when industries across the globe are facing an unprecedented surge in counterfeit products and intellectual property theft. Ennoventure Inc. Ennoventure’s platform is built on a combination of AI and cryptography, allowing brands to embed an invisible digital signature within product packaging. This invisible signature is a game-changer for brands because it enables instant authentication without disrupting the manufacturing process or requiring additional capital investments. The company’s patented technology has already gained traction across sectors such as FMCG, automotive, and pharmaceuticals, where counterfeiting poses severe financial and reputational risks. By harnessing AI for real-time product authentication, Ennoventure’s solution is designed to protect both brands and consumers from fraudulent goods. “Ennoventure is addressing a critical need for effective, scalable brand protection with its innovative, AI-based approach,” said Ahmad Chatila, Chairman of Ennoventure. “This funding round empowers us to expand our footprint and enhance our product offerings to safeguard more brands globally.” “Ennoventure is addressing a critical need for effective, scalable brand protection with its innovative, AI-based approach,” said Ahmad Chatila, Chairman of Ennoventure. “This funding round empowers us to expand our footprint and enhance our product offerings to safeguard more brands globally.” The Rising Demand for Anti-Counterfeit Solutions The size of the global trade in counterfeit goods could reach $1.79 trillion by 2030. Industries worldwide are urgently seeking effective anti-counterfeit solutions, with Asia-Pacific emerging as a significant market for product authentication technologies. Ennoventure’s technology is poised to meet this demand with a platform that is both efficient and scalable for large enterprises. Sankalp Gupta, Partner at Tanglin Venture Partners, emphasized Ennoventure’s potential in addressing this market need. “Ennoventure’s AI-driven technology stands out for its seamless integration and scalability. We believe they’re well-positioned to redefine brand protection across multiple sectors.” How Ennoventure’s AI-Based Technology Works Ennoventure’s solution leverages AI and cryptography to create a digital “fingerprint” on product packaging. This invisible marker enables brands to authenticate products in real-time, protecting against counterfeit goods that threaten both revenue and customer trust. The AI system is process-agnostic, meaning it can be applied without changes to existing packaging lines –a critical feature for brands aiming to streamline operations without compromising security. The company’s technology has already proven effective in securing billions of product units, and its platform continues to evolve with advanced AI capabilities. The New Beginning With the Series A investment, Ennoventure plans to enhance its R&D capabilities and drive adoption across high-risk markets. The company’s expansion in the U.S., UAE, and India marks a strategic step forward as it positions itself as a leader in the anti-counterfeit industry. “We’re committed to building innovative solutions that empower brands to tackle counterfeiting,” said Padmakumar Nair, CEO and co-founder of Ennoventure. “With the support of our investors, we’re positioned to drive meaningful change in brand protection.” “We’re committed to building innovative solutions that empower brands to tackle counterfeiting,” said Padmakumar Nair, CEO and co-founder of Ennoventure. “With the support of our investors, we’re positioned to drive meaningful change in brand protection.” Key Takeaways Series A Funding Round: Ennoventure raises $8.9 million led by Tanglin Venture Partners, with contributions from Fenice Investment Group. Innovative Technology: Uses AI and cryptography to create invisible digital markers for product authentication. Global Expansion: Funding will support growth in key markets, including the U.S., UAE, and India, in industries where counterfeiting is a significant risk. Industry Impact: Ennoventure’s solution provides real-time authentication, making it a sought-after technology for brands facing counterfeiting threats. Series A Funding Round: Ennoventure raises $8.9 million led by Tanglin Venture Partners, with contributions from Fenice Investment Group. Innovative Technology: Uses AI and cryptography to create invisible digital markers for product authentication. Global Expansion: Funding will support growth in key markets, including the U.S., UAE, and India, in industries where counterfeiting is a significant risk. Industry Impact: Ennoventure’s solution provides real-time authentication, making it a sought-after technology for brands facing counterfeiting threats. As the global fight against counterfeiting intensifies, Ennoventure’s AI-powered platform is poised to play a pivotal role in safeguarding brand integrity and consumer safety. This latest funding round not only accelerates Ennoventure’s mission but also reinforces the company’s standing as a leader in AI-based anti-counterfeit solutions.

Ennoventure Secures $8.9M in Series A to Scale AI-Powered Brand Protection Technology

Abstract: This article presents an advanced framework for processing petabyte-scale datasets using AWS Glue Interactive Sessions, custom libraries, and Amazon Athena. The framework addresses significant challenges in scalability, cost-efficiency, and performance for massive data volumes. Our results demonstrate substantial improvements in processing efficiency and cost reduction, with up to 60% decrease in processing costs and 40% improvement in processing times for compressed datasets. Introduction: The exponential growth of data across industries has created an urgent need for robust processing frameworks capable of handling petabyte-scale datasets. Traditional data processing methods often struggle with scalability, cost-efficiency, and performance when applied to such massive volumes of data. Moreover, the diversity of data formats and compression types introduces additional complexity to data processing pipelines. This article explores the development of a scalable data processing framework utilizing AWS Glue Interactive Sessions and custom libraries, supplemented by Amazon Athena for specific use cases. The framework aims to address the challenges associated with petabyte scale data processing, including efficient handling of various file compression formats, cost effective resource management, maintenance of processing speed of scale, and reliable processing across diverse data formats. Background Previous research has highlighted the limitations of traditional data processing systems in handling large-scale data efficiently. Cloud based solutions, particularly AWS services like Amazon Athena and AWS Glue are two services which provide big data processing and querying. However, there is a need for a cohesive framework that intelligently leverages these services based on data characteristics to optimized performance and cost. Methodology The proposed data processing framework integrates AWS Glue Interactive Sessions as the primary processing engine, with Amazon Athena utilized for specific compression scenarios. The architecture is designed to provide: Interactive Development Capabilities: Allowing data engineers to rapidly iterate and develop data processing skills.


Flexible Processing Options: Selecting the optimal processing engine based on data characteristics such as size and compression type.


Cost optimization: Dynamically allocating compute resources to balance cost and resources.


Scalability: Ensure the framework can handle petabyte scale datasets efficiently. Implementation Details Core Processing Class: The framework centers around a processing class that orchestrates workloads across AWS Glue and Amazon Athena. This class initializes a SparkSession specifically configured for AWS Glue, which is crucial for handling large scale data processing tasks. It also establishes connections to AWS services by creating clients for Athena- used for querying data-and Glue, which handles data processing operations. Intelligent Processing Selection: A standout feature of the framework is its ability to select the most appropriate processing engine based on the characteristics of the dataset, such as compression type and size. The selection criteria include: Gzip-Compressed Data: If the data is compressed with ‘gzip’ and surpasses a certain size threshold, the framework opts for Amazon Athena.
Bzip2-Compressed Data: Datasets compressed with ‘bzips2’ are always processed using Athena.
Snappy and Uncompressed Data: For data compressed with ‘snappy’ or uncompressed data, AWS Glue is typically the chosen engine. Cost Optimized Resource Allocation: To maintain a balance between performance and cost, the framework dynamically adjusts computing resources based on the size of the dataset. Larger datasets are allocated more powerful computing resources, while smaller ones use standard resources. This strategy involves setting the number of involves setting the number of workers and memory requirements in proportion to the data size, ensuring that resources are effectively matched to the workload. Performance Optimization Techniques: The framework incorporates several optimization strategies: Adaptive Query Execution:
i) Enables Spark to optimize queries dynamically based on the data being processed.
ii) Results in improved query performance.


Smart Partition Management:
i) Analysis data partitions to identify optimization opportunities.
ii) Adjusts partitions to enhance processing efficiency when beneficial. Monitoring and Observability: Effective monitoring is essential for maintaining high performance and quickly identifying any issues during processing. The framework integrates with AWS CloudWatch to track metrics related to processing efficiency, such as bytes processed per second. This integration provides valuable insights into the systems’ performance and aids in troubleshooting when necessary. Results: Implementation of the proposed framework has demonstrated significant benefits: Cost reduction: Up to 60% reduction in processing costs through intelligent selection of the processing engine and dynamic resource allocation.
Performance Improvement: Approximately 40% improvement in processing times for compressed datasets, particularly when using Amazon Athena for suitable scenarios.
Scalability: The framework efficiently handles petabyte-scale datasets, providing near real time processing capabilities.
Simplified Development Workflow: Interactive sessions and modular code design simplify the development and maintenance of data processing jobs. Discussion: The framework’s intelligent processing selection ensures that datasets are processed using the most efficient engine based on their characteristics. By leveraging AWS Glue for uncompressed or lightly compressed data and Amazon Athena for heavily compressed data or specific compression types (e.g., bzip2), the framework optimizes both performance and cost. Dynamic resource allocation aligns computing resources with workload demands, preventing over-provisioning and underutilization of resources. The adaptive query execution and smart partition management further enhance performance by optimizing data processing operations at runtime. Future Work: Several enhancements are proposed for future iterations of the framework: Integration with AWS Lake Formation: To provide fine grained access control and data governance capabilities.
Automated Testing Frameworks: Implementation of comprehensive testing to ensure reliability and robustness.
Support for Streaming Data Processing: Extending the framework to handle real-time data streams using technologies such as Apache Kafka and AWS Kinesis.
Complex Orchestration with AWS Step Functions: Integration for orchestrating complex workflows and handling dependencies between data processing tasks. Conclusion: This paper presents a robust and cost-effective solution for processing petabyte scale data by combining AWS Glue Interactive sessions, custom libraries, and Amazon Athena. The framework addresses the challenges of handling diverse data formats and compression types while maintaining optimal performance and cost efficiency. Organizations adopting this framework can expect significant improvements in processing efficiency, cost savings, and scalability. Abstract: Abstract: This article presents an advanced framework for processing petabyte-scale datasets using AWS Glue Interactive Sessions, custom libraries, and Amazon Athena. The framework addresses significant challenges in scalability, cost-efficiency, and performance for massive data volumes. Our results demonstrate substantial improvements in processing efficiency and cost reduction, with up to 60% decrease in processing costs and 40% improvement in processing times for compressed datasets. Introduction: The exponential growth of data across industries has created an urgent need for robust processing frameworks capable of handling petabyte-scale datasets. Traditional data processing methods often struggle with scalability, cost-efficiency, and performance when applied to such massive volumes of data. Moreover, the diversity of data formats and compression types introduces additional complexity to data processing pipelines. This article explores the development of a scalable data processing framework utilizing AWS Glue Interactive Sessions and custom libraries, supplemented by Amazon Athena for specific use cases. The framework aims to address the challenges associated with petabyte scale data processing, including efficient handling of various file compression formats, cost effective resource management, maintenance of processing speed of scale, and reliable processing across diverse data formats. Background Previous research has highlighted the limitations of traditional data processing systems in handling large-scale data efficiently. Cloud based solutions, particularly AWS services like Amazon Athena and AWS Glue are two services which provide big data processing and querying. However, there is a need for a cohesive framework that intelligently leverages these services based on data characteristics to optimized performance and cost. Methodology The proposed data processing framework integrates AWS Glue Interactive Sessions as the primary processing engine, with Amazon Athena utilized for specific compression scenarios. The architecture is designed to provide: Interactive Development Capabilities: Allowing data engineers to rapidly iterate and develop data processing skills. Flexible Processing Options: Selecting the optimal processing engine based on data characteristics such as size and compression type. Cost optimization: Dynamically allocating compute resources to balance cost and resources. Scalability: Ensure the framework can handle petabyte scale datasets efficiently. Interactive Development Capabilities: Allowing data engineers to rapidly iterate and develop data processing skills. Interactive Development Capabilities: Allowing data engineers to rapidly iterate and develop data processing skills. Flexible Processing Options: Selecting the optimal processing engine based on data characteristics such as size and compression type. Flexible Processing Options: Selecting the optimal processing engine based on data characteristics such as size and compression type. Cost optimization: Dynamically allocating compute resources to balance cost and resources. Cost optimization: Dynamically allocating compute resources to balance cost and resources. Scalability: Ensure the framework can handle petabyte scale datasets efficiently. Scalability: Ensure the framework can handle petabyte scale datasets efficiently. Implementation Details Core Processing Class: The framework centers around a processing class that orchestrates workloads across AWS Glue and Amazon Athena. This class initializes a SparkSession specifically configured for AWS Glue, which is crucial for handling large scale data processing tasks. It also establishes connections to AWS services by creating clients for Athena- used for querying data-and Glue, which handles data processing operations. Core Processing Class: Intelligent Processing Selection : A standout feature of the framework is its ability to select the most appropriate processing engine based on the characteristics of the dataset, such as compression type and size. The selection criteria include: Intelligent Processing Selection Gzip-Compressed Data: If the data is compressed with ‘gzip’ and surpasses a certain size threshold, the framework opts for Amazon Athena. Bzip2-Compressed Data: Datasets compressed with ‘bzips2’ are always processed using Athena. Snappy and Uncompressed Data: For data compressed with ‘snappy’ or uncompressed data, AWS Glue is typically the chosen engine. Gzip-Compressed Data: If the data is compressed with ‘gzip’ and surpasses a certain size threshold, the framework opts for Amazon Athena. Bzip2-Compressed Data: Datasets compressed with ‘bzips2’ are always processed using Athena. Snappy and Uncompressed Data: For data compressed with ‘snappy’ or uncompressed data, AWS Glue is typically the chosen engine. Cost Optimized Resource Allocation: Cost Optimized Resource Allocation: To maintain a balance between performance and cost, the framework dynamically adjusts computing resources based on the size of the dataset. Larger datasets are allocated more powerful computing resources, while smaller ones use standard resources. This strategy involves setting the number of involves setting the number of workers and memory requirements in proportion to the data size, ensuring that resources are effectively matched to the workload. Performance Optimization Techniques : Performance Optimization Techniques The framework incorporates several optimization strategies: Adaptive Query Execution:
i) Enables Spark to optimize queries dynamically based on the data being processed.
ii) Results in improved query performance. Smart Partition Management:
i) Analysis data partitions to identify optimization opportunities.
ii) Adjusts partitions to enhance processing efficiency when beneficial. Adaptive Query Execution: i) Enables Spark to optimize queries dynamically based on the data being processed. ii) Results in improved query performance. Adaptive Query Execution: i) Enables Spark to optimize queries dynamically based on the data being processed. ii) Results in improved query performance. Smart Partition Management: i) Analysis data partitions to identify optimization opportunities. ii) Adjusts partitions to enhance processing efficiency when beneficial. Smart Partition Management: i) Analysis data partitions to identify optimization opportunities. ii) Adjusts partitions to enhance processing efficiency when beneficial. Monitoring and Observability: Monitoring and Observability: Effective monitoring is essential for maintaining high performance and quickly identifying any issues during processing. The framework integrates with AWS CloudWatch to track metrics related to processing efficiency, such as bytes processed per second. This integration provides valuable insights into the systems’ performance and aids in troubleshooting when necessary. Results: Results: Implementation of the proposed framework has demonstrated significant benefits: Cost reduction: Up to 60% reduction in processing costs through intelligent selection of the processing engine and dynamic resource allocation. Performance Improvement: Approximately 40% improvement in processing times for compressed datasets, particularly when using Amazon Athena for suitable scenarios. Scalability: The framework efficiently handles petabyte-scale datasets, providing near real time processing capabilities. Simplified Development Workflow: Interactive sessions and modular code design simplify the development and maintenance of data processing jobs. Cost reduction: Up to 60% reduction in processing costs through intelligent selection of the processing engine and dynamic resource allocation. Performance Improvement: Approximately 40% improvement in processing times for compressed datasets, particularly when using Amazon Athena for suitable scenarios. Scalability: The framework efficiently handles petabyte-scale datasets, providing near real time processing capabilities. Simplified Development Workflow: Interactive sessions and modular code design simplify the development and maintenance of data processing jobs. Discussion: The framework’s intelligent processing selection ensures that datasets are processed using the most efficient engine based on their characteristics. By leveraging AWS Glue for uncompressed or lightly compressed data and Amazon Athena for heavily compressed data or specific compression types (e.g., bzip2), the framework optimizes both performance and cost. Dynamic resource allocation aligns computing resources with workload demands, preventing over-provisioning and underutilization of resources. The adaptive query execution and smart partition management further enhance performance by optimizing data processing operations at runtime. Future Work: Future Work: Several enhancements are proposed for future iterations of the framework: Integration with AWS Lake Formation: To provide fine grained access control and data governance capabilities. Automated Testing Frameworks: Implementation of comprehensive testing to ensure reliability and robustness. Support for Streaming Data Processing: Extending the framework to handle real-time data streams using technologies such as Apache Kafka and AWS Kinesis. Complex Orchestration with AWS Step Functions: Integration for orchestrating complex workflows and handling dependencies between data processing tasks. Integration with AWS Lake Formation: To provide fine grained access control and data governance capabilities. Automated Testing Frameworks: Implementation of comprehensive testing to ensure reliability and robustness. Support for Streaming Data Processing: Extending the framework to handle real-time data streams using technologies such as Apache Kafka and AWS Kinesis. Complex Orchestration with AWS Step Functions: Integration for orchestrating complex workflows and handling dependencies between data processing tasks. Conclusion: Conclusion: This paper presents a robust and cost-effective solution for processing petabyte scale data by combining AWS Glue Interactive sessions, custom libraries, and Amazon Athena. The framework addresses the challenges of handling diverse data formats and compression types while maintaining optimal performance and cost efficiency. Organizations adopting this framework can expect significant improvements in processing efficiency, cost savings, and scalability.

Revolutionizing Petabyte-Scale Data Processing on AWS: Advanced Framework Unveiled

Architecting Scalable Databases for Large-Scale Systems

Infrastructure as code (IaC) has become increasingly challenging as organization and usage scale. Inconsistent configurations, security gaps, and increasing costs remain to be hard to manage without the right level of abstraction, but what's the right level of abstraction? From DevOps to Infrastructure as Code DevOps started with engineers clicking around the AWS console and writing scripts to automate tasks like creating EC2 instances and setting up VPCs, but as usage grew, Infrastructure as Code (IaC) tools such as Terraform and CloudFormation emerged. Instead of writing shell scripts, infrastructure engineers can write declarative code to define how the cloud resources should be deployed. This is a great step up in terms of abstraction from the one-off automation scripts. On one hand, the declarative nature of the IaC tools makes it easy enough for most developers to write (you can think of it as configuration instead of code); on the other, it's not abstracted enough to give platform engineers the ability to write modularized code for developers to create infrastructure across the organization. Take resource tagging for example, for both cost tracking and governance purposes, it's a very common use case to want to tag cloud resources by application, team, business unit, cost center, etc. With Terraform, application engineers need to manually specify these tags for each resource. Even if platform engineering teams create the relevant terraform "tag" module to enforce common fields, application engineers still need to provide the tag values every time. The declarative nature of IaC makes it hard for platform engineering teams to automatically populate these tag values from a central registry of applications and teams. This becomes especially problematic as the number of resources and size of the organization grows. This led to the next question: how can we create a higher level of abstraction on top of IaC that addresses these enterprise-scale challenges? Creating Platform Abstraction for Scale Creating the right level of platform abstraction enables large organizations to manage infrastructure at scale more effectively and increase developer velocity. Platform engineering teams often use a mix of strategies, depending on use cases, resources, and expertise within the organization. 1. Terraform Modules or Templates Creating reusable terraform modules or templates is a common and easy way to create higher-level abstractions. While flexible, it's a lower-level abstraction, and making organization-wide changes can be time-consuming as they require review and testing by individual application teams. For organizations lacking resources and expertise, vendors like Resourcely create templated terraform code with enforced practice and security. 2. Imperative IaC With declarative IaC, integrating external processes into infrastructure deployment often requires workarounds. Platform teams frequently combine declarative IaC with one-off scripts or API calls in their CI/CD pipelines. Tools like AWS CDK (Cloud Development Kit), CDK for Terraform, and Pulumi allow teams to generate IaC using imperative programming languages, enabling platform engineers to create infrastructure code alongside dynamic processes. While this provides greater flexibility and abstraction, it poses a higher adoption barrier for application teams less familiar with these "CDK" frameworks. 3. Infrastructure and Platform as a Service At the highest abstraction level, platforms like Databricks, Confluent, and Elastic wrap entire infrastructure deployments, exposing only relevant configurations to developers. This allows the provisioning of complex, distributed systems with minimal effort. Kubernetes similarly abstracts compute resource management, enabling developers to create deployments using high-level configurations. While the high-level abstraction allows platform teams to standardize provisioning, governance, and security at scale for supported use cases, it comes at the cost of reduced flexibility and potential vendor lock-in. New or custom use cases can require much longer implementation cycles. Platform Abstraction: Balancing Flexibility and Control Defining the "right" level of abstraction is never easy. Too little abstraction can lead to frequent manual updates and migrations, inconsistent standards, and security and governance implementations. However, too much abstraction can force teams to adapt their use cases to fit the platform's limitations, slowing down developer velocity. The ideal abstraction varies case by case. For real-time fraud detection or experimental features, teams often need flexibility. Templated terraform might provide sufficient guardrails while enabling developers to easily create custom deployments. In contrast, common use cases like data streaming involving Kafka, connectors, and multi-platform integrations typically require more abstraction. Platform teams often create end-to-end deployments, exposing only necessary configs. However, data pipelines can be nuanced, and platform teams often expose lower-level components, allowing application teams to create custom implementations easily and bring back these features to the platform as they mature. For less complicated scenarios like application deployment, most teams only need to specify a Docker image, CPU, and memory requirements. Platform teams can wrap up the CI/CD process without exposing lower-level components. Knowing the different implementation strategies and levels of abstraction for various scenarios, how can we build platforms that scale? Future of Platform Engineering Data and AI Platform as a Service Kubernetes has simplified infrastructure and application deployment, but data platforms have become increasingly complex. The proliferation of specialized data tools, compounded by AI and ML solutions, has made standardization, integration, security, and governance much harder to manage than infrastructure or application deployment. Vendors like Confluent, Snowflake, Databricks, and dbt are improving the developer experience with more automation and integrations, but they often operate independently. This fragmentation makes standardizing multi-directional integrations across identity and access management, data governance, security, and cost control even more challenging. Developing a standardized, secure, and scalable solution for multi-platform environments is now a fast-evolving area for platform engineering teams. Higher Abstraction for Standardization, Security, and Governance While application developers can use low-level terraform to create infrastructure on their own, this approach doesn’t scale at the enterprise level. Standardization, security, governance, and cost efficiency aren't just “nice to have” but "required" for large organizations. Many companies undertake multi-year, large-scale migrations due to a lack of abstraction where it’s needed, meanwhile slowing down developer velocity in areas where abstraction isn’t necessary. Developers are looking for a combination of flexibility and simplicity depending on the features they are trying to build. Platform teams must continue to simplify the developer experience while strategically exposing lower-level components where necessary. By exposing the right components, they provide developers with greater flexibility and reduce the time needed to integrate new features back into the platform. Imperative IaC for Platform Teams To enable platform teams to build for higher abstraction and flexibility, offload the complexity from CI/CD pipelines, which are designed for orchestration rather than abstraction. Currently, platform teams typically use either configuration-driven systems or declarative IaC, but both approaches have limitations when it comes to creating sophisticated, dynamic deployments and abstraction. The gap in tooling continues to be a challenge, especially for data and AI platform engineering. While there's a proliferation of data product vendors, few focus on simplifying multi-platform deployments. Instead, there's a surge of governance, lineage, and cataloging products to compensate for the lack of integrations. These solutions aim to solve the complexity that arises from using multiple platforms without addressing the core issue of creating streamlined and flexible deployments across data and AI products and platforms. While infrastructure development tools and frameworks are a highly competitive space, there's still room for tools that can provide the advantage of imperative programming for complex logic, while preserving the audit trails, and version control of declarative IaC for governance and compliance. Summary The value of platform engineering is often less tangible than customer-facing product features. While platform improvements may not directly contribute to top-line growth, they are one of the most effective ways to manage cloud costs, which are often one of the biggest drivers of bottom-line costs. Choosing the right toolings and abstraction levels can not only improve developer experience and productivity but also strengthen security, governance, and compliance across the organization. Infrastructure as code (IaC) has become increasingly challenging as organization and usage scale. Inconsistent configurations, security gaps, and increasing costs remain to be hard to manage without the right level of abstraction, but what's the right level of abstraction? Infrastructure as code (IaC) From DevOps to Infrastructure as Code DevOps started with engineers clicking around the AWS console and writing scripts to automate tasks like creating EC2 instances and setting up VPCs, but as usage grew, Infrastructure as Code (IaC) tools such as Terraform and CloudFormation emerged. Instead of writing shell scripts, infrastructure engineers can write declarative code to define how the cloud resources should be deployed. This is a great step up in terms of abstraction from the one-off automation scripts. Terraform CloudFormation On one hand, the declarative nature of the IaC tools makes it easy enough for most developers to write (you can think of it as configuration instead of code); on the other, it's not abstracted enough to give platform engineers the ability to write modularized code for developers to create infrastructure across the organization. Take resource tagging for example, for both cost tracking and governance purposes, it's a very common use case to want to tag cloud resources by application, team, business unit, cost center, etc. With Terraform, application engineers need to manually specify these tags for each resource. Even if platform engineering teams create the relevant terraform "tag" module to enforce common fields, application engineers still need to provide the tag values every time. The declarative nature of IaC makes it hard for platform engineering teams to automatically populate these tag values from a central registry of applications and teams. This becomes especially problematic as the number of resources and size of the organization grows. This led to the next question: how can we create a higher level of abstraction on top of IaC that addresses these enterprise-scale challenges? Creating Platform Abstraction for Scale Creating the right level of platform abstraction enables large organizations to manage infrastructure at scale more effectively and increase developer velocity. Platform engineering teams often use a mix of strategies, depending on use cases, resources, and expertise within the organization. 1. Terraform Modules or Templates Creating reusable terraform modules or templates is a common and easy way to create higher-level abstractions. While flexible, it's a lower-level abstraction, and making organization-wide changes can be time-consuming as they require review and testing by individual application teams. For organizations lacking resources and expertise, vendors like Resourcely create templated terraform code with enforced practice and security. Resourcely 2. Imperative IaC With declarative IaC, integrating external processes into infrastructure deployment often requires workarounds. Platform teams frequently combine declarative IaC with one-off scripts or API calls in their CI/CD pipelines. Tools like AWS CDK (Cloud Development Kit) , CDK for Terraform , and Pulumi allow teams to generate IaC using imperative programming languages, enabling platform engineers to create infrastructure code alongside dynamic processes. AWS CDK (Cloud Development Kit) CDK for Terraform Pulumi While this provides greater flexibility and abstraction, it poses a higher adoption barrier for application teams less familiar with these "CDK" frameworks. 3. Infrastructure and Platform as a Service At the highest abstraction level, platforms like Databricks , Confluent , and Elastic wrap entire infrastructure deployments, exposing only relevant configurations to developers. This allows the provisioning of complex, distributed systems with minimal effort. Kubernetes similarly abstracts compute resource management, enabling developers to create deployments using high-level configurations. Databricks Confluent Elastic Kubernetes While the high-level abstraction allows platform teams to standardize provisioning, governance, and security at scale for supported use cases, it comes at the cost of reduced flexibility and potential vendor lock-in. New or custom use cases can require much longer implementation cycles. Platform Abstraction: Balancing Flexibility and Control Defining the "right" level of abstraction is never easy. Too little abstraction can lead to frequent manual updates and migrations, inconsistent standards, and security and governance implementations. However, too much abstraction can force teams to adapt their use cases to fit the platform's limitations, slowing down developer velocity. The ideal abstraction varies case by case. For real-time fraud detection or experimental features, teams often need flexibility. Templated terraform might provide sufficient guardrails while enabling developers to easily create custom deployments. In contrast, common use cases like data streaming involving Kafka, connectors, and multi-platform integrations typically require more abstraction. Platform teams often create end-to-end deployments, exposing only necessary configs. However, data pipelines can be nuanced, and platform teams often expose lower-level components, allowing application teams to create custom implementations easily and bring back these features to the platform as they mature. For less complicated scenarios like application deployment, most teams only need to specify a Docker image, CPU, and memory requirements. Platform teams can wrap up the CI/CD process without exposing lower-level components. Knowing the different implementation strategies and levels of abstraction for various scenarios, how can we build platforms that scale? Future of Platform Engineering Data and AI Platform as a Service Kubernetes has simplified infrastructure and application deployment, but data platforms have become increasingly complex. The proliferation of specialized data tools, compounded by AI and ML solutions, has made standardization, integration, security, and governance much harder to manage than infrastructure or application deployment. Vendors like Confluent , Snowflake , Databricks , and dbt are improving the developer experience with more automation and integrations, but they often operate independently. This fragmentation makes standardizing multi-directional integrations across identity and access management, data governance, security, and cost control even more challenging. Developing a standardized, secure, and scalable solution for multi-platform environments is now a fast-evolving area for platform engineering teams. Confluent Snowflake Databricks dbt Higher Abstraction for Standardization, Security, and Governance While application developers can use low-level terraform to create infrastructure on their own, this approach doesn’t scale at the enterprise level. Standardization, security, governance, and cost efficiency aren't just “nice to have” but "required" for large organizations. Many companies undertake multi-year, large-scale migrations due to a lack of abstraction where it’s needed, meanwhile slowing down developer velocity in areas where abstraction isn’t necessary. Developers are looking for a combination of flexibility and simplicity depending on the features they are trying to build. Platform teams must continue to simplify the developer experience while strategically exposing lower-level components where necessary. By exposing the right components, they provide developers with greater flexibility and reduce the time needed to integrate new features back into the platform. Imperative IaC for Platform Teams To enable platform teams to build for higher abstraction and flexibility, offload the complexity from CI/CD pipelines, which are designed for orchestration rather than abstraction. Currently, platform teams typically use either configuration-driven systems or declarative IaC, but both approaches have limitations when it comes to creating sophisticated, dynamic deployments and abstraction. The gap in tooling continues to be a challenge, especially for data and AI platform engineering. While there's a proliferation of data product vendors, few focus on simplifying multi-platform deployments. Instead, there's a surge of governance, lineage, and cataloging products to compensate for the lack of integrations. These solutions aim to solve the complexity that arises from using multiple platforms without addressing the core issue of creating streamlined and flexible deployments across data and AI products and platforms. While infrastructure development tools and frameworks are a highly competitive space, there's still room for tools that can provide the advantage of imperative programming for complex logic, while preserving the audit trails, and version control of declarative IaC for governance and compliance. Summary The value of platform engineering is often less tangible than customer-facing product features. While platform improvements may not directly contribute to top-line growth, they are one of the most effective ways to manage cloud costs, which are often one of the biggest drivers of bottom-line costs. Choosing the right toolings and abstraction levels can not only improve developer experience and productivity but also strengthen security, governance, and compliance across the organization.

Platform Engineering Abstraction: How to Scale IaC for Enterprise

Good news! HackerNoon is joining Cloudflare’s Workers Launchpad Cohort #4. Using the content management system we built to power HackerNoon.com, we’ve been quietly building out an open and technical blogging network. Some examples from the blogging network include Memeology.tech, TextModels.tech, Escholar.tech, LearnRepo.com, Homology.tech, Dataology.tech, Browserology.tech, HistoricalEmails.com, CoinWikis.com,  Blog.Slogging.com, Noonion.tech, HackerNoon.tech,  Blog.DavidSmooke.net,  PublicDomain.tech,  LegalPDF.tech,  OpenDatasets.tech, Abstraction.tech, FewShot.tech, Diction.tech, Computational.tech, Editorialist.tech, Reinforcement.tech, and MediaBias.tech. We are dogfooding our publishing software. Via our EditingProtocol.com, these technology learning hubs publish user-generated and open source content. We’re also reusing many publishing software and systems we’ve built to make HackerNoon what it is today, such as: story pages that load fast worldwide, curations of quality niche stories, AI editor assistance, blog post translation into (almost) any language, audio file creation, email delivery, blockchain backups, and more. Cloudflare’s Workers Launchpad provides up to $250k in Cloudflare credits, business office hours and training, access to technical leaders at Cloudflare, VC introductions, and a Cloudflare.tv demo day (tentatively, Feb 13 mark your calendar). The other companies in the cohort look pretty cool (full list on the Cloudflare Blog). We are joining this program to 1) efficiently scale our technical blogging network’s performance and cost, and 2) better use AI with our technology library to adeptly teach any niche technology. I’m excited to work with an established partner like Cloudflare to make the internet a better place. Good news! HackerNoon is joining Cloudflare’s Workers Launchpad Cohort #4 . Using the content management system we built to power HackerNoon.com , we’ve been quietly building out an open and technical blogging network. Cloudflare’s Workers Launchpad Cohort #4 HackerNoon.com Some examples from the blogging network include Memeology.tech , TextModels.tech , Escholar.tech , LearnRepo.com , Homology.tech , Dataology.tech , Browserology.tech , HistoricalEmails.com , CoinWikis.com , Blog.Slogging.com , Noonion.tech , HackerNoon.tech, Blog.DavidSmooke.net , PublicDomain.tech , LegalPDF.tech , OpenDatasets.tech , Abstraction.tech , FewShot.tech , Diction.tech , Computational.tech , Editorialist.tech , Reinforcement.tech, and MediaBias.tech . Memeology.tech Memeology.tech TextModels.tech TextModels.tech Escholar.tech Escholar.tech LearnRepo.com LearnRepo.com Homology.tech Dataology.tech Browserology.tech HistoricalEmails.com HistoricalEmails.com CoinWikis.com CoinWikis.com Blog.Slogging.com Blog.Slogging.com Noonion.tech Noonion.tech HackerNoon.tech , HackerNoon.tech Blog.DavidSmooke.net Blog.DavidSmooke.net PublicDomain.tech PublicDomain.tech LegalPDF.tech LegalPDF.tech OpenDatasets.tech OpenDatasets.tech Abstraction.tech FewShot.tech Diction.tech Computational.tech Editorialist.tech Reinforcement.tech, MediaBias.tech We are dogfooding our publishing software. Via our EditingProtocol.com, these technology learning hubs publish user-generated and open source content. We’re also reusing many publishing software and systems we’ve built to make HackerNoon what it is today, such as: story pages that load fast worldwide, curations of quality niche stories, AI editor assistance, blog post translation into (almost) any language , audio file creation , email delivery , blockchain backups , and more. EditingProtocol.com , EditingProtocol.com blog post translation into (almost) any language audio file creation email delivery blockchain backups Cloudflare’s Workers Launchpad provides up to $250k in Cloudflare credits, business office hours and training, access to technical leaders at Cloudflare, VC introductions , and a Cloudflare.tv demo day (tentatively, Feb 13 mark your calendar). The other companies in the cohort look pretty cool ( full list on the Cloudflare Blog ). Cloudflare’s Workers Launchpad VC introductions Cloudflare.tv full list on the Cloudflare Blog We are joining this program to 1) efficiently scale our technical blogging network’s performance and cost, and 2) better use AI with our technology library to adeptly teach any niche technology. I’m excited to work with an established partner like Cloudflare to make the internet a better place.

HackerNoon Joins Cloudflare's Workers Launchpad To Scale Its Tech Blogging Network

হ্যাকারনুন তার টেক ব্লগিং নেটওয়ার্ককে স্কেল করতে ক্লাউডফ্লেয়ারের কর্মীদের লঞ্চপ্যাডে যোগ দেয়

Το HackerNoon συμμετέχει στο Workers Launchpad του Cloudflare για να κλιμακώσει το τεχνολογικό του δίκτυο blogging

HackerNoon se suma a la plataforma de lanzamiento Workers de Cloudflare para ampliar su red de blogs tecnológicos

HackerNoon מצטרף ל-Launchpad Workers של Cloudflare כדי להרחיב את רשת הבלוגים הטכנית שלה

हैकरनून अपने टेक ब्लॉगिंग नेटवर्क को बढ़ाने के लिए क्लाउडफ्लेयर के वर्कर्स लॉन्चपैड से जुड़ा

HackerNoon が Cloudflare の Workers Launchpad に参加し、技術ブログ ネットワークを拡大

HackerNoon се придружува на Работнички Launchpad на Cloudflare за да ја зголеми својата технолошка мрежа за блогирање

HackerNoon se alătură Launchpad-ului lucrătorilor Cloudflare pentru a-și extinde rețeaua de bloguri tehnologice

HackerNoon එහි තාක්ෂණික බ්ලොග්කරණ ජාලය පරිමාණය කිරීම සඳහා Cloudflare හි කම්කරු දියත් කිරීමේ පෑඩ් සමඟ සම්බන්ධ වේ

HackerNoon เข้าร่วม Workers Launchpad ของ Cloudflare เพื่อขยายเครือข่ายบล็อกเทคโนโลยี

ሃከርኑን ንናይ ክላውድፍሌር ሰራሕተኛታት ላንችፓድ ተጸንቢሩ ንናይ ቴክ ብሎጊንግ ኔትወርኩ ንምዕባይ

“HackerNoon”, “Tehniki blog” ulgamyny giňeltmek üçin “Cloudflare” -iň işçileriniň “Launchpad” -yna goşulýar

AUSTIN, TX, September 18th, 2024/CyberNewsWire/--Research indicates that an infostealer malware infection is often a precursor to a ransomware attack SpyCloud, the leader in Cybercrime Analytics, today announced new cybersecurity research highlighting the growing and alarming threat of infostealers – a type of malware designed to exfiltrate digital identity data, login credentials, and session cookies from infected devices. SpyCloud’s latest findings reveal the staggering scale of identity exposure caused by infostealers, the influence this type of malware has had on the surge in ransomware incidents, and the profound implications for businesses worldwide. Massive Scale of Identity Exposure Creates New Risks According to SpyCloud, 61% of all data breaches in the past year were malware-related, with infostealers responsible for the theft of 343.78 million credentials. These stolen credentials are then sold in criminal communities for use in further attacks. The research also found that one in five individuals has been a victim of an infostealer infection. Each infection, on average, exposes 10-25 third-party business application credentials, creating fertile ground for further access and exploitation, particularly by ransomware operators. “Our latest findings reveal a critical shift in the cybersecurity landscape,” said Damon Fleury, chief product officer at SpyCloud. “Infostealers have become the go-to tool for cybercriminals, with their ability to exfiltrate valuable data in a matter of seconds, creating a runway for cyberattacks like ransomware off the vast amounts of stolen access to SSO, VPN, admin panels, and other critical applications.” Infostealers: The Precursor To Ransomware Attacks The link between infostealers and ransomware is becoming increasingly evident. Through deep analysis of recaptured infostealer logs, SpyCloud discovered a worrying trend: companies with employees and contractors who are infected with infostealer malware are significantly more likely to experience a ransomware attack. In fact, nearly one-third of companies that suffered a ransomware attack last year had previously experienced an infostealer infection. According to the report, this is based on publicly known incidents and confirmed ransomware events. The true exposure is potentially even higher as not all ransomware incidents are made publicly available. “The correlation between infostealer infections and subsequent ransomware attacks is a wake-up call for businesses,” said Trevor Hilligoss, vice president of SpyCloud Labs, SpyCloud. “However, this field is incredibly complex and fast-moving. This year, we’re seeing new infostealers families that make use of expanded capabilities such as advanced encryption to stay stealthy or the ability to restore expired authentication cookies for more persistent access.” The Rise of Malware-as-a-Service And Account Takeover Attacks The infostealer threat is further exacerbated by the rise of Malware-as-a-Service (MaaS). This off-the-shelf model allows even low-skilled cybercriminals to purchase and deploy sophisticated malware, including infostealers, with ease. Through MaaS, these criminals can acquire fresh and accurate identity data in bulk, fueling the cycle of cybercrime. SpyCloud’s findings also shed light on the evolution of account takeover (ATO) attacks, powered by infostealers. Unlike traditional ATO, which relies on stolen credentials (username and password combinations), next-generation ATO leverages stolen session cookies to sidestep traditional authentication methods in what is known as session hijacking. By taking over these already-authenticated sessions, cybercriminals can mimic legitimate users and infiltrate networks undetected. This method significantly increases the success rate of attacks and poses a severe threat to organizational security. “The sheer volume of credentials and session cookies being siphoned by infostealers is staggering,” said Hilligoss. “In the last 90 days alone, SpyCloud has recaptured over 5.4 billion stolen cookie records – with an average of nearly 2,000 exposed records per infected device. This vast trove of data is increasingly used by ransomware operators and initial access brokers to facilitate their attacks, highlighting the need for advanced defense strategies.” Antivirus, MFA And Traditional Defenses Are No Longer Enough At least 54% of devices infected with infostealers in the first half of 2024 had antivirus or endpoint detection and response (EDR) solutions installed, underscoring the limitations of traditional cybersecurity measures in combating the techniques used by modern cybercriminals. Furthermore, infostealers and session hijacking attacks render multi-factor authentication (MFA) and passwordless authentication methods like passkeys ineffective. By hijacking already-authenticated sessions, cybercriminals can impersonate legitimate users and side-step even the most robust authentication methods. The call for next-generation cybersecurity The findings from SpyCloud make it clear: traditional malware mitigation is no longer sufficient and ignoring the problem only exacerbates the impact on businesses. Organizations must move beyond merely removing infections and focus on remediating the long-term risks posed by exposed data. This includes resetting compromised application credentials and invalidating session cookies siphoned by infostealers. By understanding the risks posed by infostealers and working to mitigate the data that has been exfiltrated, organizations are able to limit the likelihood of devastating cyberattacks such as ransomware that stem from this stolen data. SpyCloud remains committed to helping organizations navigate these challenges and safeguard their digital assets. Readers can download the full 2024 Malware and Ransomware Defense Report. To learn more about how SpyCloud helps organizations defend against ransomware, readers can visit https://spycloud.com/use-case/ransomware-prevention/. About the SpyCloud 2024 Malware and Ransomware Defense Report For this fourth annual report, SpyCloud surveyed 510 individuals in active cybersecurity roles within organizations in the US and the UK with at least 500 employees. The report examines the top concerns and real-life impacts of ransomware, including popular entry points, ransom payments, and the cumulative costs of these attacks to the business. It also highlights key cyber threat prevention strategies and future security priorities identified by these experts. About SpyCloud SpyCloud transforms recaptured darknet data to disrupt cybercrime. Its automated identity threat protection solutions leverage advanced analytics to proactively prevent ransomware and account takeover, safeguard employee and consumer accounts, and accelerate cybercrime investigations. SpyCloud's data from breaches, malware-infected devices, and successful phishes also powers many popular dark web monitoring and identity theft protection offerings. Customers include more than half of the Fortune 10, along with hundreds of global enterprises, mid-sized companies, and government agencies worldwide. Headquartered in Austin, TX, SpyCloud is home to more than 200 cybersecurity experts whose mission is to protect businesses and consumers from the stolen identity data criminals are using to target them now. To learn more and see insights on their company’s exposed data, readers can visit spycloud.com Contact EVP, Public Relations Katie Hanusik REQ on behalf of SpyCloud spycloud@req.co This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here AUSTIN, TX, September 18th, 2024/CyberNewsWire/--Research indicates that an infostealer malware infection is often a precursor to a ransomware attack SpyCloud, the leader in Cybercrime Analytics, today announced new cybersecurity research highlighting the growing and alarming threat of infostealers – a type of malware designed to exfiltrate digital identity data, login credentials, and session cookies from infected devices. SpyCloud’s latest findings reveal the staggering scale of identity exposure caused by infostealers, the influence this type of malware has had on the surge in ransomware incidents, and the profound implications for businesses worldwide. Massive Scale of Identity Exposure Creates New Risks According to SpyCloud, 61% of all data breaches in the past year were malware-related, with infostealers responsible for the theft of 343.78 million credentials. These stolen credentials are then sold in criminal communities for use in further attacks. The research also found that one in five individuals has been a victim of an infostealer infection. Each infection, on average, exposes 10-25 third-party business application credentials, creating fertile ground for further access and exploitation, particularly by ransomware operators. “Our latest findings reveal a critical shift in the cybersecurity landscape,” said Damon Fleury, chief product officer at SpyCloud. “Our latest findings reveal a critical shift in the cybersecurity landscape,” said Damon Fleury, chief product officer at SpyCloud. “Infostealers have become the go-to tool for cybercriminals, with their ability to exfiltrate valuable data in a matter of seconds, creating a runway for cyberattacks like ransomware off the vast amounts of stolen access to SSO, VPN, admin panels, and other critical applications.” “Infostealers have become the go-to tool for cybercriminals, with their ability to exfiltrate valuable data in a matter of seconds, creating a runway for cyberattacks like ransomware off the vast amounts of stolen access to SSO, VPN, admin panels, and other critical applications.” Infostealers: The Precursor To Ransomware Attacks The link between infostealers and ransomware is becoming increasingly evident. Through deep analysis of recaptured infostealer logs, SpyCloud discovered a worrying trend: companies with employees and contractors who are infected with infostealer malware are significantly more likely to experience a ransomware attack. In fact, nearly one-third of companies that suffered a ransomware attack last year had previously experienced an infostealer infection. According to the report, this is based on publicly known incidents and confirmed ransomware events. The true exposure is potentially even higher as not all ransomware incidents are made publicly available. “The correlation between infostealer infections and subsequent ransomware attacks is a wake-up call for businesses,” said Trevor Hilligoss, vice president of SpyCloud Labs, SpyCloud. “The correlation between infostealer infections and subsequent ransomware attacks is a wake-up call for businesses,” said Trevor Hilligoss, vice president of SpyCloud Labs, SpyCloud. “However, this field is incredibly complex and fast-moving. This year, we’re seeing new infostealers families that make use of expanded capabilities such as advanced encryption to stay stealthy or the ability to restore expired authentication cookies for more persistent access.” “However, this field is incredibly complex and fast-moving. This year, we’re seeing new infostealers families that make use of expanded capabilities such as advanced encryption to stay stealthy or the ability to restore expired authentication cookies for more persistent access.” The Rise of Malware-as-a-Service And Account Takeover Attacks The infostealer threat is further exacerbated by the rise of Malware-as-a-Service (MaaS). This off-the-shelf model allows even low-skilled cybercriminals to purchase and deploy sophisticated malware, including infostealers, with ease. Through MaaS, these criminals can acquire fresh and accurate identity data in bulk, fueling the cycle of cybercrime. SpyCloud’s findings also shed light on the evolution of account takeover (ATO) attacks, powered by infostealers. Unlike traditional ATO, which relies on stolen credentials (username and password combinations), next-generation ATO leverages stolen session cookies to sidestep traditional authentication methods in what is known as session hijacking. By taking over these already-authenticated sessions, cybercriminals can mimic legitimate users and infiltrate networks undetected. This method significantly increases the success rate of attacks and poses a severe threat to organizational security. “The sheer volume of credentials and session cookies being siphoned by infostealers is staggering,” said Hilligoss. “The sheer volume of credentials and session cookies being siphoned by infostealers is staggering,” said Hilligoss. “In the last 90 days alone, SpyCloud has recaptured over 5.4 billion stolen cookie records – with an average of nearly 2,000 exposed records per infected device. This vast trove of data is increasingly used by ransomware operators and initial access brokers to facilitate their attacks, highlighting the need for advanced defense strategies.” “In the last 90 days alone, SpyCloud has recaptured over 5.4 billion stolen cookie records – with an average of nearly 2,000 exposed records per infected device. This vast trove of data is increasingly used by ransomware operators and initial access brokers to facilitate their attacks, highlighting the need for advanced defense strategies.” Antivirus, MFA And Traditional Defenses Are No Longer Enough At least 54% of devices infected with infostealers in the first half of 2024 had antivirus or endpoint detection and response (EDR) solutions installed, underscoring the limitations of traditional cybersecurity measures in combating the techniques used by modern cybercriminals. Furthermore, infostealers and session hijacking attacks render multi-factor authentication (MFA) and passwordless authentication methods like passkeys ineffective. By hijacking already-authenticated sessions, cybercriminals can impersonate legitimate users and side-step even the most robust authentication methods. The call for next-generation cybersecurity The findings from SpyCloud make it clear: traditional malware mitigation is no longer sufficient and ignoring the problem only exacerbates the impact on businesses. Organizations must move beyond merely removing infections and focus on remediating the long-term risks posed by exposed data. This includes resetting compromised application credentials and invalidating session cookies siphoned by infostealers. By understanding the risks posed by infostealers and working to mitigate the data that has been exfiltrated, organizations are able to limit the likelihood of devastating cyberattacks such as ransomware that stem from this stolen data. SpyCloud remains committed to helping organizations navigate these challenges and safeguard their digital assets. Readers can download the full 2024 Malware and Ransomware Defense Report . 2024 Malware and Ransomware Defense Report 2024 Malware and Ransomware Defense Report To learn more about how SpyCloud helps organizations defend against ransomware, readers can visit https://spycloud.com/use-case/ransomware-prevention/ . https://spycloud.com/use-case/ransomware-prevention/ https://spycloud.com/use-case/ransomware-prevention/ About the SpyCloud 2024 Malware and Ransomware Defense Report For this fourth annual report , SpyCloud surveyed 510 individuals in active cybersecurity roles within organizations in the US and the UK with at least 500 employees. fourth annual report fourth annual report The report examines the top concerns and real-life impacts of ransomware, including popular entry points, ransom payments, and the cumulative costs of these attacks to the business. It also highlights key cyber threat prevention strategies and future security priorities identified by these experts. About SpyCloud SpyCloud transforms recaptured darknet data to disrupt cybercrime. Its automated identity threat protection solutions leverage advanced analytics to proactively prevent ransomware and account takeover, safeguard employee and consumer accounts, and accelerate cybercrime investigations. SpyCloud SpyCloud SpyCloud's data from breaches, malware-infected devices, and successful phishes also powers many popular dark web monitoring and identity theft protection offerings. Customers include more than half of the Fortune 10, along with hundreds of global enterprises, mid-sized companies, and government agencies worldwide. Headquartered in Austin, TX, SpyCloud is home to more than 200 cybersecurity experts whose mission is to protect businesses and consumers from the stolen identity data criminals are using to target them now. To learn more and see insights on their company’s exposed data, readers can visit spycloud.com spycloud.com spycloud.com Contact EVP, Public Relations Katie Hanusik REQ on behalf of SpyCloud spycloud@req.co This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here here

SpyCloud Unveils Massive Scale Of Identity Exposure, Highlighting Need For Advanced Security Measure

SpyCloud تكشف عن نطاق هائل من الكشف عن الهوية، مما يسلط الضوء على الحاجة إلى تدابير أمنية متقدمة

SpyCloud ukax Jach’a Escala de Exposición de Identidad ukar uñacht’ayi, Medida de Seguridad Avanzada ukan munatapat uñacht’ayi

SpyCloud revela una escala masiva de exposición de identidad, lo que pone de relieve la necesidad de medidas de seguridad avanzadas

SpyCloud مقیاس عظیمی از قرار گرفتن در معرض هویت را معرفی می کند، که نیاز به اقدامات امنیتی پیشرفته را برجسته می کند

स्पाईक्लाउड ने पहचान उजागर करने के बड़े पैमाने का खुलासा किया, उन्नत सुरक्षा उपाय की आवश्यकता पर प्रकाश डाला

SpyCloud が大規模な個人情報漏洩を発表、高度なセキュリティ対策の必要性を浮き彫りに

SpyCloud გამოაქვეყნებს პირადობის ექსპოზიციის მასიურ მასშტაბებს, ხაზს უსვამს უსაფრთხოების გაფართოებული ზომების საჭიროებას

SpyCloud විසින් අනන්‍යතා නිරාවරණයේ දැවැන්ත පරිමාණයක් එළිදක්වයි, උසස් ආරක්ෂක පියවරක අවශ්‍යතාවය ඉස්මතු කරයි

СпиЦлоуд открива огромну скалу откривања идентитета, наглашавајући потребу за напредном мјером сигурности

SpyCloud Inafichua Kiwango Kikubwa cha Ufichuaji wa Utambulisho, Ikiangazia Haja ya Kipimo cha Kina Usalama

ஸ்பைக்ளவுட் அடையாள வெளிப்பாட்டின் பாரிய அளவை வெளிப்படுத்துகிறது, மேம்பட்ட பாதுகாப்பு நடவடிக்கையின் தேவையை எடுத்துக்காட்டுகிறது

SpyCloud tiết lộ quy mô lớn về việc tiết lộ danh tính, làm nổi bật nhu cầu về biện pháp bảo mật tiên tiến

The Economics of Public Cloud Repatriation and Why You Can't Stay in the Cloud at Scale

İctimai Bulud Repatriasiyasının İqtisadiyyatı və Niyə Ölçəkdə Xərcli Qadağandır

পাবলিক ক্লাউড প্রত্যাবর্তনের অর্থনীতি এবং কেন এটি স্কেলে ব্যয়-নিষিদ্ধ

Ekonomija repatrijacije javnog oblaka i zašto je skupo na nivou

La economía de la repatriación a la nube pública y por qué resulta prohibitivamente costosa a gran escala

パブリック クラウドの回帰の経済性と、大規模になるとコストがかかりすぎる理由

د عامه کلاوډ بیرته ستنیدو اقتصاد او ولې دا په پیمانه لګښت منع دی

Iyo Economics yePublic Cloud Repatriation uye Nei Iri Mutengo-unorambidza paChikero

பொது கிளவுட் திருப்பி அனுப்புதலின் பொருளாதாரம் மற்றும் அது ஏன் அளவில் செலவு-தடை

Kamu Bulutu Geri Yüklemenin Ekonomisi ve Ölçekte Maliyet Açısından Neden Engelleyici Olduğu

Ommaviy bulutli repatriatsiya iqtisodiyoti va nega u miqyosda qimmatga tushadi

Kinh tế học của việc hồi hương đám mây công cộng và lý do tại sao nó lại quá tốn kém ở quy mô lớn

I-Economics of Public Cloud Repatriation kanye nokuthi kungani ibiza kakhulu ngokwesikali

Stories, News & Mentions about SCALE

Scale’s Stories on HackerNoon