In the bustling world of online shopping (revolutionised by Jeff Bezos) where millions of clicks translate to real-world transactions every minute, keeping up with the pace of the ever-growing clicks is no small feat. In comes Gautham Rajendiran, who helped one leading e-commerce company, transform its digital backbone to handle the ever-growing flood of online orders.
The company was using a system called RabbitMQ to manage its inventory and process orders. Think of RabbitMQ as a super-fast postal service for digital information transferring communication from one channel to another. It worked well when everything was happening in one place, but as the company grew and spread its operations across multiple locations, things got tricky.
"We started noticing that sometimes, the order of events would get mixed up," explains Rajendiran the lead engineer, behind the project. It's like if you're reading a story, but the pages get shuffled. This issue became particularly noticeable during high-traffic periods like Black Friday or the unexpected e-commerce boom triggered by the COVID-19 pandemic. The company needed a solution to handle more orders without losing track of what came first.
Therefore, Rajendiran helped the company transfer from the old system to Apache Kafka, a different digital messaging system. Kafka works more like a library with multiple, well-organized bookshelves containing numerous books. Each "book" (or piece of data) has its specific place, and Kafka makes sure that the book goes to its place, making it easier to keep track of large volumes of information in the right order.
“We can now process up to 2 million events per minute, up from 1.3 million with our old system,” he explains. “The leap was possible due to Kafka's ability to scale horizontally across multiple partitions without compromising performance or event ordering."
But it's not just about speed. The new system ensures that related pieces of information - like all the updates about a single order - stay together and in the right sequence, even as the data goes through different parts of the company's digital network.
Another effective feature was if something goes out of order, the new system can "rewind" and replay the events correctly. It's like being able to reshuffle those mixed-up story pages back into the right order. However, given the advantages, the transition wasn't without its challenges. The team had to carefully plan the switch to avoid disrupting the millions of customers who rely on the platform daily. They also had to train staff to work with the new system and adapt their existing software to work with Kafka.
The improved system has allowed the company to handle major shopping events with ease, from game day surges to holiday/festival shopping sprees. It's also traffic-proofed the platform, allowing for further growth and expansion. To make the e-commerce platforms more efficient, Rajendiran suggests using a tool called Apache Flink, which can mark orders that arrive late and handle them correctly.
Apache Flink also waits for all the necessary information before processing an order. It also keeps the old system around (as an alternative), which can help if something happens to the new system.
While Rajendiran acknowledges that managing data in the fast-paced world of e-commerce will always be a complex task, he is confident that their new system can be a foundation they need to keep experimenting to tackle heavy traffic flows. As online shopping continues to grow, handling major traffic flows will become critically important. These invisible improvements and the people behind the scenes like Gautham Rajendiran keep our digital world spinning smoothly, one click at a time.
This content is produced by Menka Gupta.
