Client: Global provider of technology solutions for retail industry

Industry: Retail

Goal: Assist in reviewing and implementing a Kafka-based real-time stock counting subsystem

Tech: Java, Spring, Apache Kafka, RabbitMQ, Apache Cassandra, InfluxDB, Grafana



Speak to us about your software development requirements:



The client wanted to develop a new addition to their existing solution – a real-time stock counting subsystem (RTS) integrated into the existing system. RTS receives a stream of EPCIS events, processes them in real-time and maintains the status of all known EPCs and stock account information per business location. RTS needs to answer high data volume demands, as well as ensuring 100% correct stock information (no EPC event can be lost).

In parallel, the whole system is connected to ERP systems of clients and in batch mode stock is imported from the ERP system as well. When there is both real-time stock count and stock count from the ERP system, the difference list is calculated and presented to the client through web UI.


The engagement was split into three phases. The goal for the first phase was to review the existing RTS project, to apply the best practice applicable to the development of a Kafka-based system and, finally, to suggest any necessary modifications and tuning options. The scope of the second phase included the implementation of the suggested modifications. The third, and final phase, was implementing a monitoring system as a part of RTS to be used in a production environment, and a load generator used to perform end-to-end tests.


The first phase was executed taking into account the latest Kafka development (Kafka version 0.9) and best practices and recommendations related to it, and matching them against the existing version of the RTS project (Kafka version 0.8.2). The review also revealed problems related to horizontal scaling and no message loss guarantees. The findings from the first phase were addressed in the second phase by switching to the new Kafka consumer API. The necessary preparations for the third phase were made during this phase by integrating metrics libraries/tools (Java Metrics and Kafka Burrows) into RTS. In the third phase a monitoring stack, consisting of Telegraf, InfluxDB, Grafana, and Burrow, was developed and integrated with RTS.


At the end of this engagement, RTS was deployed in the production environment together with the monitoring stack. Having RTS in the production environment enables end users (retail organizations) to access the stock information such as stock counts and product lists in real-time, easing and improving their daily operations. The monitoring stack allows for faster spotting and preventing situations that could lead to potential problems in production. Using Apache Kafka for implementing the data pipeline builds a scalable platform for further system extensions, easing the new development and cutting down the time-to-market.


Speak to us about your software development requirements: