Posts

Showing posts with the label Pub/Sub Redis queue

Evolving code and adding new technologies - Part 5

Image
The implementation of the proposed architecture is at hand. We dive into the change and see how Redis (or any other message queue or pub/sub broker) makes our life easier. TL;DR Time to make the change. Read it all!

For a recap here is Part 1Part 2Part 3 and Part 4
Components of our exampleGitHub branch
Our monolithic example will be divided to three processes: ProducerRedis (3rd party broker)Consuming ProcessorThin client After we are done with the change, the server will look like this:

On to the code The Dataflow library from the previous part will be removed in the consumer. Now that the processes are separated the consumer is super fast up until the actual processing which takes around 1 second.

The different components will work directly with Redis which will in fact be the backbone of our pipeline.

Ten producers are being used to send messages into 10 Redis lists. Each list has a key named {Producer_i} where i is the number of the producer. The consumer is aware that there a…