Part 8: From Microservice to Microservice Ecosystems – Using Docker Compose To Orchestrate a Microservice Harness

In our previous post we finally began to see the individual microservices coming together to create an  abstract plausible data flow. In this post, we are going to chain our microservices together to run an entire calculation stream sequentially. We are going to illustrate how to orchestrate a collection of microservices using Docker Compose, and we are going to illustrate how to use the Docker Registry to avoid recreating the wheel when creating microservice ecosystems.

Microservice Ecosystem

In our current design we have a microservice ecosystem defined by 3 distinct applications that we have packaged using Docker containers. We have

  1. A Web Content Extractor that accepts a URL and returns the textual content within that URL – Golang.
  2. A Tone Analysis service that accepts a string of text and submits that text for analysis using IBM Watson’s Tone Analysis Service. It returns the content scored along 10 emotional traits. Python.
  3. A Stock News Microservice that will accept a stock ticker symbol and return the latest 20 stories related to the stock symbol. Python.

Instead of writing another post on creating another microservice, we are going to do what any good computer programmer does, and reuse code. We need a microservice that will accept a stock symbol and return us the stock price. So we search the Docker Registry for “stock price” and find us a simple stock price microservice.

For this we use this random fella’s stock price microservice.

Defining a Microservice Ecosystem with Docker Compose.

To use this microservice ecosystem to start chaining together microservices, we have to start up 4 different containers. This can be time consuming so we instead will use Docker Compose to start up the entire ecosystem of containers. We create a docker-compose.yml file below to define each of our containers we want started, and the ports they are assigned to

Now with our ecosystem defined, we can instruct Docker to spawn up the whole orchestra of containers with one command

Additionally we will need to place an “env.list” file within the directory to hold our credentials for bluemix. A sample env.list file

After we have setup our secrets file, we can start our ecosystem with:

As we can now see, a visit to http://localhost:5000/status or http://localhost:5003/healthz returns that our microservice ecosystem is up and running.

Using Our Ecosystem

At this point we are going to try and run a single stock ticker through the entire series of microservices in a very sequential fashion. This means that we are taking no advantage of container scaling, or event driven architectures to optimize resource usage and computation speed. The rough flow will be

  1. Submit Ticker to get Stock Price
  2. Submit Ticker to get story URLs
  3. Submit each story URL to scraping
  4. Submit each URL’s scraped content to tone analysis
  5. Report statistics on the entire batch of URLs

For this flow we write a simple python script that assumes the Docker ecosystem has been started with docker-compose. Our script is as follows:

From the command line, we run

We will see the stock price and name information and a note that analysis is running. Then we see the following output after a significant amount of time (it takes a while to run 19 stories through emotional analysis):

Our analysis shows that over 19 stories, they are more analytical than anything. Thats to be expected from a stock market ticker news feed. Interestingly, for this particular stock, we are seeing elevated sadness with a very low variance and deviation.  This is interesting, as this particular stock has been on a 1 month freefall in price. Now I might be cherry picking a result that makes sense to bolster the effectiveness of an algorithm that on the whole is inconclusive – the data has yet to be collected to determine if that is the case – but for this example we might have a good predictor of stock performance.

As you can tell, calculation time takes a while. Obviously to run this on 3000 ticker symbols will not be feasible if each ticker takes on the order of minutes to complete. There are only 1440 minutes in a day. We have to look at parallelizing the architecture of the computations. Additionally we have concerns with caching and running our computations for the same input multiple times. If we can consolidate what has to be computed, and then adopt an architecture that scales parts of the computation with the load for that part, we will produce a system that can very quickly gauge the emotional setting for each and every stock ticker symbol – and track it by stock price at that time.

I have included the source code here on GitHub.