Monday, June 15, 2015

Gemfire: Market Data Application Case Study

There are a lot of applications where Gemfire will turn out to extremely useful. I will discuss one such application that I worked on for an Investment bank where we implemented Gemfire. I would not be able to give exact business usage as it was a proprietary application and am bound with non-disclosures. Hence won't be able to give code details.

Application: Market Data Platform 

Just like Reuters/Bloomberg which provides market data across the world, we developed and internal market data platform spanning across Newyork, London, Mumbai, Singapore, Tokyo. 

Point to notice here is that the application has to provide consistent data without time lag as for Investment banks it's time that matters.

Case 1: Server in Newyork: This won't work since there would be a considerable time lag due to network, while accessing data from London/Mumbai/Singapore/Tokyo. Also the inbuilt cache would not be reliable and scalable. If application goes down due to any reason, along with that entire cache. Recovery,Replication has to be developed from scratch.

Case 2: Multiple servers across continents: This would in-turn induce consistency issues. A data update from Tokyo will take time to reach Newyork. In between system will show different data in Newyork and Tokyo (Imagine Reuters returning different values for same ric Code in Tokyo and Newyork, It would wreak havoc in share market).

Hence we use gemfire. It will take care of servers in multiple continents. Cache will be available will multiple nodes (on different servers, running in each location). This would keep redundant copies of data in each location. Say we keep 2 nodes each in NY/Lon/SG/Tok. Now the data once updated in NY, Gemfire copies it immediately across all nodal points in other regions. Yes, there would be some time lag, but that is acceptable.

Replication Process via Gemfire goes on across Locations, keeping the Data consistent. We can also run manual replication from time to time if required.

We successfully implemented Gemfire. We were among the first in the world to implement it at this level. About 4-5 years ago. There was not much online support provided for Gemfire at that point. VMware provided two dedicated consultants in London to help us out if we get stuck at any point.

No comments:

Post a Comment