Gemfire: Market Data Application Case Study

There are a lot of applications where Gemfire will turn out to extremely useful. I will discuss one such application that I worked on for an Investment bank where we implemented Gemfire. I would not be able to give exact business usage as it was a proprietary application and am bound with non-disclosures. Hence won't be able to give code details.

Application: Market Data Platform 

Just like Reuters/Bloomberg which provides market data across the world, we developed and internal market data platform spanning across Newyork, London, Mumbai, Singapore, Tokyo. 

Point to notice here is that the application has to provide consistent data without time lag as for Investment banks it's time that matters.

Case 1: Server in Newyork: This won't work since there would be a considerable time lag due to network, while accessing data from London/Mumbai/Singapore/Tokyo. Also the inbuilt cache would not be reliable and scalable. If application goes down due to any reason, along with that entire cache. Recovery,Replication has to be developed from scratch.

Case 2: Multiple servers across continents: This would in-turn induce consistency issues. A data update from Tokyo will take time to reach Newyork. In between system will show different data in Newyork and Tokyo (Imagine Reuters returning different values for same ric Code in Tokyo and Newyork, It would wreak havoc in share market).

Hence we use gemfire. It will take care of servers in multiple continents. Cache will be available will multiple nodes (on different servers, running in each location). This would keep redundant copies of data in each location. Say we keep 2 nodes each in NY/Lon/SG/Tok. Now the data once updated in NY, Gemfire copies it immediately across all nodal points in other regions. Yes, there would be some time lag, but that is acceptable.

Replication Process via Gemfire goes on across Locations, keeping the Data consistent. We can also run manual replication from time to time if required.

We successfully implemented Gemfire. We were among the first in the world to implement it at this level. About 4-5 years ago. There was not much online support provided for Gemfire at that point. VMware provided two dedicated consultants in London to help us out if we get stuck at any point.

VMware Gemfire v/s Pivotal Gemfire

VMGemfire is now Pivotal Gemfire. Pivotal and VMware have come together for this latest version of Gemfire. It is a Distributed Data Management System.

Gemfire is used across spectrum of applications for low-latency, high-volume and transaction systems.

Below are the Important Features of Pivotal Gemfire:

1. Enhance UI for monitoring

An enhanced UI for monitoring entire Gemfire deployment is provided with Pivotal Gemfire. Nodes can be tracked and started/stopped for the consolidated UI. Metrics foe each node can be viewed in nice graphics.

2. JSON Support

We can store JSON documents in Pivotal Gemfire with support to query inside the documents.

3. Better Persistence of Data

Pivoal Gemfire provides persistence to both partitioned and replicated data. Database queries become minimal thereby reducing the costs of fancy databases.

4. Spring Integration

Pivotal Gemfire has easy integration with Spring, thereby making the code writing easier. It helps segregating the data access from business logic effectively.

5. Higher Throughput and Lower latency and Higher Availability

Further optimizations, concurrent data structures for storage has led to higher throughput and further lowered the latency in Pivotal Gemfire. Coupled with shared-nothing architecture, Pivotal Gemfire provide very high and continuous availability.

6. L2 Caching

This helps in increasing the Hibernate performance and minimizes database slowness. 

7. WAN distribution

With Pivotal Gemfire distributed systems can scale out over the WAN in an un-bounded way while maintaining reliability and consistency of data.

8. Cross Languages

Business Objects can be shared seamlessly across C++,C# and Java applications. No need for cumbersome transformations.

Pivotal Gemfire: Introduction

Pivotal Gemfire was earlier known as VMware Gemfire, is an

enterprise caching platform used across the industry. Most avid user of Gemfire are investment banks where availability of data means money.

Gemfire is like a Distributed Data Grid. It pools in memory of multiple processes  and maintains the application level cache. Gemfire provides three most sought after things, availability of data, reliability of data and manageability of large data.

Region

It works by creating partition regions (where data is stored a Maps). Regions are created inside the process heap memory itself. The regions are partitioned to store chunks of data separately. These partitions are then replicated to introduce redundancy in case of some failure.

In case of Failure

If any one node fails, recovery of data takes place immediately. Meanwhile, user can get a seamless experience due to fail-over to replicated region.

P-2-P

In Gemfire, Each node connects to all other peers directly. It also supports Clients of this Distribute System. Clients can have their own local cache which can get updates from server if and when any data changes.

There is no single point of failure in Gemfire which is very critical these day. Also very high throughput systems can be built using Gemfire.

Next we will read about various technical aspects of Gemfire.