Spring Batch Architecture

Spring batch takes care of one major aspect. Keep batch processing logic and business logic separate. In the sense that application code and batch processing code remains separate.

We have below three chunks which comprises of the Spring Batch Architecture.

1. Spring Batch Infrastructure: Readers/Writers and Services such as RetryTemplate are a part of basic batch infrastructure. Batch Core and Program Application is built upon this Infra.

2. Spring Batch Core: Includes JobLauncher, Job, Step. All the core runtime classes required to launch and control a batch job.

3. Application: This is the Application code that we write which is controlled by Core and written on top of Infra.

Spring Batch Introduction

Coding with Spring has always been like a breeze. With Spring  Batch, we go a notch higher. Remember batch processing of the old, where we would wait for considerable time to get the results back. Spring batch mimics same behavior.

Target Applications

It is preferred in applications which are time intensive such as computing multi-carlo simulations on the grid. This computation takes time and a lot of such computations takes places in parallel. So it's better to define steps which will be executed one after the other. Following steps will wait for hours if required, if the previous steps are not completed.

It is used in batch applications which process large volume of records. Inherent support is provided for job start/stop/restart, transaction management, logging, job statistics among others. Minimal code needs to be written. Spring Batch takes care of most of the things. High-volume and complex batch processes can take advantage of Spring batch.

Billions of Computations in a application I worked on

Such applications proceed in an automated manner without user involvement. One such application that I worked on was to calculate month end risk numbers via multi-carlo simulations. Billions of computation used to take place in a single batch (computations used to run on Grid of computers). The repetitive steps used to wait on external system which used to send inputs on JMS. The results of Grid were then bundled up with inputs from external systems and output was generated.

Open Source

Spring Batch is open source framework. It is very helpful for people from Spring background. A couple of hours tutorial and viola, a Batch routine is ready. With Spring Batch, we can leverage other Spring tools like Spring-JMS, Spring File-IO services etc.

Typical Batch Scenario

A typical batch scenario comprises of the following steps.

Spring batch provides robust, enterprise scale solution to automate these steps so as no user effort is required during processing.

VMware Gemfire v/s Pivotal Gemfire

VMGemfire is now Pivotal Gemfire. Pivotal and VMware have come together for this latest version of Gemfire. It is a Distributed Data Management System.

Gemfire is used across spectrum of applications for low-latency, high-volume and transaction systems.

Below are the Important Features of Pivotal Gemfire:

1. Enhance UI for monitoring

An enhanced UI for monitoring entire Gemfire deployment is provided with Pivotal Gemfire. Nodes can be tracked and started/stopped for the consolidated UI. Metrics foe each node can be viewed in nice graphics.

2. JSON Support

We can store JSON documents in Pivotal Gemfire with support to query inside the documents.

3. Better Persistence of Data

Pivoal Gemfire provides persistence to both partitioned and replicated data. Database queries become minimal thereby reducing the costs of fancy databases.

4. Spring Integration

Pivotal Gemfire has easy integration with Spring, thereby making the code writing easier. It helps segregating the data access from business logic effectively.

5. Higher Throughput and Lower latency and Higher Availability

Further optimizations, concurrent data structures for storage has led to higher throughput and further lowered the latency in Pivotal Gemfire. Coupled with shared-nothing architecture, Pivotal Gemfire provide very high and continuous availability.

6. L2 Caching

This helps in increasing the Hibernate performance and minimizes database slowness. 

7. WAN distribution

With Pivotal Gemfire distributed systems can scale out over the WAN in an un-bounded way while maintaining reliability and consistency of data.

8. Cross Languages

Business Objects can be shared seamlessly across C++,C# and Java applications. No need for cumbersome transformations.

Pivotal Gemfire: Introduction

Pivotal Gemfire was earlier known as VMware Gemfire, is an

enterprise caching platform used across the industry. Most avid user of Gemfire are investment banks where availability of data means money.

Gemfire is like a Distributed Data Grid. It pools in memory of multiple processes  and maintains the application level cache. Gemfire provides three most sought after things, availability of data, reliability of data and manageability of large data.

Region

It works by creating partition regions (where data is stored a Maps). Regions are created inside the process heap memory itself. The regions are partitioned to store chunks of data separately. These partitions are then replicated to introduce redundancy in case of some failure.

In case of Failure

If any one node fails, recovery of data takes place immediately. Meanwhile, user can get a seamless experience due to fail-over to replicated region.

P-2-P

In Gemfire, Each node connects to all other peers directly. It also supports Clients of this Distribute System. Clients can have their own local cache which can get updates from server if and when any data changes.

There is no single point of failure in Gemfire which is very critical these day. Also very high throughput systems can be built using Gemfire.

Next we will read about various technical aspects of Gemfire.

Java hashcode() and equals()

hashcode() and equals() occupies the first round of interviews in most firms. As the interview proceeds, so does the difficulty level.

Fun part is, if you understand the concept once, all the questions would seem easy.

To the uninitiated, hashcode() and equals() belong to java.lang.Object class. Since all classes extendsObject class, hence all classes inherit the default hashcode() and equals()

Equals

1. By default For not null Objects, equals() method return true if they occupy same place in memory, i.e if the Objects are same. else return false.

2. equals() perform for Object by default, what we call shallow comparison, i.e just that the object is same or not. Sometimes Object can be different but they might have the same data. In such a case, we need to do a deep copy, i.e. compare each member variable to come to a conclusion (by overriding equals)

3. However if equals() is overridden, so should be hashcode(). As two objects that are equal, must return the same hashcode (vice-versa not true). However more on this later.

Hashcode

1. By default, hashcode() varies from on Java implementation to another. Mostly it converts the memory address of the Object into and integer and that becomes the hashcode.

2. By default It is unique for two different Objects.

3. As two objects that are equal, must return the same hashcode. Two unequal objects need not have different hashcode. However it's better to have.

4. Hashcode need to consistent. i.e. no matter how many times we calculate hashcode of a give Object, we should get the same value.

We can override hashcode() and equals() implementation as per our requirement. They are used extensively to store values in HashMap and HashTables.

Memory Leak In Java

An advanced interview question is, "Can we have memory leaks in Java"? More often than not the interviewer wants to know how much you know Garbage Collector in Java.

What all objects are Garbage Collected and when. If we explain on the basis of this, the interviewer (who is just as human as you are) will be satisfied 9 times out of 10.

As to the answer to the question of Memory Leaks, it is a bit contentious issue with no concrete answer.

My take is a follows, Java as a language was designed to not let memory leaks of C creep in. Then daemon Garbage Collector, takes care of memory allocation and deallocation. Once the object is not referenced it will in time be Garbage collected and memory released. So on the face of it Java does not seem to have classic memory leaks of C.

To go deep into this, we first need to know what exactly is memory leak? Well memory leak occurs when some an application allocates (repeatedly) a piece of memory which is not acquired back after it is not longer referenced. Eventually program will throw OutOfMemory Exception.

Java Garbage Collector doe not let this happen. A case can be when memory is allocated faster than it is de-allocated. This is more of an programming issue.

How can we create memory leak in Java? By code bug so that either, memory fills faster than Garbage Collection or fooling Garbage Collection not to collect a piece of memory.

1. Static variable should not allowed to hold large data. They can cause memory leaks.

2. A cache (HashMap) with a badly written hascode() and equals(0 can lead to memory leaks. One can store the value to a Map but can't get it back.

3. Inner Classes referencing Outer class can lead to memory leak.

4. A long running Thread can cause memory leak in Java.

5. A simple code would be to keep on growing a Linked List in Java till OutOfMemory occurs.

Google Protobuf: proto3 enhancements

Proto3 is the new version of Google Protobuf with various enhancements and additions. Let's go over them one by one.

1. Since all fields are optional by default. Hence "optional" keyword has been done away with. If nothing is specified, it will be considered optional. (Protof files just became a tad smaller)

2. Support to group fields are not longer there. Group was just another way of nesting information in messages. But it was inefficient way to do (better way, Nested Messages). Hence is no longer supported.

3. Packed repeated fields were introduced in 2.1.0. Repeated fields needed to be explicitly set packed=true to use this encoding. In proto3, repeated primitive fields have been made packed by default. They have to be set false to be disabled. 

4. Common proto types (timestamp.proto etc) have been added. They can be used after importing. This takes a step towards usable library in protobuf. See below,



5. reserverd keyword introduced. By this, user has the control over reserved field_names and field_numbers. Once declared as reserved, these can't be used anywhere in the message.

Below, the field_numbers and names can't be used anywhere in the message again. Gives us more control.


6. Support for Objective-C and C# introduced in proto3. yay!

7. There are other language specific changes for various languages.

Google Protobuf: Introduction

Google Protobuf has simplified a historically cumbersome process of sending messages from one component of an application to another.

They have made the communication, data storage, platform independent. An object serialized in Java can easily be de-serialized in C++.

Google protobuf aids in managing structured data in a better way. Definitions are easily extensible and simple to use.

.Proto File
Data Definition is created once in a simple format. It is stored as .proto file. These proto files are much simpler to write and manage than xml files that we have been using till now.

This file is compiled and we get generated source code in whatever programming language we want. We can then populate the object using Builder/Getter-Setter, Serialize it and send/store. Later de-serialize it in a different language (or same) and retrieve the information.

 An example below



This .proto can now be compiled and language specific Data Access Classes are created. These classes have auto-generated getter/setter functions for members.

We can declare package name at the top of protofiles, dependency on other protofiles, just like any Java code. Compiler takes care of it all.

More about Google Protobuf below;

Google Protobuf: Historical Significance

With the advent of distributed applications, the messaging between one component to another has become all too prominent. There are a lot of middleware mechanisms to do so. Tibco EMS, ZMQ, etc. I have worked on projects involving a few of such mechanisms. All have their own positives and negatives depending on the problem at hand.

While communicating between components, we need data units to be sent to and fro. What those data units will be? You might have used xml units send across. Remember how cumbersome was to write parsing code at both places. And if the components are written in different languages (Java and C++ say), then it becomes all too messy. Writing receivers at both ends and parsing logic too. Handling exceptions and what not. 

Some might have used json. Programmers of the old might remember Serialization/De-serialization and sending Java as a whole. But what to do when other component is not in Java? Again it becomes messy.

Well Google Protobuf's is one such endeavour to put those issues to an end.

Google provides libraries for various programming languages. A .proto file is created (looks somewhat like json, but not all too much), in which we can define structured data structure of the Object to be move to and fro. The same proto file can be compiled in different languages into class objects. 

These classes can now be used simply, Call setters and populate the object. Send it across. Easy de-serialization at other end, and we get local language specific Object. Call getters and get the data.

Thus a Java Object created Serialized, Sent Across, Received, De-Serialized as C++ Object.

Thus from messy history, this seems like a fresh breeze. Doesn't it?

Multithreaded Quick Sort

Quick sort is a classic example where we can use multiple threads, since execution proceeds on separate paths on separate segments. You can read about classic Quick Sort here.

Quick Sort Steps

1. Select an element as pivot (preferable the last element).

2. Shift the pivot such that all elements before it are smaller and after it after bigger (Pivot reaches correct position)

3. Sort the two segments (Before Pivot and After Pivot) separately

Now check the last step, The two segments can be sorted and processes by 2 separate Threads easily as they don't depend on each other. In turn they can spawn more segments and more threads. For this we can use ThreadPool.

Multi-threaded Quick Sort Steps

1. Select an element as pivot (preferable the last element).

2. Shift the pivot such that all elements before it are smaller and after it after bigger (Pivot reaches correct position)

3. Sort the two segments (Before Pivot and After Pivot) separately via Two Threads

INPUT

2

6

7

5

3

1

8

4

PROCESSING

2

6

7

5

3

1

8

4

2

8

7

5

3

1

4

6

2

1

7

5

3

4

8

6

2

1

3

5

4

7

8

6

2

1

3

4

5

7

8

6

THREAD 1 2 1 3	THREAD 2 5 7 8 6
1 2 3	5 7 6 8 5 6 7 8

FINAL RESULT

1

2

3

4

5

6

7

8