Shahzad Bhatti Welcome to my ramblings and rants!

As REST based Micro-Services have become prevalent, I often find that web and mobile clients have to connect to different services for gathering data. You may have to call dozens of services to display data on a single screen or page. Also, you may only need subset of data from each service but you still have to pay for the bandwidth and parsing cost.

I created a new Java framework PlexDataProviders for aggregating and querying data from various underlying sources, which can be used to build a general-purpose data service. PlexDataProviders is a light-weight Java framework that abstract access to various data providers such as databases, files, web services, etc. It allows aggregation of data from various data providers.

The PlexDataProviders framework is divided into two components:

• Data Provider – This component defines interfaces that are implemented to access data sources such as database or web services.
• Query Engine – This component is used for querying and aggregating data.

The query engine can determine dependency between providers and it also allow you to use output of one of the data provider as input to another data provider. For example, let’s assume:

• data-provider A requires input-a1, input-a2 and produces output-a1, output-a2
• data-provider B requires input-b1 and output-a1 and produces output-b1, output-b2

Then you can pass input-a1, input-a2 to the query engine and request output-a1, output-a2, output-b1, output-b2 output data fields.

Benefits

PlexDataProviders provides offers following benefits:

• It provides a unified way to search data and abstracts integration to underlying data sources.
• It helps simplifying client side logic as they can use a single data service to query all data instead of using multiple data services.
• This also help with managing end-points as you only a single end-point instead of connecting to multiple web services.
• As clients can specify the data they need, this helps with payload size and network bandwidth.
• The clients only need to create a single data parser so it keeps JSON parsing logic simple.
• As PlexDataProviders supports multi-threading, it also helps with latency of the data fetch requests.
• It partial failure so that a failure in a single data provider doesn’t effect other data providers and the data service can still return partial results. User
• It supports timeout so that clients can receive available data that completes in given timeout interval

Data Structure

Following are primary data structures:

• MetaField – This class defines meta information for each data field such as name, kind, type, etc.
• MetaFieldType – This enum class supports primitive data types supported, i.e.
  • SCALAR_TEXT – simple text
  • SCALAR_INTEGER – integer numbers
  • SCALAR_DECIMAL – decimal numbers
  • SCALAR_DATE – dates
  • SCALAR_BOOLEAN – boolean
  • VECTOR_TEXT – array of text
  • VECTOR_INTEGER – array of integers
  • VECTOR_DECIMAL – array of decimals
  • VECTOR_DATE – array of dates
  • VECTOR_BOOLEAN – array of boolean
  • BINARY – binary data
  • ROWSET – nested data rowsets
• Metadata – This class defines a set of MetaFields used in DataRow/DataRowSet
• DataRow – This class abstracts a row of data fields
• DataRowSet – This class abstracts a set of rows

PlexDataProviders also supports nested structures where a data field in DataRow can be instance of DataRowSet.

Adding a Data Provider

The data provider implements following two interfaces

[codesyntax lang="java"]

public interface DataProducer {
    void produce(DataRowSet requestFields, DataRowSet responseFields,
            QueryConfiguration config) throws DataProviderException;
}

[/codesyntax]

Note that QueryConfiguration defines additional parameters such as:

• pagination parameters
• ordering/grouping
• filtering parameters
• timeout parameters

The timeout parameter can be used to return all available data within defined time, e.g. query engine may invoke underlying data providers in multiple threads and if underlying query takes a long time then it would return available data.

[codesyntax lang="java"]

public interface DataProvider extends DataProducer, Comparable<DataProvider> {
    String getName();

    int getRank();

    Metadata getMandatoryRequestMetadata();

    Metadata getOptionalRequestMetadata();

    Metadata getResponseMetadata();

    TaskGranularity getTaskGranularity();
}

[/codesyntax]

Each provider defines name, rank (or priority when matching for best provider), set of mandatory/optional input and output data fields. The data provider can also define granularity as coarse grain or fine grain and the implementation may execute those providers on different threads.

PlexDataProviders also provides interfaces for converting data from domain objects to DataRowSet. Here is an example of provider implementation:

Typically, you will create data-provider for each different kind of query that you want to support. Each data provider specifies set of required and optional data fields that can be used to generate output data fields.

Here is an example of marshalling data from Securty domain objects to DataRowSet:

PlexDataProviders provides DataProviderLocator interface for registering and looking up provider, e.g.

PlexDataProviders comes with a small application that provides data services by implementing various data providers. It uses PlexService framework for defining the service, e.g.

As you can see the data service simply builds DataRequest with input data fields and sends back response back to clients.

Here is an example client that passes a search query data field and requests quote data fields with company details

public void testGetQuoteBySearch() throws Throwable {
    String jsonResp = TestWebUtils.httpGet("http://localhost:" + DEFAULT_PORT
                    + "/data?responseFields=exchange,symbol,quote.bidPrice,quote.askPrice,quote.sales,company.name&symbolQuery=AAPL");
    ...

Note that above request will use three data providers, first it uses SymbolSearchProvider provider to search for matching symbols with given query. It then uses the symbol data field to request company and quote data fields from QuotesBySymbolsProvider and CompaniesBySymbolsProvider. The PlexDataProviders framework will take care of all dependency management for providers.

Here is an example JSON response from the data service: