April 21, 2023

Taming the Chaos: Sourcing Financial Data from Multiple Providers

Combining multiple data sources can help your app or website stand apart from the pack. However, it also comes with several challenges. This guide can get you started on the right foot.

Cloud icon

James Konik

https://iexcloud.io/community/blog/taming-the-chaos-sourcing-financial-data-from-multiple-providers
@iexcloud
https://iexcloud.io/community/blog/taming-the-chaos-sourcing-financial-data-from-multiple-providers
@iexcloud

Financial applications often need to source their data from multiple providers. For instance, if you're trading stocks, watching currency fluctuations, or tracking product sales, amalgamating multiple data sources can help present a broader picture than that of your competitors'. Or if you're performing credit analysis, the more information you have, the better.

Getting data from different sources into your app or website quickly and in a consistent format isn't easy, though. Different sources will structure their data differently, use different storage formats, and expose their data using different methods. You could also find repetition or inconsistencies between data sets. As joining data is costly in performance terms, you have to minimize that as much as possible too.

Apperate is a cloud data platform that lets you pull in data from multiple providers, manipulate it in one place, and make it available to other applications. It includes integrated financial data bundles from IEX Cloud, giving you a source of easy-to-access data to get started with. It also makes it easy to add streamed data from other sources.

In this article, you'll learn more about the challenges of working with multiple data providers and how you can use Apperate to overcome these challenges.

Challenges of Sourcing Data from Multiple Providers

Let's consider some of the challenges of sourcing data from multiple providers more comprehensively.

Complexity of the Integration

Different providers use different data formats. When amalgamating the data, you need to convert it into a common format. This is known as data transformation and mapping. You need to set that up for each external source you use to make sure your data ends up in the same format. And if a provider changes their access format, you'll need to update your system—for example, updating API calls to match the source's updates.

You also have to consider the possibility of duplicate data. If you use different sources for stocks, they will almost certainly cover many of the same companies. You need to think about how you will handle this duplication.

Eliminating duplicates is one solution, but you'll still have the challenge of needing to decide which data to use. You could average the data, use the most recent entry, or present your users with the different values with appropriate labels.

Data Governance

How you use data is important. If your data sources include potentially sensitive information, such as payment details or credit scoring, there could be restrictions on how it's used. These data governance requirements could include what you're able to store and where you're allowed to store it, along with rules on how it's protected.

Of course, these rules will change from one jurisdiction to another. To make things even harder, rules typically change quickly, meaning you're working with a moving target.

Data governance requirements can also be ambiguous, and those ambiguities might stay there until they're tested in court, which may or may not ever happen. If you're unsure how to interpret any data governance regulation, you probably need to consult with your legal team to make sure you're compliant and to find out how you can remain so.

You also need to think about data lineage and how reliable your data is. While some sources update daily, financial data, such as stock values, often change faster than that. If you have several data sources, you will likely have to opt for using the most recent. You can also look out for discrepancies between sources to detect if any of them provide inaccurate data.

Costs

The most obvious cost of sourcing financial data from multiple providers is the price of any paid sources. If a source charges by data volume, you may be able to reduce it by caching calls and reusing duplicate data. You could also investigate different providers and see if any offer cheaper ways to get the information you require.

The more significant cost, though, is that of maintaining your data pipelines. Complex integrations and data governance requirements means that you need to pay highly skilled resources to ensure your data is integrated and managed correctly.

How Apperate Can Help

Apperate provides a single place for you to collect, transform, and collate financial data for external access. It makes it easy to combine and manage your data to give your app a single source of truth.

Apperate lets you create custom calls that pull data into a common store, or you could use its ready-to-go data. Once the data is captured, you can perform transformations using SQL or Apperate's low code interface before using the results as the source for views. When you create a view, Apperate creates custom API calls that your own apps can use to access the data.

Let's see how it works in practice.

Getting Started

It's very straightforward to get started with Apperate. You need an IEX Cloud/Apperate account with the appropriate data bundle; accounts with services you want to source data from, such as Bloomberg or Yahoo Finance; and of course, the website or app for which you need the data and that will consume the data you generate with Apperate.

To set up your project in Apperate, you need to create a workspace and generate the API keys you'll use for data access.

Your workspace name is used to generate your domain, which is used in the URLs you call when accessing data sets. On the **overview** page, you'll see the public and private API tokens used to secure your API requests.

Pulling in Data from Multiple Providers

Next, you need to get some data. This can be sourced from a URL, an S3 bucket, or a local file.

For this example, let's assume you're using only public sources of information, such as Bloomberg, Yahoo, or XE.com. You can pull data in via an API call to these services.

IEX Cloud also provides a core API you can access using the token you just created. All kinds of financial data are available there, and it's easy to browse through them. You also get lists of all the available attributes you can access.

For example, to get Apple's share price, you can call the following:

``` API Call

https://api.iex.cloud/v1/data/core/quote/aapl?token=YOUR_TOKEN

```

You can call other financial APIs, such as Yahoo or Bloomberg, in the same way. Each data source is then accessible elsewhere in Apperate via SQL.

If you have files containing data, you can make these a data source too. Similarly, AWS buckets can be accessed by providing your bucket name and other credentials.

Cleaning the Data

Once you've got the data into Apperate, you need to translate it into a common format so that you can use it consistently and export it to other services. Data cleaning  is a challenge. You need to factor in formatting, different names for columns, and different ways of handling currencies.

Apperate includes an automated financial identifier resolution that helps you [resolve some of this chaos. While there are various official formats for labeling data, they don't match. Fortunately, Apperate understands them and can organize them for you.

Also, when ingesting data, Apperate can spot invalid records and record them in the logs. You can review these to resolve any problems manually.

Creating and Using a View

Next, let's see how to create a view. For this example, you'll show the value of a stock from different sources.

One way to write your SQL is to find the data set you want to use in the left-hand menu and click it. It will start you off with a **SELECT** statement extracting its data. You can alter that to be more specific or to join it with data from other sources.

You can click the **Run** button in the bottom right to test the query. As with API calls, it's really useful for making changes and testing them quickly. You can check the results of your query to verify whether the statement works as intended.

You can base your SQL views upon any of the various attributes that data sets include. That allows you to compare figures from different providers if you join them. As mentioned before, you could compare values for the same stock from different sources to check their accuracy.

When you're happy with your SQL, click **Create view**.

After your view has been created, you'll see it listed in the left-hand menu along with your data sets. The view now counts as a data source like any other. If you click it, you'll see SQL to pull data out of the view, as follows:

``` SQL

SELECT * FROM APPERATEEXAMPLE.`VIEW_T43QAUVDB` LIMIT 10;

```

Because many data sources change with time, Apperate allows you to specify your own ingestion schedule to control how often updates occur.

Joining Different Data Sources

You can join data from multiple sources and create views by using SQL.

The code snippet below gets data about the same company from different views, in this case data sets imported from Yahoo and Bloomberg:

``` SQL

SELECT c.symbol, c.HIGH_TDY as BHigh, q.daily_high as YHigh

 FROM APPERATEEXAMPLE.`DATASET_BLOOMBERG` c

 JOIN APPERATEEXAMPLE.`DATASET_YAHOO` q

   ON c.symbol = q.symbol;

```

And here's that running as a query in Apperate:

You can use SQL to combine data sets in any way you want. As shown above, you could pull in data from different sources and show the prices alongside each other to compare them. You could also grab specific values to present your own selection of companies.

Conclusion

Combining multiple data sources can help you offer something unique, but there are challenges in doing so. Getting data in different formats into your systems isn't easy. Besides gathering data, you also need to store it and write code to present it in the format you need.

With Apperate, you can easily create data stores and add data from files, websites, and cloud storage. All this can be piped into a common store where you are free to analyze it, draw insights, and present it to your own clients. You can extract the data to any other tool or website via Apperate's dynamically created API calls.

To get started with Apperate, sign up for a free trial and see how easy it is to create a project and pipe in data for your financial website and app.