As a private investor, the sheer amount of information that can be found on the internet is rather daunting. Trying to understand what types of companies or ETFs are available is incredibly challenging, with millions of companies and derivatives available on the market. Sure, the most traded companies and ETFs can quickly be found simply because they are known to the public (for example, Microsoft, Tesla, S&P 500 ETF, or an All-World ETF). However, what else is out there is often unknown.

This is why I created the FinanceDatabase, a database featuring 300,000+ symbols containing Equities, ETFs, Funds, Indices, Currencies, Cryptocurrencies and Money Markets. It allows you to obtain a broad overview of sectors, industries, types of investments and much more, entirely for free. Everything is stored in CSV files that anyone can read and edit, so the database grows and improves through the community.

The aim of this database is explicitly not to provide up-to-date fundamentals or stock data, as those can be obtained with ease (with the help of this database) by using the Finance Toolkit 🛠️. Instead, it gives insights into the products that exist in each country, industry and sector and provides the most essential information about each product. With this information, you can analyze specific areas of the financial world and/or find a product that is hard to find. By utilising both, it is possible to do a fully-fledged competitive analysis with the tickers found from the FinanceDatabase inputted into the FinanceToolkit.

Some key statistics of the database:

1. Installation
2. Functionality
3. Questions & Answers
4. Contributing
5. Contact

Before installation, consider starring the project on GitHub which helps others find the project as well.

To install the FinanceDatabase it simply requires the following:

pip install financedatabase -U

Then within Python use:

import financedatabase as fd

equities = fd.Equities()

No API key is needed. Each asset class is downloaded once and cached locally, see Questions & Answers for how the cache works.

This section is an introduction to the FinanceDatabase. Find with the link below a Jupyter Notebook in which you can see many more examples, including the full output of each query.

A basic example of how to use the FinanceDatabase is shown below. Every code snippet in the sections that follow builds on this same equities instance. Initialization of each asset class is only required once, so save it to a variable to query the database much more quickly.

import financedatabase as fd

# Initialize the Equities database
equities = fd.Equities()

# Select all equities
equities.select()

A portion of the output is shown below. The tables in this section are cut off to five entries and a selection of the columns due to the sheer size of the database.

And below the actively listed equities are counted per sector.

Each asset class has the same three functions: select to filter on the database's categories, search to look for any text in any column and show_options to see which values a category can take. The asset classes are fd.Equities(), fd.ETFs(), fd.Funds(), fd.Indices(), fd.Currencies(), fd.Cryptos() and fd.MoneyMarkets().

Three capabilities cut across all of them:

• Lists of values. Every filter accepts a single value or a list, e.g. country=['Netherlands', 'Belgium'], returning the entries that match any of them.
• only_primary_listing and exclude_delisted. A company is often listed on many exchanges. only_primary_listing=True keeps only its primary listing, and delisted symbols are left out by default (exclude_delisted=False to include them).
• pandas or Polars. Queries run lazily with Polars and return a pandas DataFrame by default, or a Polars DataFrame with as_pandas=False.

With show_options, all possible options are given per column. This is useful as it doesn't require loading the larger data files.

# Show the options of every column of the equities
options = fd.show_options("equities")

# Select the sectors
options["sector"]

This returns the eleven sectors used for equities, which approximate GICS:

['Communication Services', 'Consumer Discretionary', 'Consumer Staples',
 'Energy', 'Financials', 'Health Care', 'Industrials',
 'Information Technology', 'Materials', 'Real Estate', 'Utilities']

Once an asset class is loaded, show_options is also available on the class itself, where it shows the options that remain after filtering. For example, the industries of the financial companies in the Netherlands:

# Show the industries of financial companies in the Netherlands
equities.show_options(
    selection="industry",
    sector="Financials",
    country="Netherlands",
)

Which returns:

['Banks', 'Capital Markets', 'Consumer Finance',
 'Diversified Financial Services', 'Insurance']

And below the number of companies in each of these industries is shown. Each company is counted once, however many exchanges it is listed on: listings that share an ISIN, share class FIGI, website or the first word of their name (such as Aegon and its perpetual bonds) are combined.

The options of every column can be shown this way, including currency, exchange, market, country and market_cap (Mega, Large, Mid, Small, Micro and Nano Cap). Find the Notebook here and the full documentation here.

Given these options, it becomes possible to filter the database on the categories you are interested in. For example, the 'Insurance' companies in the 'United States', one row per company. The sector can be omitted here since the industry already implies 'Financials'.

# Select the primary listings of insurance companies in the United States
equities.select(
    country="United States",
    industry="Insurance",
    only_primary_listing=True,
)

This returns 181 companies, of which a few of the larger ones are shown below.

Without only_primary_listing=True, the same query returns 454 listings on 23 exchanges, because every exchange a company trades on is shown by default. Progressive, for example, also appears as PGV.F (Frankfurt), PGV.SG (Stuttgart), PGR.MX (Mexico) and P1GR34.SA (B3). Primary listings are the symbols without an exchange suffix, which makes the option mostly useful for US companies. For companies elsewhere, filter on the exchange or market instead. The chart below shows how the Dutch insurers' listings are spread over the markets.

For the Netherlands, it makes sense to select the market "Euronext Amsterdam" (exchange "AMS"), here together with the market cap:

# Select the large insurance companies on Euronext Amsterdam
equities.select(
    country="Netherlands",
    industry="Insurance",
    market="Euronext Amsterdam",
    market_cap="Large Cap",
)

This gives the following three companies:

Every filter also accepts a list, so both queries can be combined into one with country=["Netherlands", "United States"] and market=["Euronext Amsterdam", "New York Stock Exchange", "NASDAQ Global Select"]. Equities can be selected on country, sector, industry_group, industry, currency, exchange, mic, market and market_cap. Find the Notebook here and the full documentation here.

If the categorization doesn't lead to the results you are looking for, search filters on any column via a custom string. If the text is found anywhere in the column, the entry is returned. Searches are not case sensitive unless case_sensitive=True is set, index searches the symbol and, just like select, every argument accepts a list.

# Search for robotics or education companies with equipment on the Frankfurt Stock Exchange
equities.search(
    summary=["Robotics", "Education"],
    industry_group="Equipment",
    market="Frankfurt",
    index=".F",
)

This returns 60 instruments listed on the Frankfurt Stock Exchange, in an industry group containing "Equipment" and with "Robotics" or "Education" in their summary. Filtering on index=".F" is an alternative way to find the exchange or market you are looking for.

And below the 60 results are counted per industry.

The search function works on every column of every asset class, including name, isin, cusip and figi. Find the Notebook here and the full documentation here.

All of these functions are also available for the other asset classes. The only difference is the class name and the columns. For example, ETFs use fd.ETFs() and are selected on category_group, category and family instead.

# Initialize the ETFs database
etfs = fd.ETFs()

# Select the Fixed Income ETFs of Vanguard
etfs.select(
    category_group="Fixed Income",
    family="Vanguard Asset Management",
)

For example, see some of the Vanguard bond ETFs listed in Berlin below:

The same applies to search, for example to find the funds that focus on pension plans:

# Initialize the Funds database
funds = fd.Funds()

# Search for funds with "Pension" in their summary
funds.search(summary="Pension")

Which returns 628 funds, of which a few are shown below:

And below all actively listed ETFs are divided over their category groups.

The categories of each asset class can again be found with show_options, e.g. fd.Indices().show_options(selection="category") returns the 42 index categories, from 'Corporate Bonds' and 'Emerging Markets' to 'Small Cap' and 'Value'. Cryptocurrencies are selected on cryptocurrency and currency, currencies on base_currency and quote_currency and money markets on currency and family. Find the Notebook here and the full documentation here.

The FinanceDatabase has a direct integration with the Finance Toolkit, making it possible to do financial analysis on the instruments you've found. Any selection can be loaded into the Finance Toolkit with to_toolkit.

To be able to get started, you need to obtain an API Key from FinancialModelingPrep. This is used to gain access to 30+ years of financial statements, both annually and quarterly. Note that the Free plan is limited to 250 requests each day, 5 years of data, and only features companies listed on US exchanges.

Returning to the three large insurance companies in the Netherlands:

# Select the large insurance companies on Euronext Amsterdam
dutch_insurance_companies = equities.select(
    country="Netherlands",
    industry="Insurance",
    market="Euronext Amsterdam",
    market_cap="Large Cap",
)

# Load them into the Finance Toolkit
toolkit = dutch_insurance_companies.to_toolkit(api_key="FINANCIAL_MODELING_PREP_KEY")

# Obtain historical market data for all tickers
historical_data = toolkit.get_historical_data()

# Select the results for ASR Nederland
historical_data.xs("ASRNL.AS", axis=1, level=1)

For example, a portion of the historical data for ASR Nederland is shown below.

And below the cumulative returns of Aegon, ASR Nederland and NN Group are plotted, including the S&P 500 as benchmark.

Now let's make it more advanced by calculating the profitability ratios for each company:

# Collect all Profitability Ratios for all tickers
profitability_ratios = toolkit.ratios.collect_profitability_ratios()

# Select the results for ASR Nederland
profitability_ratios.loc["ASRNL.AS"]

For example, see some of the profitability ratios of ASR Nederland below.

And below these and other profitability ratios of ASR Nederland, each with its latest value, the change over the period and its trend.

This works for the other asset classes too. For example, Ethereum quoted in BTC, CAD, EUR, GBP and USD can be loaded with fd.Cryptos().select(cryptocurrency="ETH").to_toolkit(api_key=...), after which get_historical_data(period="quarterly") returns its quarterly returns in each currency. This is just a small snippet of what is available within the Finance Toolkit, see the GitHub page of the Finance Toolkit here or the example Notebook here for more information.

This section includes frequently asked questions. If you have any questions that are not answered here, consider creating an Issue or reach out to me via the contact details below.

How is the data obtained?

The data is an aggregation of various publicly available sources. I strictly maintain the rule that all data in this database must be freely accessible to everyone. Data requiring API keys or paid subscriptions is never included. Information that companies charge for is typically owned and maintained by those companies, making public sharing of such data a violation of their Terms of Service (ToS). However, publicly available data can be freely shared (read more about the legality of web scraping here). This database will always remain completely free.

What categorization method is used?

The categorization for Equities is based on a loose approximation of GICS (Global Industry Classification Standard). This database attempts to reflect sectors and industries as accurately as possible through manual curation, without collecting any actual data from MSCI's proprietary sources. The official GICS datasets curated by MSCI remain the most up-to-date, paid solution and were not used in developing any part of this database. All other categorizations in the database are independently developed and can be freely modified.

How can I find out which countries, sectors and/or industries exist within the database without needing to check the database manually?

For this you can use the show_options function, either for an asset class as a whole (fd.show_options("equities")), which doesn't require any data to be loaded, or on a loaded asset class to see the options that remain after filtering. See Exploring the Options for more information.

When I try collect data I notice that not all tickers return output, why is that?

Some tickers are merely holdings of companies and therefore do not really have any data attached to them. Therefore, it makes sense that not all tickers return data. If you are still in doubt, search the ticker on Google to see if there is really no data available. If you can't find anything about the ticker, consider updating the database by visiting the Contributing Guidelines.

How does the database handle changes to companies over time - like symbol/exchange migration, mergers, bankruptcies, or symbols getting reused?

For American exchanges, the database automatically updates every Sunday using data from this repository. This process includes checks for market cap changes and updates asset classifications accordingly. Delisted tickers are intentionally retained for historical research purposes.

While professional financial data services like Bloomberg charge over $25,000 annually for comprehensive market data maintenance, this database relies on community contributions. When companies outside American exchanges undergo changes (migrations, mergers, bankruptcies), we depend on community members to identify and update these entries.

Most companies don't change so rapidly that the database becomes obsolete - major changes like Facebook's rebrand to META are quickly incorporated. Even when companies go bankrupt, their ticker information remains valuable for historical analysis.

If you notice outdated information, please consider contributing through the Contributing Guidelines.

Is the data downloaded every time I use the package?

No. Each dataset is downloaded once and cached in your user cache folder (or the folder set in FINANCEDATABASE_CACHE_DIR). Once a day the package checks for a newer version and only downloads it when it changed; offline, the cached copy is used. Queries run lazily with Polars and return pandas by default, or Polars with as_pandas=False, e.g. equities.select(country="Canada", as_pandas=False).

First off all, thank you for taking the time to contribute (or at least read the Contributing Guidelines)! 🚀

The FinanceDatabase serves the role of providing anyone with any type of financial product categorization entirely for free. To achieve this, it relies on community involvement to add, edit and remove tickers over time. This is made easy enough that anyone, even those with a lack of coding experience, can contribute because of the use of CSV files that can be manually edited with ease.

Below are those that made significant contributions to the project. Thank you!

If you have any questions about the FinanceDatabase or would like to share with me what you have been working on, feel free to reach out to me via:

• Website: https://jeroenbouma.com/
• LinkedIn: https://www.linkedin.com/in/boumajeroen/
• Email: jer.bouma@gmail.com

If you'd like to support my efforts, either help me out via the Contributing Guidelines or Sponsor Me.