DistributedATS

Distributed ATS

DistributedATS is a FIX Protocol-based alternative trading system that integrates QuickFIX and LiquiBook over FastDDS. This project simplifies the process of communication between multiple FIX gateways and multiple matching engines(CLOB) in real-time. FIX Gateways communicate with clients via FIX and Matching Engines and Middleware(Data Service) components via topic-based DDS IDL Pub/Sub mechanism.

Components

• FIX Gateway communicates with clients via FIX and with Matching Engines and Data Services via DDS. FIX Gateway converts FIX messages into DDS IDL and publishes converted IDLs to Matching Engines or Data Services, it converts DDS IDL messages received from Matching Engines and Data Services into FIX and sends them to FIX clients.

• Matching Engine maintains order-books, publishes market data(conflated), matches orders, and publishes IDL-based Execution Reports. Matching Engine services critical to the order flow DDS messages including NewOrderSingle, OrderCancelRequest, MassOrderCancel, etc.

• Data Service authenticates users, provides reference data, services mass order status requests, and market data snapshots. Data Service services all non-critical to order flow DDS messages including Logon, Logout, MassOrderStatusRequest, MarketDataSnapshot. Data Service can service one or more FIX Gateways and Matching Engines.

Examples

Crypto CLOB/ATS – three matching engines, each handling a subset of instruments.

• Users: CRYPTO_TRADER_1, CRYPTO_TRADER_2, CRYPTO_TRADER_3, CRYPTO_TRADER_4 : Password: TEST
• http://localhost:8080/

services:
  fast_dds_discovery:
    container_name: fast_dds_discovery 
    image: ghcr.io/mkipnis/distributed_ats:latest
    command: >
      bash -c "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/fastdds discovery -q 51000"
    ports:
      - "51000:51000"
    restart: unless-stopped

  distributed_ats:
    container_name: distributed_ats
    image: ghcr.io/mkipnis/dats_crypto_clob:latest
    depends_on:
      - fast_dds_discovery
    command: >
      bash -c "cd /usr/local && source ./dats_env.sh && cd MiscATS && BASEDIR_ATS=`pwd`/CryptoCLOB python3 start_ats.py --ats CryptoCLOB/crypto_ats.json"
    volumes:
      - ./logs_ats:/usr/local/MiscATS/CryptoCLOB/logs
    ports:
      - "15001:15001"
      - "16001:16001"
      - "17001:17001"
    restart: unless-stopped

  fix-ws-proxy:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: fix-ws-proxy
    image: ghcr.io/mkipnis/fix_ws_proxy:latest
    ports:
      - "9002:9002"
    environment:
      LD_LIBRARY_PATH: "/usr/local/lib"
    restart: unless-stopped

  # WebTrader Front-End
  distributed_ats_webtrader:
    container_name: distributed_ats_webtrader
    image: ghcr.io/mkipnis/distributed_ats_webtrader:latest
    volumes:
      - ./webtrader_logs:/var/log/nginx
    ports:
      - "8080:80"
    restart: "no"

US Treasuries CLOB/ATS – one matching engine that handles hundreds of instruments.

• Users: UST_TRADER_1, UST_TRADER_2, UST_TRADER_3, UST_TRADER_4 : Password: TEST
• http://localhost:8080/

version: '2'

services:
  fast_dds_discovery:
    container_name: discovery_service
    image: ghcr.io/mkipnis/distributed_ats:latest
    command: >
      bash -c "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/fastdds discovery -q 51000"
    ports:
      - "51000:51000"
    restart: unless-stopped

  distributed_ats:
    container_name: distributed_ats
    image: ghcr.io/mkipnis/dats_ust_clob:latest 
    depends_on:
      - fast_dds_discovery 
    command: >
      bash -c "cd /usr/local && source ./dats_env.sh && cd MiscATS && BASEDIR_ATS=`pwd`/USTreasuryCLOB python3 start_ats.py --ats USTreasuryCLOB/ust_ats.json"
    volumes:
      - ./logs_ats:/usr/local/MiscATS/USTreasuryCLOB/logs
    ports:
      - "15001:15001"
      - "16001:16001"
    restart: unless-stopped

  fix-ws-proxy:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: fix-ws-proxy
    image: ghcr.io/mkipnis/fix_ws_proxy:latest
    ports:
      - "9002:9002"
    environment:
      LD_LIBRARY_PATH: "/usr/local/lib"
    restart: unless-stopped

  # WebTrader Front-End
  distributed_ats_webtrader:
    container_name: distributed_ats_webtrader
    image: ghcr.io/mkipnis/distributed_ats_webtrader:latest
    volumes:
      - ./webtrader_logs:/var/log/nginx
    ports:
      - "8080:80"
    restart: "no"

Multi Matching Engine ATS, with each component running in a separate container

• Docker Compose - Multi Matching Engine ATS
• Docker Compose - WebTrader

Dependencies

Dependency	Component
FastDDS	DDS/Middleware
QuickFIX	FIXGateway
LiquiBook	MatchingEngine
SQLite3	DataService

Sample Exchanges

MultiMatchingEngineATS - Two Matching Engines, three FIX Gateways, and two Data Services. The first matching engine consumes and processes orders from the instrument group MARKET_Y, the second from the instrument MARKET_Z

USTreasuryCLOB - A single Matching Engine, three FIX Gateways, and two Data Services. The matching engine consumes and processes orders from the instrument group MARKET_UST that covers 300+ instruments (OrderBooks)

CryptoCLOB - Three Matching Engines, three FIX Gateways, and two Data Services. The first matching engine consumes and processes orders from the instrument group BTC_MARKET, the second from the instrument ETH_MARKET, and the third from OTHER_COIN_MARKET

Other Utilities

• LatencyTest - A NewOrderSingle path latency measurement utility. LatencyTest correlates critical to the order flow messages across DistributedATS hops by serving as a FIX and DDS IDL client.
• SimulatorATS - A random activity simulator that consumes market data, aggress on it with arbitrary quantities, cancels unfilled orders and sends mass cancel requests.

Sample Clients

• Python3/QuickFIX
• ReactJS and SpringBoot-QuickFIX/J
• C++/FIX/JSON/ReactJS
• Go/Rest

High Level Data Flow Overview

Overview

FIX Messages, Topics, Sources and Destinations

FIX Message	DDS Topic	Source	Destination	Content Description
Login(A)	LOGON_TOPIC	FIXGateway	DataService	Converted to IDL FIX Login
Login(A)	LOGON_TOPIC	DataService	FIXGateway	Successfully Authenticated FIX Session by a Data Service
Logout(5)	LOGOUT_TOPIC	DataService	FIXGateway	Unsuccessfully Authenticate FIX Session by Data Service
NewOrderSingle(D)	NEW_ORDER_SINGLE_TOPIC	FIXGateway	MatchingEngine	New order single
ExecutionReport(8)	EXECUTION_REPORT_TOPIC	MatchingEngine	FIXGateway	Execution Report to be sent to a client
ExecutionReport(8)	EXECUTION_REPORT_TOPIC	MatchingEngine	DataService	Execution Report to be store to service Order Mass Status Requests
OrderCancel Request(F)	ORDER_CANCEL_ REQUEST_TOPIC	FIXGateway	MatchingEngine	Converted to IDL Cancel Request from FIX Client
OrderCancel Reject(9)	ORDER_CANCEL_REJECT_ TOPIC	MatchingEngine	FIXGateway	IDL Cancel Reject, if order can't be cancelled
OrderCancelReplace Request(G)	ORDER_CANCEL_ REPLACE_REQUEST_TOPIC_NAME	FIXGateway	MatchingEngine	Converted to IDL Cancel Replace Request from FIX Client
MassCancel Request(q)	ORDER_MASS_CANCEL_ REQUEST_TOPIC	FIXGateway	MatchingEngines	Mass Cancel requested by FIX Client or FIX Client Logout/Disconnect
MassCancel Report(r)	ORDER_MASS_CANCEL_ REPORT_TOPIC	MatchingEngines	FIXGateway	Results of Mass Cancel Request
MarketData Request(V)	MARKET_DATA_REQUEST _TOPIC	FIXGateway	DataService	Request for the current state of Order Book(Top 5 levels)
MarketDataSnapshotFull Refresh(W)	MARKET_DATA_SNAPSHOT _FULL_REFRESH_TOPIC	DataService	FIXGateway	Current Market Data Snapshot(Top 5 levels)
MarketDataSnapshotFull Refresh(W)	MARKET_DATA_SNAPSHOT _FULL_REFRESH_TOPIC	Data Service	MatchingEngine	OpeningPrices
MarketDataIncremental Refresh(X)	MARKET_DATA_SNAPSHOT _FULL_REFRESH_TOPIC	MatchingEngine	FIXGateway	Incremental Market Data Request(Top 5 levels)
MarketDataIncremental Refresh(X)	MARKET_DATA_SNAPSHOT _FULL_REFRESH_TOPIC	MatchingEngine	FIXGateway	Incremental Market Data Request(Top 5 levels)
Security ListRequest(x)	SECURITY_LIST_REQUEST _TOPIC	FIXGateway	DataService	FIX Client initiated Security List Request
SecurityList(y)	SECURITY_LIST_TOPIC	DataService	FIXGateway	Security list reply for FIX Client Request
SecuritList Request(x)	SECURITY_LIST_REQUEST _TOPIC	MatchingEngine	DataService	Security List Request for Order Book setup
SecurityList(y)	SECURITY_LIST_TOPIC	DataService	MatchingEngine	Security List for OrderBook Setup
OrderMassStatus Request(AF)	ORDER_MASS_STATUS_ REQUEST_TOPIC	FIXGateway	DataService	Initiated by FIX Client Mass Status Request

Autogeneration of IDL from QuickFIX XML

GenTools is a utility that generates DDS IDL, FIX to IDL, and IDL to FIX adapters and IDL logger helper classes from QuickFIX's XML data dictionary.