Data integrity

— Last updated: 23 Sep 2019 12:13 PM (GMT) —

What is this page?

This section presents the most updated quality tests for our cryptocurrency bar generator engine. Additionally, it summarizes the different types of bars and steps offered (each step has a different meaning depending on the bar type. More info in the API docs) .

When data is updated by pulling raw trades from the exchanges (for now only Bitfinex), bars are automatically built and data quality is evaluated based on volume conservation. Essentially, we check that the sum of all trades volume is equal to the sum of all bars volume. If the difference observed is low, then we have a guarantee that the bars have been built correctly and have included all the existing trades provided by the exchange, no more and no less. However, note that small errors can be observed due to floating point operations.

We set our error tolerance to 0.01 — any error higher than this threshold should raise our concern and the affected bars should be revised and possibly regenerated from scratch using the raw trade data.

Time Bars

1m5m15m30m1h4h12h1d
XMRUSD 1.12e-08 1.12e-08 2.24e-08 1.12e-08 1.49e-08 1.49e-08 1.49e-08 1.49e-08
ZECUSD 1.30e-08 9.31e-09 1.12e-08 1.49e-08 1.12e-08 9.31e-09 9.31e-09 1.12e-08
XRPUSD 9.16e-04 9.46e-04 9.16e-04 9.38e-04 9.38e-04 9.46e-04 9.38e-04 9.38e-04
IOTUSD 4.79e-04 4.75e-04 4.77e-04 4.77e-04 4.75e-04 4.75e-04 4.75e-04 4.75e-04
ETHBTC 7.45e-08 1.04e-07 7.45e-08 1.19e-07 1.04e-07 8.94e-08 7.45e-08 7.45e-08
LTCUSD 1.23e-05 1.23e-05 1.23e-05 1.23e-05 1.22e-05 1.23e-05 1.23e-05 1.23e-05
XLMUSD 0.00e+00 1.19e-07 1.19e-07 0.00e+00 2.38e-07 1.19e-07 2.38e-07 1.19e-07
NEOUSD 4.47e-07 5.36e-07 5.66e-07 5.36e-07 5.36e-07 5.07e-07 5.07e-07 5.07e-07
XRPBTC 1.18e-04 1.26e-04 1.28e-04 1.30e-04 1.28e-04 1.30e-04 1.30e-04 1.30e-04
ETHUSD 1.90e-05 1.91e-05 1.90e-05 1.90e-05 1.91e-05 1.91e-05 1.91e-05 1.91e-05
OMGUSD 6.40e-05 6.40e-05 6.39e-05 6.39e-05 6.39e-05 6.39e-05 6.39e-05 6.39e-05
DSHUSD 9.13e-08 9.13e-08 8.75e-08 8.94e-08 8.57e-08 8.75e-08 8.94e-08 8.94e-08
ZILUSD 1.49e-08 0.00e+00 1.49e-08 0.00e+00 0.00e+00 0.00e+00 0.00e+00 0.00e+00
ZRXUSD 1.13e-06 1.19e-06 1.07e-06 1.01e-06 1.01e-06 1.07e-06 1.13e-06 1.13e-06
EOSUSD 3.62e-05 3.72e-05 3.53e-05 3.72e-05 3.62e-05 3.53e-05 3.53e-05 3.72e-05
BTCUSD 2.09e-07 2.01e-07 2.24e-07 2.09e-07 2.24e-07 2.24e-07 2.16e-07 2.16e-07

Tick Bars

501002005001000
XMRUSD 3.73e-09 7.45e-09 1.86e-08 3.73e-09 1.12e-08
ZECUSD 1.86e-09 5.59e-09 1.86e-08 1.86e-09 9.31e-09
XRPUSD 9.23e-04 9.16e-04 9.16e-04 9.08e-04 9.31e-04
IOTUSD 4.73e-04 4.77e-04 4.77e-04 4.71e-04 4.79e-04
ETHBTC 7.45e-08 8.94e-08 8.94e-08 2.98e-08 1.04e-07
LTCUSD 1.22e-05 1.21e-05 1.19e-05 1.22e-05 1.20e-05
XLMUSD 3.58e-07 3.58e-07 4.77e-07 5.96e-07 2.38e-07
NEOUSD 5.36e-07 5.07e-07 5.07e-07 5.36e-07 5.07e-07
XRPBTC 1.28e-04 1.28e-04 1.30e-04 1.28e-04 1.30e-04
ETHUSD 1.90e-05 1.90e-05 1.90e-05 1.89e-05 1.87e-05
OMGUSD 6.39e-05 6.35e-05 6.35e-05 6.38e-05 6.37e-05
DSHUSD 9.13e-08 9.69e-08 9.31e-08 9.13e-08 9.13e-08
ZILUSD 0.00e+00 1.49e-08 1.49e-08 0.00e+00 1.49e-08
ZRXUSD 1.07e-06 1.31e-06 9.54e-07 1.25e-06 1.19e-06
EOSUSD 4.48e-05 3.62e-05 3.91e-05 3.91e-05 3.91e-05
BTCUSD 2.76e-07 2.53e-07 2.53e-07 2.16e-07 3.05e-07

Imbalance bars

Tick imbalance bars (TIB)Volume imbalance bars (VIB)Dollar imbalance bars (DIB)
100200400100200400100200400
XMRUSD 7.45e-09 1.12e-08 1.86e-08 1.86e-08 2.61e-08 1.86e-08 2.24e-08 1.86e-08 1.86e-08
ZECUSD 9.31e-09 1.30e-08 1.49e-08 5.59e-09 3.73e-09 7.45e-09 3.73e-09 3.73e-09 1.86e-09
XRPUSD 9.23e-04 9.23e-04 9.16e-04 9.23e-04 9.46e-04 9.16e-04 9.31e-04 9.31e-04 9.46e-04
IOTUSD 4.75e-04 4.77e-04 4.75e-04 4.79e-04 4.71e-04 4.69e-04 4.79e-04 4.79e-04 4.73e-04
ETHBTC 1.49e-07 1.34e-07 1.04e-07 1.49e-07 1.19e-07 1.19e-07 1.19e-07 1.19e-07 1.04e-07
LTCUSD 1.21e-05 1.19e-05 1.17e-05 1.23e-05 1.22e-05 1.21e-05 1.20e-05 1.20e-05 1.20e-05
XLMUSD 1.19e-07 2.38e-07 1.19e-07 3.58e-07 2.38e-07 2.38e-07 3.58e-07 3.58e-07 2.38e-07
NEOUSD 4.77e-07 4.77e-07 4.77e-07 3.58e-07 4.47e-07 3.87e-07 4.17e-07 3.28e-07 4.17e-07
XRPBTC 1.30e-04 1.30e-04 1.26e-04 1.30e-04 1.30e-04 1.26e-04 1.28e-04 1.30e-04 1.32e-04
ETHUSD 1.90e-05 1.90e-05 1.90e-05 1.91e-05 1.90e-05 1.90e-05 1.90e-05 1.89e-05 1.90e-05
OMGUSD 6.37e-05 6.38e-05 6.39e-05 6.40e-05 6.37e-05 6.40e-05 6.37e-05 6.38e-05 6.38e-05
DSHUSD 9.13e-08 9.13e-08 9.13e-08 8.38e-08 8.94e-08 9.13e-08 8.75e-08 8.57e-08 9.87e-08
ZILUSD 2.98e-08 1.49e-08 0.00e+00 1.49e-08 0.00e+00 1.49e-08 1.49e-08 2.98e-08 1.49e-08
ZRXUSD 1.07e-06 8.94e-07 1.25e-06 1.13e-06 1.13e-06 1.01e-06 1.01e-06 1.31e-06 2.38e-07
EOSUSD 4.10e-05 4.01e-05 3.72e-05 3.91e-05 4.29e-05 4.20e-05 4.10e-05 3.91e-05 4.20e-05
BTCUSD 2.98e-07 2.31e-07 2.01e-07 2.76e-07 2.31e-07 2.09e-07 2.61e-07 1.94e-07 2.16e-07

Volume bars

XMRUSD
201.49e-08
1003.73e-09
3002.24e-08
6002.98e-08
10001.86e-08
40003.73e-09
100002.24e-08
200001.86e-08
ZECUSD
101.86e-09
501.86e-09
1009.31e-09
3001.86e-09
6001.12e-08
20005.59e-09
70001.30e-08
100009.31e-09
XRPUSD
300009.23e-04
1000009.16e-04
5000009.00e-04
10000009.31e-04
20000009.38e-04
80000009.38e-04
200000009.31e-04
500000009.31e-04
IOTUSD
100004.63e-04
600004.83e-04
1000004.69e-04
3000004.73e-04
7000004.73e-04
30000004.73e-04
90000004.73e-04
100000004.79e-04
ETHBTC
401.34e-07
2001.04e-07
6008.94e-08
10001.19e-07
20001.34e-07
100001.34e-07
300001.04e-07
600001.19e-07
LTCUSD
801.20e-05
4001.23e-05
10001.22e-05
20001.20e-05
40001.18e-05
100001.19e-05
500001.18e-05
1000001.23e-05
XLMUSD
9000.00e+00
40000.00e+00
100002.38e-07
200002.38e-07
500003.58e-07
2000000.00e+00
6000004.77e-07
10000001.19e-07
NEOUSD
2005.07e-07
10004.77e-07
30005.07e-07
60005.36e-07
100004.77e-07
500004.77e-07
1000005.07e-07
3000004.77e-07
XRPBTC
80001.37e-04
400001.26e-04
1000001.30e-04
2000001.24e-04
4000001.28e-04
10000001.28e-04
50000001.26e-04
100000001.28e-04
ETHUSD
1001.88e-05
6001.90e-05
10001.88e-05
30001.88e-05
70001.90e-05
300001.91e-05
900001.90e-05
1000001.92e-05
OMGUSD
5006.38e-05
20006.38e-05
80006.39e-05
100006.41e-05
300006.37e-05
1000006.37e-05
4000006.39e-05
8000006.40e-05
DSHUSD
108.75e-08
408.75e-08
1008.75e-08
2008.94e-08
5008.57e-08
20009.31e-08
60008.75e-08
100009.50e-08
ZILUSD
1001.49e-08
5000.00e+00
10001.49e-08
30001.49e-08
60002.98e-08
200000.00e+00
800000.00e+00
1000005.96e-08
ZRXUSD
4001.01e-06
20001.01e-06
70001.13e-06
100009.54e-07
200009.54e-07
1000001.25e-06
3000009.54e-07
7000001.01e-06
EOSUSD
40004.20e-05
200004.39e-05
700004.20e-05
1000004.10e-05
2000003.72e-05
10000003.62e-05
30000003.81e-05
70000004.20e-05
BTCUSD
102.61e-07
802.46e-07
2002.09e-07
5002.68e-07
10002.31e-07
40002.46e-07
100002.68e-07
200002.24e-07

Dollar bars

XMRUSD
10002.24e-08
20002.61e-08
100001.49e-08
300002.98e-08
600002.24e-08
1000001.49e-08
5000002.61e-08
10000002.98e-08
30000001.86e-08
ZECUSD
10003.73e-09
20001.12e-08
100001.12e-08
300003.73e-09
600001.30e-08
1000001.30e-08
5000001.86e-09
10000001.30e-08
30000001.12e-08
XRPUSD
90008.93e-04
100001.01e-03
900009.16e-04
2000009.38e-04
5000009.23e-04
10000008.85e-04
40000009.16e-04
100000009.23e-04
200000009.23e-04
IOTUSD
60004.35e-04
100004.54e-04
600004.84e-04
1000004.79e-04
3000004.73e-04
7000004.77e-04
30000004.79e-04
90000004.79e-04
100000004.79e-04
LTCUSD
10001.19e-05
20001.20e-05
100001.18e-05
300001.20e-05
700001.18e-05
1000001.22e-05
6000001.22e-05
10000001.20e-05
30000001.19e-05
XLMUSD
902.38e-07
1002.38e-07
9002.38e-07
20001.19e-07
50003.58e-07
100001.19e-07
400003.58e-07
1000001.19e-07
2000001.19e-07
NEOUSD
40005.07e-07
90004.47e-07
400004.77e-07
1000004.77e-07
2000004.77e-07
5000005.36e-07
20000004.47e-07
60000005.36e-07
100000004.47e-07
ETHUSD
100001.90e-05
300001.88e-05
1000001.91e-05
4000001.91e-05
9000001.90e-05
10000001.91e-05
70000001.90e-05
200000001.89e-05
400000001.90e-05
OMGUSD
20006.32e-05
40006.40e-05
200006.40e-05
600006.37e-05
1000006.39e-05
2000006.38e-05
10000006.37e-05
30000006.38e-05
60000006.38e-05
DSHUSD
10009.50e-08
20007.82e-08
100009.13e-08
400009.69e-08
800009.31e-08
1000009.31e-08
6000008.75e-08
10000009.31e-08
30000008.94e-08
ZRXUSD
1004.77e-07
3003.58e-07
10001.07e-06
50001.31e-06
100001.13e-06
200008.34e-07
900001.01e-06
2000001.01e-06
5000001.13e-06
EOSUSD
100003.62e-05
200004.10e-05
1000004.01e-05
4000004.10e-05
8000003.72e-05
10000003.81e-05
60000003.91e-05
200000004.01e-05
400000004.01e-05
BTCUSD
100002.38e-07
300001.94e-07
1000002.61e-07
5000002.24e-07
10000002.38e-07
20000002.01e-07
90000002.38e-07
200000001.71e-07
500000002.31e-07
ZILUSD
101.49e-08
402.98e-08
900.00e+00
1002.98e-08
7001.49e-08
20004.47e-08
40001.49e-08