Probability-Based Adaptive Detection

🙏🏻 PBAD (Probability-Based Adaptive Detection): adaptive control tool for outliers || novelty detection, made for worst case data & processes, for the highest time complexity O(n^2) compared with the alternatives (would be explained in a sec). Thresholds are completely data driven and axiomatic, no need in provided hyperparameters, are not learned or optimized. The method accepts multiple weights, e.g. both temporal and volatility weights.

Method briefly explained (I can go deeper if any1 asks explicitly):

1. Performs weighted KDE on initial input data, finds KDE global maximum (mode), creates new “residuals” dataset by centering initial data around this value;
   
2. Performs weighted KDE on residuals, uses sigmoid based probability mass targets with increasing probability coverage to construct a set of non-disjoint High Density Intervals (also called HDR, HPD in Bayesian terms);
   
3. Uses these intervals to calculate analogs of centralized & standardized moments;
   
4. Uses these ^^ moments to construct a set of control thresholds. The scheme used in PBAD is not only based on a central threshold, or on neighboring ones, it utilizes all previous thresholds, gaining more information.
   

...

The most important part is to understand whether you really need PBAD. Because even tho it seems to be the best one given highest algocomplexity, irl it would work worse in cases when it’s not required by your data.

Here’s the menu (aka taxonomy omg) of methods you can use that would let you make the right choice:

Moment-Based Adaptive Detection (MBAD):

• Norm: L2
  
• Time complexity: original O(n), successfully reduced to O(1) in online version
  
• Use case: default, general purpose
  
• Based on: method of moments (powers of residuals from mean)
  
• Thresholds architecture: centralized
  

Quantile-Based Adaptive Detection (QBAD):

• Norm: L1
  
• Time complexity: O(nlogn)
  
• Use case: either bad data Or process instability
  
• Based on: quantile moments (dyadic percentiles of residuals from median)
  
• Thresholds architecture: chained/recursive/sequential
  

Probability-Based Adaptive Detection (PBAD):

• Norm: L0
  
• Time complexity: O(n^2)
  
• Use case: both bad data And process instability
  
• Based on: probability moments (target probability masses of residuals from KDE mode)
  
• Thresholds architecture: decentralized (for lack of a better name xd, the idea is that these thresholds gain information from the all other threshold and are Not exclusively based on the central or neighboring thresholds)
  

...

Examples of true use cases:

^^ an appropriate financial instrument to use PBAD


^^ and another one

...

Additional details about how to use it:

• Keep the student5 kernel, it’s the best you can do. I added others mostly for comparisons and if you want to use the tool Not for its primary purpose (on a fine data)
  
• “Calculate for N bars” and “Starting at bar N” options allow to reduce calculation period only on the N number of last bars or next bars from a chosen one. It's vital, because calculations here are heavy
  
• Keep plotting offset at 1 (allows to visually compare current bar with the previous threshold values). This is the way it should be done on price data.
  
• HLC3 is the optimal source input, unless you want to use your own better one point estimate of each datapoint (in the best case done by using PBAD itself on OHLC+ values).
  

In essence it should be used just like MBAD or QBAD, fade/push extensions and limit, fade/push/skip deviations & basis, or other strategies of your. Again, the only reason for 3 methods to exist is to be chosen for according data characteristics.


Btw:

• This is the initial version, I don’t consider it perfected tbh, even tho it works as expected, however this method is very situational anyways.
  
• In this script KDE function is modified to ensure the outcoming probabilities Do sum up to 1. I didn’t do this normalization in Weighted KDE Mode script, but there it’s not required since we just need a KDE global max.
  

see ya

∞