P / 04 · Intelligent firewall · Machine learning
ML WAF
An intelligent web application firewall: a LightGBM model sits inline in a reverse proxy, unwraps obfuscated payloads, scores every request value for SQL injection and XSS, and explains each verdict live.
Working demo · Four test layers
Explore repository- Python
- LightGBM
- scikit-learn
- FastAPI
- SQLite
- Docker
The operator console inspecting demo traffic and model decisions. Original project screenshot.
The operator console inspecting demo traffic and model decisions. Original project screenshot.
Image sourceProblem & users
Rule-based WAFs miss encoded or obfuscated payloads, and naive ML WAFs quietly learn the wrong thing: an early version of this model blocked /users/42/profile and let 1' OR 1=1-- through, because its training data taught it vocabulary instead of attacks. The goal was a firewall that recognises the shape of an injection on any site, and can show why it decided.
Implementation
Every request is first normalised: recursive URL decoding, HTML entities, JS escapes, base64 data URIs, Unicode folding, SQL comment stripping and hex or CHAR() literals. It is then split into path and individual parameter values, each turned into character n-grams plus 25 numeric features and classified by LightGBM as benign, SQLi or XSS. A fast scorer bypasses the sklearn pipeline for inline speed with identical scores. The operator console streams decisions over SSE with the decode trace and per-feature contributions, lets operators preview a new threshold against recent traffic, and writes false-positive feedback straight back into the training format.
Engineering decision
Score each value on its own and block on the worst one, so benign context cannot dilute an attack and site vocabulary carries no signal. Train on real web-server traces where the same paths carry both labels. Run in detect mode by default and fail open on errors, with scoring bounded to 64 values and 180ms so a broken model never becomes an outage.
Evidence
On independent datasets the shipped model blocks 96% of SQL injection and 100% of XSS and obfuscated evasions, with 0% false positives on real NASA traffic and under 1% on an unseen site. Retraining on real traces cut false positives 38×. In the Docker demo, sqlmap finds the injection in OWASP Juice Shop directly but is refused through the firewall. Four layers of tests cover the model pipeline, the proxy and control plane, adversarial hardening and endurance under sustained load.
Try it · runs in your browser
What the model sees
The firewall’s real normalisation chain, ported to JavaScript: a request is peeled layer by layer, then split into values that are scored on their own. The LightGBM score needs the trained model, so the signals below are readable hints, not the verdict.
- 1Pick an example or type a URL
- 2Each value is decoded layer by layer
- 3The most suspicious value is highlighted
- Hover ? for details
What’s happeningThe quote is URL-encoded twice, so a filter that decodes once never sees it.
- pathThe URL path is scored as its own unit, separately from every parameter.depth 0How many decoding rounds it took to reach plain text. Normal traffic needs 0 or 1; more is itself suspicious.
/itemno signals - idEach query parameter value is scored alone. Its name is left out, so site vocabulary cannot sway the model.depth 2How many decoding rounds it took to reach plain text. Normal traffic needs 0 or 1; more is itself suspicious. · ← flaggedThe firewall blocks on the most suspicious value, so harmless parameters around it cannot dilute the attack.
1%2527%2520OR%25201%253D1--- decode round 1URL-decoding, HTML entities and JavaScript escapes, repeated until the text stops changing (up to 5 rounds).
1%27%20OR%201%3D1-- - decode round 2URL-decoding, HTML entities and JavaScript escapes, repeated until the text stops changing (up to 5 rounds).
1' OR 1=1-- - lowercaseCase is removed so UNION and union look identical to the model.
1' or 1=1--
signalsReadable pattern hints for this demo. The real decision is a LightGBM score over character n-grams and 25 numeric features.quoteA quote can close a string in a SQL query or an HTML attribute.boolean tautologyA condition like OR 1=1 that is always true, used to bypass WHERE clauses.sql comment-- or # cuts off the rest of the original query. - decode round 1URL-decoding, HTML entities and JavaScript escapes, repeated until the text stops changing (up to 5 rounds).
- sortEach query parameter value is scored alone. Its name is left out, so site vocabulary cannot sway the model.depth 0How many decoding rounds it took to reach plain text. Normal traffic needs 0 or 1; more is itself suspicious.
priceno signals
Current limitations
- Detection covers SQL injection and XSS only, with no cross-request session analysis. Very short payloads such as admin'# can score too low, and values past the 64-unit cap go unscored.
- Errors and timeouts allow traffic by design. The decision cache and console stream are per process.
On GitHub
- Jupyter Notebook
- 0 stars
- Updated