burterm/ARCHITECTURE.md

# burterm Architecture

Русская версия: [ARCHITECTURE.ru.md](ARCHITECTURE.ru.md).
Design history, bugs found, and reliability caveats (Russian only): [NOTES.md](NOTES.md).

## Overview

burterm is a terminal tool for web pentesting, bug bounty, and CTF work:
a single static Go binary, 13 TUI tabs, and a headless CLI mode running on
the same engine. Builds without cgo.

```
cmd/burterm/          entry point: CLI flags or TUI launch
internal/
  engine/                 Intruder core: markers, attacks, filters
  payload/                 payload generators
  cli/                      flag and payload-spec parsing
  proxy/                     MITM proxy
  spider/                     crawler
  sitemap/                     scope rules, site tree
  scanner/                      active/passive vulnerability checks
  fuzzer/                        blackbox binary fuzzer
  project/                        session storage (SQLite+FTS5)
  decoder/ comparer/ sequencer/     utilities
  tui/                                interface (bubbletea)
```

## General principles

- **Async I/O via `tea.Cmd`/`tea.Msg`.** The TUI follows the Elm
  architecture: `Update(msg) (Model, Cmd)`. Any side effect (network,
  disk, timer) is a `tea.Cmd` — a zero-argument function returning a
  `tea.Msg`, run by the framework in its own goroutine. `Update` never
  blocks on I/O.
- **Streaming operations go through channels.** Intruder attacks,
  Scanner, Spider, Fuzzer all stream results via `<-chan T`. Shared
  pattern: a producer goroutine writes via `select` with `<-ctx.Done()`;
  the TUI reads via `waitForXxx(ch) tea.Cmd`, which reissues itself after
  every message.
- **Values are snapshotted out of the model before a `tea.Cmd` runs in the
  background** — the closure never holds a reference to a model that may
  have already changed by the time the command executes.

## internal/engine — Intruder core

- `§markers§` in a raw HTTP request mark payload insertion points.
  `ParseMarkers` extracts them into `[]InsertionPoint{Start, End}` (byte
  offsets into the cleaned string) and returns marker-free text.
  `InsertBeforeMarker` inserts arbitrary text right before the Nth
  marker — used by recursion (see below).
- `BuildRequest` substitutes payloads and parses the result into
  `*http.Request` via `http.ReadRequest`.
- Four attack modes: `RunSniper`, `RunBattering`, `RunPitchfork`,
  `RunClusterBomb` — a shared worker pool reading jobs from a channel.
  Pitchfork/ClusterBomb use `drainGenerator`, which reads a generator
  fully into memory before starting.
- `EstimateTotal` computes the expected request count from generators'
  `Len()` before the attack starts — powers the TUI's progress indicator.
- `ParseStatusFilter`/`ParseLengthFilter`/`ParseWordFilter`/
  `ParseLineFilter` share a common operator syntax (`>N`, `<N`, `N-M`,
  exact number) through one internal parser, `parseIntRangeFilter`.
- `New(cfg)` configures `http.Transport` with `MaxIdleConnsPerHost:
  cfg.Concurrency` for connection reuse under concurrent requests to the
  same host.

## internal/payload — payload generators

The `Generator` interface (`Next`/`Reset`/`Len`). Implementations:
`WordlistGenerator`, `NumericRangeGenerator`, `BruteForceGenerator`
(alphabet enumeration up to a given length), `ExtensionGenerator` (wraps
any generator, appending extensions to each word —
`spec|ext:.php,.html`). `presets.go` holds built-in presets (numbers,
letters, common usernames/passwords, SQLi/XSS/path-traversal probes).
`smart.go` is the heuristic that suggests a preset based on the parameter
name near a marker.

## internal/cli — CLI flags

`ParsePayloadSpec` parses strings like `wordlist:<path>` /
`range:<from>:<to>:<step>` / `preset:<name>` / `brute:<alphabet>:<min>:<max>`,
with an optional `|ext:...` suffix. Used by both the headless mode
(`cmd/burterm`) and the TUI — a single shared grammar.

## internal/proxy — MITM proxy

- `Server` is a forward proxy doing MITM via a self-owned CA (`ca.go`,
  generated under `~/.burterm` on first run), a `CONNECT` tunnel with
  `tls.Server`, and on-the-fly leaf certificate issuance keyed by SNI.
- Each transaction is an `Entry{Raw, StatusCode, Length, Headers,
  ResponseBody, ...}` in a thread-safe `Store` (`sync.Mutex`,
  subscription to new entries via a channel).
- `Server.SetScope(*sitemap.Scope)` under a `sync.RWMutex` — if set,
  `capture`/`captureErr` skip storing entries outside scope. The proxy
  still serves traffic regardless of scope.
- `session.go`: `ExtractSessionHeaders(entries, host)` pulls
  `Cookie`/`Authorization` out of the history for a specific host
  (parsed from `Entry.Raw`).

## internal/spider — crawler

`Spider.Crawl` does a BFS walk bounded by `scope *url.URL` and
`maxDepth`. `extractLinksAndForms` parses HTML, pulling out links and
form descriptions. GET forms (`Method=="GET"`) are automatically
converted into URLs with `test` placeholders per field and queued
(`buildGetFormURL`); POST forms are skipped. `SetHeaders` sets headers
applied to every crawl request — used to carry session
Cookie/Authorization from Proxy.

## internal/sitemap — scope and site tree

`Scope` is an ordered list of include/exclude regex rules
(`Add`/`RemoveAt`/`Matches`/`Rules`). `Tree`/`Node`/`Endpoint` is a site
path tree, built from a stream of URLs via `Build`, exportable to JSON.

## internal/scanner — vulnerability checks

- `Check` is the active-check interface (`Name`/`Probes`/`Analyze`),
  substituting payloads into one insertion point via the same
  `engine.BuildRequest` Intruder uses. Implementations: SQLi, XSS
  (reflected, with a unique canary), SSTI, LFI, RCE (time-based), SSRF,
  XXE.
- `Scanner.Run` returns a channel of `Event{Kind:
  EventProgress|EventFinding}` — progress and findings in one stream.
- `CheckCORS` is a one-off check via the `Origin` header (doesn't fit
  the `Check` interface since it doesn't operate through a marker).
- `DecodeJWT`/`AlgNoneVariant` parse a JWT and generate an unsigned
  variant for bypass testing.

## internal/fuzzer — binary fuzzer

A blackbox mutation fuzzer for an arbitrary executable (CTF pwn).
`Config{Target, Args, UseStdin, Timeout, Workers, CrashDir}`. Mutations
(`mutate.go`): bit-flip, byte-flip, arithmetic, interesting values,
block delete/duplicate, splice, `havoc` (several mutations chained in
one pass). Crashes are detected via the process's exit signal
(`SIGSEGV`/`SIGABRT`/`SIGFPE`/`SIGILL`/`SIGBUS`) through
`syscall.WaitStatus`. `Run` returns a channel of `Event{Kind:
EventProgress|EventCrash}`; each worker is an independent goroutine with
its own `rand.Source`.

## internal/project — session storage

SQLite (`modernc.org/sqlite`, no cgo) with an FTS5 index in
external-content mode (text stored once in `requests`, `requests_fts`
holds only the inverted index), kept in sync via
`INSERT`/`UPDATE`/`DELETE` triggers. `Store.Open` sets
`db.SetMaxOpenConns(1)`. `ParseQuery` is a search mini-language
(`host:… method:… status:… color:… text`) — recognized `key:value`
tokens go into structured SQL, everything else into an FTS5 `MATCH`.
`SaveMeta`/`LoadMeta` store arbitrary key-value pairs (used for Target's
scope rules). `SaveLastPath`/`LoadLastPath` keep the path of the last
opened project outside the database itself
(`~/.burterm/last-project`).

## internal/decoder, comparer, sequencer

`decoder` — URL/Base64/HTML/Hex/MD5 codecs plus an auto-detect chain.
`comparer` — token-level LCS diff. `sequencer` — FIPS 140-2 statistical
tests (monobit/poker/runs/long-run) on 20000-bit blocks.

## internal/tui — interface

- `Model` (`model.go`) is the root model; the `active view` field
  determines the visible tab. `Update` delegates unhandled messages to
  the active tab.
- Each tab has its own `xxxModel` in a separate file
  (`requestbuilder.go`, `results.go`, `proxyview.go`, `repeater.go`,
  `spiderview.go`, `targetview.go`, `decoderview.go`, `comparerview.go`,
  `sequencerview.go`, `dashboardview.go`, `scannerview.go`,
  `projectview.go`, `fuzzerview.go`), with
  `Update(tea.Msg) (xxxModel, tea.Cmd)` and `View() string` methods.
- `theme.go` — the color palette, `panel()` (a bordered box with a title,
  via `lipgloss.RoundedBorder`), `renderTabs()` (the tab bar with the
  active tab filled in), `renderLogo()`.
- `results.go` caches compiled filter predicates (`recompileFilters`),
  accumulates filtered output into a `strings.Builder` incrementally
  (`AddResult`), and renders into the `viewport` on a throttled 300ms
  timer (`resultsTickCmd`), only for the attack currently visible.
- `attacks []*attackRun` is a list of independent concurrent attacks,
  each with its own `id`/channel/`cancel`/`resultsModel`.
- `recursionMeta` holds the parameters for Intruder's auto-recursion
  into discovered directories: request template, insertion point,
  payload specs, depth, status predicate.
- Messages follow the `xxxStartedMsg`/`xxxResultMsg`/`xxxDoneMsg`/
  `xxxErrMsg` pattern for every long-running operation.

## On-disk storage

Everything lives under `~/.burterm/`: `ca.pem`/`ca-key.pem` (the MITM
CA), `last-project` (path of the last project), `fuzz-crashes/<unix>/`
(fuzzer crash inputs), `sitemap-<unix>.json` (site map export).

## Running modes

The TUI (no arguments) and the headless CLI (`-request`/`-payload`/
`-mode` etc., see `internal/cli/flags.go`) both run on the same
`internal/engine` — CI/scripting scenarios don't duplicate attack logic.