"Security by obscurity" is the implicit design assumption of many proprietary protocols: nobody knows the format, so nobody can attack it. In practice, an experienced reverse engineer reconstructs message structure, command semantics and state transitions from captured traffic in days, sometimes hours. After that, the attacker knows the protocol as well as the engineering team, and often better than the documentation describes it.
What remains after reconstruction is the actual security: authentication decisions, freshness enforcement, privilege boundaries and parser robustness. These are exactly the components that receive the least design attention in proprietary protocols, because the team's threat model assumed the format would stay secret.
The most common post-reconstruction failures are structural: length fields trusted before validation, parsers running in privileged contexts, state machines that expose administrative commands in early states, and authentication that checks presence rather than identity. None of these are visible in a pentest that stops at the network boundary.
Protocol security depends on several technical assumptions.
- Message structure How quickly does the format yield to systematic analysis, and what does structure alone reveal?
- Authentication Does the protocol authenticate identity and bind the correct peer, or merely detect presence?
- Freshness Are replay attacks prevented on state-changing messages through nonces, counters or timestamps?
- State machine Which commands are reachable in which states, and do error or recovery states skip authorization?
- Parser robustness How do parsers handle malformed, truncated and adversarial input, especially in privileged components?
- Privilege boundaries Which protocol messages reach privileged operations directly, and what separates them?
- Key handling How are session keys established, and what does protocol-state manipulation do to them?
The attacker read the manual you never wrote.
After reconstructing the protocol from captured traffic, the attacker maps the command set and discovers that the parser runs in a privileged firmware service. A malformed length field reaches a memory operation before validation.
The protocol was never documented because it was internal. The attacker documented it, with better coverage of the error states than the engineering team ever tested.
Reconstruct first, then attack as an informed adversary.
We reverse engineer the protocol from captures and live interaction, reconstruct the state model, and identify where the protocol makes trust decisions. Then we test it as the informed attacker the design must ultimately survive: mutation, replay and spoofing against the reconstructed model, with special attention to parser surfaces in privileged contexts.
- New proprietary protocol before product release
- Protocol documentation is incomplete or outdated
- Protocol parser added or significantly changed
- Product connects device-to-device or device-to-backend over custom formats
- A reported protocol-level vulnerability needs assessment
- Protocol has never been tested against an informed adversary