Why this blog (and why it matters)

Wireshark is phenomenal—but your job doesn’t end where its built-in protocol list does. I’ve long been intrigued by how Wireshark turns raw bytes into a clean, hierarchical packet view. Understanding that mechanism—fields, lengths, tags, and how a dissector maps bytes to meaning—pays off far beyond packet analysis. It’s the same foundation you need to develop reliable detection logic in SIEM and NIDS: translate protocol semantics into precise match conditions, field extractions, and anomaly rules, instead of relying solely on signatures.

This guide presents a repeatable method to go from unknown bytes to a working decode. I use IEC 61850-90-5 R-GOOSE over UDP as the example (the Wireshark version I started with didn’t dissect R-GOOSE), but the process applies to any unsupported or proprietary protocol.

By the end, you’ll know how to analyze a protocol, build a working Lua dissector for fast iteration, and demonstrate tangible value with a before/after Wireshark view that analysts—and detection engineers—can operationalize.

Bookmark this. It’s the workflow I rely on whenever Wireshark shows me a hex blob and a shrug.

What we’ll build

• A detailed document that provides methods to decode the protocol structure
• A working Lua dissector for IEC 61850-90-5 (R-GOOSE over UDP)
• A before/after demo: Wireshark showing “unknown UDP payload” → rich, named fields.

Prerequisites

• Wireshark 4.x (I used 4.2.2)
• A pcap with your target protocol (for me: R-GOOSE over UDP/102)
• Your protocol notes/spec (I ingest a Word doc I created: R-GOOSE Protocol.docx)
• Basic Lua (for rapid prototyping)

A word about Lua

In the context of Wireshark, Lua is a lightweight, embedded scripting language that acts as a bridge between you and Wireshark’s powerful internal engine. While Wireshark is built in C for raw performance, it includes a Lua interpreter that allows users to write plugins (like dissectors and taps) in simple text files. Think of it as a “shortcut” to the packet processing core: instead of setting up a complex C compiler environment and rebuilding the entire software to add a new protocol, you simply write a few lines of Lua script, and Wireshark executes it on the fly. It is the industry standard for rapid prototyping and dissecting custom or proprietary protocols.

References for Further Reading

• Wireshark Developer’s Guide (Lua Support): The official manual documentation for the Wireshark Lua API.

• Wireshark Wiki – Lua: A community-maintained hub with examples, tutorials, and common pitfalls.

• Lua.org: The official home of the language, explaining its design philosophy as a lightweight scripting tool.

The Method

Step 1: Build a single source of truth

The first and arguably most critical step is to solidify your understanding of the protocol by creating a simple and concise Protocol Definition Document. This document serves as the absolute source of truth for all subsequent decoding efforts and is the same foundation required to develop precise match conditions and field extractions for detection logic in SIEM and NIDS platforms.

This document maps the protocol structure top-down, detailing every element that turns raw bytes into meaning. Key components include:

• Layering: Precisely defining where the protocol sits within the stack (e.g., UDP → shim → session → payload → APDU).
• Field Mapping: Specifying the exact field order, associated tags, explicit lengths, and specific encodings used (such as TLV, BER, fixed-width, or bit-fields).
• Annotation: Including at least one detailed annotated hex example complete with offsets.
• Naming: Defining the exact, analyst-friendly field names you intend to see displayed within Wireshark.

A robust protocol definition document is paramount: if the document is accurate, the resulting dissector will be accurate. Furthermore, by detailing the structure clearly now, you ensure the AI tool generates high-quality code instantly when you “ingest” this file in Step 4.

I am giving away the document that I have prepared, you can find it here –  Decoding IEC 61850-90-5 (R-GOOSE) Packets – A Practical Guide.pdf

Step 2: Choosing the Approach (The Hard Way vs. The Smart Way)

With the protocol structure fully mapped, Step 2 involves selecting the most efficient method for building the dissector. In the Wireshark ecosystem, there are two primary paths:

• Path 1: C Code (The Hard Way)
  • C is the application’s native language and is considered the “official” way to write dissectors for performance. However, this approach demands writing extensive code, setting up a complex development environment, and requires recompiling—or rebuilding—the entire Wireshark application whenever even a minor change is necessary.
  • For custom or proprietary protocols, this path is often slow, frustrating, and represents significant overkill.
• Path 2: Lua Scripting (The Smart Way)
  • This is the optimized shortcut for rapid development. Instead of requiring complex compilers or rebuilding the entire application, you simply write your dissector in a plain text script,. Wireshark executes this script on the fly.
  • This approach allows developers to move quickly, test changes instantly, and is perfectly suited for letting AI generate the initial files.

Step 3 — Set up your environment for iteration

This step is dedicated to establishing an environment that facilitates rapid iteration and testing. The goal is to set up a configuration that allows you to make a change to the Lua script and observe its effect in Wireshark immediately.

To achieve this, you must locate your Wireshark personal plugin directory, which varies based on your operating system:

• Windows: %APPDATA%\Wireshark\plugins\
• macOS: ~/Library/Application Support/Wireshark/Plugins/
• Linux: ~/.local/lib/wireshark/plugins/ (or ~/.config/wireshark/plugins/)