<SYSTEM>Concepts: Source version: Markbridge 0.4.2. These docs are built from the repository and may include unreleased changes.</SYSTEM>

# Concepts

> How Markbridge works under the hood — the pipeline, the AST, parsers, renderers, and performance.

The deep dives, for when you want to understand or change how Markbridge works:

* **[Architecture](/concepts/architecture/)** — the parse → AST → render pipeline, and why it’s shaped that way.
* **[The AST](/concepts/ast/)** — node types, invariants, and how the tree is built and walked.
* **[Result objects](/concepts/result-objects/)** — what you get back: the `Conversion` and `Parse` value objects.
* **[Parsers](/concepts/parsers/)** — how each of the four parsers works, and their trade-offs.
* **[Renderers](/concepts/renderers/)** — the Discourse renderer, Tags, and the rendering interface.
* **[AST normalization](/concepts/normalization/)** — the pass that rewrites nesting Markdown can’t express before rendering.
* **[Performance](/concepts/performance/)** — where the pipeline is tuned, and how to measure your own workload.

# Architecture

> The three-phase pipeline that turns markup into Markdown.

Markbridge is built around a **Parse → AST → Render** pipeline. Each phase has a single responsibility and doesn’t know about the others. The parse and AST stages are renderer-agnostic; Discourse-flavored Markdown is what the shipped renderer produces.

![Three-phase pipeline: Input (BBCode / HTML / MediaWiki / XML) → AST (Document tree) → Discourse Markdown](/diagrams/architecture.svg)

## Phase 1 — parse

A format-specific parser consumes the input and produces an `AST::Document`. There are four parsers today:

* `Parsers::BBCode::Parser` — token scanner + handler registry (stateful handler API).
* `Parsers::HTML::Parser` — Nokogiri fragment walker (stateless handler API).
* `Parsers::TextFormatter::Parser` — Nokogiri XML walker for the s9e format.
* `Parsers::MediaWiki::Parser` — line-based wikitext parser with no handler registry.

All four produce the same AST node types.

## Phase 2 — the AST

The AST is a tree of `AST::Node` instances. It’s renderer-agnostic: nothing in the tree knows about Markdown.

```plaintext
Node (base)
├── Text (leaf)
├── LineBreak, HorizontalRule (leaf)
└── Element (container, has children)
    ├── Document (root)
    ├── Inline: Bold, Italic, Underline, Strikethrough, Superscript, Subscript
    ├── Block: Quote, List, ListItem, Code, Spoiler, Heading, HorizontalRule
    └── Content: Url, Image, Attachment, Color, Size, Align, Table, TableRow, TableCell
```

Adjacent `Text` nodes auto-merge on insert, which keeps the tree small. `Element` validates that its children are `AST::Node` instances.

## Phase 3 — render

Before the renderer runs, the AST goes through a normalization pass (`parse → yield → normalize → render`). It rewrites nesting Markdown can’t express — a link inside a link, a block inside bold — so the renderer’s tags stay simple string emitters. It’s on by default; see [AST normalization](/concepts/normalization/).

`Renderers::Discourse::Renderer` walks the tree. For each node it looks up a `Tag` in the `TagLibrary` and calls `tag.render(element, interface)`. The interface carries a `RenderContext` — an immutable parent chain that lets tags ask “am I inside a list?” or “what’s my depth?” without passing state around manually.

`RenderContext` is a linked parent chain: each nested level adds one small context object, and `has_parent?` / `find_parent` walk the chain (nesting depth is shallow in practice).

## Design patterns in use

* **Composite** — `Element` contains children forming a tree.
* **Strategy** — BBCode uses pluggable closing strategies (Strict, Reordering).
* **Registry** — `HandlerRegistry` for parsers, `TagLibrary` for the renderer.
* **Visitor** — the renderer dispatches AST nodes to tag implementations.
* **Immutable context** — `RenderContext` creates new instances instead of mutating.

## Why this shape

* **Parsers don’t know about Markdown.** You can add a new output format without touching them.
* **The renderer doesn’t know about BBCode or HTML.** You can add a new input format without touching it.
* **Registries keep customization from forking the core.** Add a handler, add a tag — no subclassing required.

## Next

* [The AST](/concepts/ast/) — node types and invariants
* [Parsers](/concepts/parsers/) — how each parser works
* [Renderers](/concepts/renderers/) — how tags and the rendering interface fit together
* [Performance](/concepts/performance/) — where the pipeline is tuned

# The AST

> Node types, invariants, and how the tree gets built.

Every parser produces an abstract syntax tree (AST). The tree stores content and structure independently of the input syntax and renderer.

## Node hierarchy

```plaintext
AST::Node (base)
├── Leaves
│   ├── AST::Text                  — string content
│   ├── AST::MarkdownText          — pre-rendered Markdown passthrough
│   ├── AST::LineBreak
│   └── AST::HorizontalRule
├── Discourse-specific leaves
│   ├── AST::Event                 — calendar event reference
│   ├── AST::Mention               — @username reference
│   ├── AST::Poll                  — Discourse poll reference
│   └── AST::Upload                — uploaded-file reference
└── AST::Element (container)
    ├── AST::Document              — root node
    ├── Inline formatting
    │   ├── AST::Bold
    │   ├── AST::Italic
    │   ├── AST::Underline
    │   ├── AST::Strikethrough
    │   ├── AST::Superscript
    │   └── AST::Subscript
    ├── Block-level
    │   ├── AST::Paragraph
    │   ├── AST::Heading           — level
    │   ├── AST::Quote
    │   ├── AST::Spoiler
    │   └── AST::Details           — collapsible [details] section
    ├── Content
    │   ├── AST::Url               — href attribute
    │   ├── AST::Email             — email address
    │   ├── AST::Image             — src, alt attributes
    │   ├── AST::Attachment
    │   ├── AST::Code              — language and block flag
    │   └── AST::Color, AST::Size, AST::Align
    ├── Lists
    │   ├── AST::List              — ordered / unordered
    │   └── AST::ListItem
    └── Tables
        ├── AST::Table
        ├── AST::TableRow
        └── AST::TableCell
```

## Invariants

* **Children are always `AST::Node` instances.** `Element#<<` validates on insert.
* **Adjacent `Text` nodes auto-merge.** Inserting `Text("a")` then `Text("b")` results in a single `Text("ab")` child — not two.
* **Leaves have no children.** `LineBreak` and `HorizontalRule` extend `Node` directly, not `Element`, and will reject children.
* **Node attributes are read-only.** You can edit the tree with `<<`, `replace_child`, and `replace_children`.

## Building and inspecting

```ruby
doc = AST::Document.new
bold = AST::Bold.new
bold << AST::Text.new("Hello, ")
bold << AST::Text.new("world")   # auto-merged into one Text("Hello, world")
doc << bold


doc.children.first.class           # => AST::Bold
doc.children.first.children.length # => 1
```

## Walking the tree

Use `each_descendant` to visit nodes in depth-first order, or `descendants(klass)` to collect nodes of a given class. Subclasses match too.

```ruby
parse = Markbridge.parse_bbcode("[b]Hello[/b] [url=/about]About[/url]")
links = parse.ast.descendants(Markbridge::AST::Url)
links.map(&:href) # => ["/about"]
```

`replace_child(old_node, new_node)` replaces a direct child at the same position. During traversal, each element uses a copy of its child list. Replacing a child is supported, but the walk continues through the original node. Added children may not be visited during that walk.

## Code spans and blocks

`AST::Code.new(language: "ruby", block: true)` forces a fenced block, even for one line. Without `block: true`, single-line content renders as a code span and multiline content renders as a fenced block. Empty code nodes produce no output.

## Why a shared AST matters

The AST is what lets four parsers share one renderer. Any new input format — Markdown, AsciiDoc, some vendor-specific XML — only has to produce the same node types, and everything downstream works without changes. Similarly, a second renderer (say, plain text or HTML) only has to walk the existing AST.

# AST normalization

> The pass that rewrites the AST so the renderer only sees markup the target format can express.

Markup can nest elements in ways Markdown can’t express: a link inside a link, a block element inside an inline container (a link label, but also bold or a heading), a fenced code block inside emphasis. If the renderer printed these as-is, the Markdown would break — the inner link wins and the outer one turns into text, a block’s blank lines break out of the emphasis around it.

`Markbridge::Normalizer` walks the AST once, between the parse-time `yield` hook and rendering, and rewrites it so the renderer only gets markup the target format can express. It runs **by default**. The renderer’s tags stay simple string emitters; the rules about what may nest in what live here instead.

## Where it runs

```plaintext
parse → yield(ast) → normalize → render
```

Because it runs after the `yield` hook, changes you make to the AST in that block are normalized too. It runs for every source format and for `Markbridge.render`, because normalization is about the *target* format, not the source.

## The default rules

The default rules are CommonMark legality. Break one and the Markdown doesn’t parse back as the tree meant:

* **No link inside a link**, at any depth (§6.3). The inner link is unwrapped.
* **A block inside an inline container is moved out.** An inline container holds inline content only. This isn’t link-specific: emphasis (`Bold`, `Italic`, …) and headings are inline containers too, so a poll inside bold or a list inside a heading is handled the same way as a block inside a link. The block nodes cover `List`, `Table`, `Quote`, `Details`, `HorizontalRule`, `Align`, and the Discourse `Poll`/`Event` nodes.
* **A fenced or multi-line code block inside an inline container is moved out.** A one-line code span is fine.

Discourse-specific policy is **not** built in. Moving an image out of a link, for example, is a rule you add yourself (see below) — a linked image (`[![alt](src)](url)`) is valid CommonMark, so the default leaves it alone.

## Strategies

Each match resolves to one strategy:

| Strategy       | Effect                                                                                                                                                                                                                                                                                     |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `:keep`        | Allow it. Records a decision and keeps it out of the report.                                                                                                                                                                                                                               |
| `:hoist_after` | Move the node out and place it right after the outermost matching ancestor, keeping document order. An image in a bold that sits in a link moves after the whole link (out of both). The walker only moves a node out to a sibling; it never puts one into a wrapper it wasn’t already in. |
| `:unwrap`      | Remove the element and put its children in its place. The built-in case is a link in a link: `[[text](inner)](outer)` becomes `[text](outer)` — the inner href is dropped, its text stays under the outer link.                                                                            |
| `:textify`     | Replace the subtree with its plain text (`@name` for a mention, the joined text otherwise).                                                                                                                                                                                                |
| `:drop`        | Remove it.                                                                                                                                                                                                                                                                                 |
| callable       | `->(boundary, node) { … }` returning a strategy symbol, an `Array<AST::Node>` to put in its place, or `nil` to drop it. For anything the built-in strategies don’t cover.                                                                                                                  |

A formatting wrapper (bold, italic, color, …) left empty after a hoist or drop is removed, so no empty `**` `**` markers remain. A link is the exception — an empty link is kept, because it renders as a plain URL.

## Diagnostics

Every change is reported through the same channel as `unknown_tags`:

```ruby
conversion = Markbridge.convert(input, format: :bbcode)
conversion.diagnostics[:normalization]
# => [{ parent: "Url", child: "Url", strategy: :unwrap, count: 1 }]
```

For a migration this feeds per-post warnings and shows which sources produce broken trees. The key is absent when nothing changed.

## Opting out and customizing

`normalize:` takes `true` (default — the shared normalizer), `false` (skip), or a `Normalizer` instance:

```ruby
# Skip normalization
Markbridge.convert(input, format: :bbcode, normalize: false)


# Add your own rules on top of the defaults
normalizer = Markbridge::Normalizer.default
normalizer.rule(parent: Markbridge::AST::Url, child: Markbridge::AST::Image, strategy: :hoist_after)
Markbridge.convert(input, format: :bbcode, normalize: normalizer)
```

Build a customized normalizer once and reuse it. `#normalize` and `#violations` keep no state on the instance, so one frozen instance is safe for every conversion, also across threads — passing your own is as fast as the default path. A rule for a `(parent, child)` pair that already exists is replaced, so your `#rule` calls override the defaults. Rules match subclasses too, for both `parent:` and `child:`. A rule for a more specific class takes priority. See [AST subclasses](/customization/extending/#ast-subclasses).

`Markbridge::Normalizer.shared_default` is the default normalizer, built once and frozen; the `normalize: true` path uses it. Don’t mutate it — call `.default` for a fresh, customizable one.

## Validation

The same rules, without changing the tree:

```ruby
Markbridge::Normalizer.default.violations(ast)
# => [{ parent: "Url", child: "Url", strategy: :unwrap }]
```

Two uses: assert in your own suite that the trees your parsers and tag fixtures build have no violations, or run it as a lint over a corpus without changing any output. After a `normalize`, `violations` returns nothing — normalization is a single pass.

# Parsers

> How the four parsers work and where they differ.

Each parser targets one input format but produces the same AST. They share philosophy (graceful degradation, bounded recursion) but not implementation.

## BBCode parser

**Path:** `Markbridge::Parsers::BBCode::Parser`

A hand-written, two-stage parser:

1. **Scanner** — streams the input and produces `TextToken`, `TagStartToken`, `TagEndToken`. Byte-offset based (`byteslice` / `byteindex` / `getbyte`, not character indices, which are O(n) on multibyte input), no regex except for character classes, minimal allocations.
2. **Parser** — consumes tokens through a `HandlerRegistry`. Each handler implements `on_open` / `on_close`. A `ParserState` tracks the node stack and enforces the max-depth limit (100).

**Unique to BBCode:** closing strategies. Real-world BBCode often has mismatched tags (`[b][i]text[/b][/i]`). A `ClosingStrategy` decides how to recover:

* `Strict` — auto-close only.
* `Reordering` (default) — reconciles sequences of up to 5 mismatched closing tags by peeking ahead.

**Handler API:** stateful. Handlers push/pop elements on the parser state stack via `on_open` / `on_close` callbacks.

## HTML parser

**Path:** `Markbridge::Parsers::HTML::Parser`

Thin wrapper over `Nokogiri::HTML.fragment` + a handler registry. Walks the DOM and dispatches each element to a handler.

**Handler API:** stateless — an object responding to `#process(element:, parent:)`. It adds an AST node to `parent` and returns either the node to descend into, or `nil` to skip children.

```ruby
class AsideHandler < Markbridge::Parsers::HTML::Handlers::BaseHandler
  # Descend the children straight into the parent — no AST node for <aside>.
  def process(element:, parent:)
    parent
  end
end


registry.register("aside", AsideHandler.new)
```

Relies on Nokogiri for malformed-HTML recovery — no need for Markbridge to do its own.

## TextFormatter parser

**Path:** `Markbridge::Parsers::TextFormatter::Parser`

Parses the XML format produced by [s9e/TextFormatter](https://github.com/s9e/TextFormatter). Used primarily for phpBB 3.2+ migrations, where BBCode is stored as parsed XML rather than raw BBCode.

Same stateless handler API as the HTML parser. The main quirks are format-level:

* Roots are `<t>` (plain) or `<r>` (rich).
* Formatted elements use **uppercase** names (`<B>`, `<URL>`).
* `<s>` / `<e>` tags preserve original BBCode markup and are ignored during parsing.

Invalid XML falls back to treating the input as plain text.

## MediaWiki parser

**Path:** `Markbridge::Parsers::MediaWiki::Parser`

Unlike the others, MediaWiki’s block syntax is fixed — wikitext lines are classified (heading, list, table, preformatted, blank, plain) and dispatched without a handler registry. An `InlineParser` handles intra-line formatting (`'''bold'''`, `[[links]]`, etc.).

The one parser-level extension point is `InlineTagRegistry` for HTML-like inline tags (`<nowiki>`, `<code>`, `<sup>`, etc.). Pass a customized registry via the `handlers:` kwarg. Block-level customization is post-parse: transform the AST or register a different renderer tag.

## Shared traits

All four parsers:

* **Don’t raise on bad input.** Unknown tags/elements are skipped (children still processed).
* **Return an `AST::Document`.** The root is always a `Document` containing children.
* **Track unknown tags** so you can audit migrations. Surfaced via `Parse#unknown_tags` and `Conversion#unknown_tags` as `Hash{String => Integer}`.
* **Normalize line endings** (CRLF → LF) before parsing.

## Picking one

| Input                         | Use           |
| ----------------------------- | ------------- |
| `[b]...[/b]`                  | BBCode        |
| `<p><strong>...</strong></p>` | HTML          |
| `<r><B>...</B></r>`           | TextFormatter |
| `'''bold''' and [[links]]`    | MediaWiki     |

If you have a choice (e.g. phpBB 3.2+ stores both), prefer the TextFormatter parser over re-parsing the raw BBCode — it’s faster and closer to the source of truth.

# Performance

> Reuse your configuration, process large inputs, and measure conversion speed.

Measure performance with representative input. Document size, nesting, custom handlers, and your Ruby engine all affect conversion time.

## Reuse your configuration

The default handler registries and tag library are shared and frozen. Use `.default` to build a fresh registry when you need to customize it.

Build a custom renderer once and reuse it across conversions:

```ruby
renderer = Markbridge.discourse_renderer(escape_hard_line_breaks: true)


posts.each do |post|
  result = Markbridge.bbcode_to_markdown(post.body, renderer:)
  puts result.markdown
end
```

The renderer stores no per-document state. Your custom handlers and tags should follow the same rule if you share them across calls or threads.

## Process one document at a time

Each conversion builds its AST and output in memory. For large collections, read and write one document at a time instead of keeping every result.

```ruby
require "csv"
require "markbridge/bbcode"


CSV.foreach("posts.csv", headers: true) do |row|
  result = Markbridge.bbcode_to_markdown(row["body"])
  puts result.markdown
end
```

## Parser limits

The BBCode parser limits nesting to 100 levels. Its built-in handlers preserve an opening tag as text when that limit is reached and record `depth_exceeded_count` in the diagnostics. A custom handler that calls `ParserState#push` without a token can raise `MaxDepthExceededError` at that limit.

Closing strategies search at most five levels when reconciling mismatched tags. These limits keep searches bounded on deeply nested input.

## How parsing works

* **BBCode:** the scanner uses byte offsets and integer checks for ASCII syntax. Offsets remain on character boundaries for multibyte input.
* **HTML and TextFormatter:** Nokogiri parses the input before Markbridge walks the document. Pass an existing Nokogiri node if your code has already parsed it.
* **MediaWiki:** the parser reads block syntax by line and uses a separate parser for inline markup.

Adjacent text nodes merge during AST construction. The renderer walks the resulting tree and uses a parent context for nested content.

## Measure your workload

Read input before timing if you want to measure conversion without file access:

```ruby
require "benchmark"
require "markbridge/bbcode"


inputs = File.readlines("corpus.txt", chomp: true)


Benchmark.bm do |benchmark|
  benchmark.report("conversion") do
    inputs.each { |input| Markbridge.bbcode_to_markdown(input) }
  end
end
```

On CRuby, compare runs with YJIT enabled:

```bash
ruby --yjit your_script.rb
```

JRuby and TruffleRuby have their own JIT compilers. Allow time for warmup and compare both startup time and sustained conversion speed.

## Repository benchmarks

Run benchmarks through `bin/bench-env`. It selects the fastest CPU cores and reports power and governor settings. Compare runs on AC power with the same governor.

```bash
bin/bench-env bundle exec ruby --yjit bench/bench.rb --isolated
DURATION=60 WINDOW=5 POSTS=2000 CORPUS=ascii bin/bench-env bundle exec ruby --yjit bench/sustained_bench.rb
DURATION=60 WINDOW=5 POSTS=2000 CORPUS=multi bin/bench-env bundle exec ruby --yjit bench/sustained_bench.rb
```

Run the ASCII and multibyte corpora in separate processes. See [Benchmark results](/concepts/benchmarks/) for recorded measurements and their limits.

# Renderers

> How the Discourse Markdown renderer walks the AST.

The renderer takes an `AST::Document` and produces a Markdown string. It lives under `Markbridge::Renderers::Discourse` and has three collaborators:

* **`Renderer`** — walks the tree.
* **`TagLibrary`** — maps `AST::Node` classes to `Tag` implementations.
* **`RenderingInterface`** — what a `Tag` actually sees when rendering.

## The rendering loop

```plaintext
renderer.render(document)
  for each child node:
    tag = tag_library[child.class] || tag_library.resolve(child.class)
    render with the tag, or render the children if no tag matches
```

The renderer holds no state beyond the running output and the context. All decisions about markup (`**`, `_`, backticks, etc.) live in the individual `Tag` classes.

## Tags

A `Tag` is any class (or block) that responds to `render(element, interface)`:

```ruby
class BoldTag < Markbridge::Renderers::Discourse::Tag
  def render(element, interface)
    context = interface.with_parent(element)
    content = interface.render_children(element, context:)
    return "<strong>#{content}</strong>" if interface.html_mode?


    interface.wrap_inline(content, "**")
  end
end
```

For simple cases, the block constructor is often enough:

```ruby
Markbridge::Renderers::Discourse::Tag.new do |element, interface|
  context = interface.with_parent(element)
  content = interface.render_children(element, context:)
  if interface.html_mode?
    "<strong>#{content}</strong>"
  else
    interface.wrap_inline(content, "**")
  end
end
```

## The rendering interface

`Tag` implementations never see the renderer directly. They receive a `RenderingInterface` that exposes only what a tag should need:

| Method                          | Use                                                                                                            |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `render_children(element)`      | Recurse into children, concatenate output                                                                      |
| `with_parent(element)`          | Return a new context that treats `element` as parent                                                           |
| `find_parent(klass)`            | Walk ancestors for a specific class                                                                            |
| `has_parent?(klass)`            | Boolean ancestor check                                                                                         |
| `count_parents(klass)`          | How deep a specific ancestor is (nested lists, quotes)                                                         |
| `wrap_inline(content, markers)` | Wrap inline content with collapsing markers                                                                    |
| `block_context?(element)`       | Block vs. inline position                                                                                      |
| `html_mode?`                    | True inside a CommonMark HTML block — Tag must emit raw HTML or wrap output as a Markdown island (`\n\n…\n\n`) |

The interface decouples tags from the renderer: you could write a second renderer (HTML, plain text, JSON) and reuse every tag by providing a compatible interface.

## RenderContext

Behind the interface is a `RenderContext` — an immutable, linked parent chain. Creating a child context (via `with_parent`) links a new instance to the current one; the old one is untouched. `has_parent?` / `find_parent` walk that chain, so each nested level is a single fixed-size allocation rather than a copied parent array, and nesting depth stays shallow in practice.

A child context does not change the context used by its siblings.

## TagLibrary

The Discourse renderer ships with a default library mapping each built-in AST class to its Tag. `TagLibrary.default` returns a fresh copy each call (mutating it doesn’t affect other callers).

For most customization, prefer the `Markbridge.discourse_renderer(tags:, unregister:)` factory over hand-mutating a library — it gives you a complete, reusable Renderer:

```ruby
RENDERER = Markbridge.discourse_renderer(
  tags: { Markbridge::AST::Url => MyUrlTag.new },     # override
  unregister: [Markbridge::AST::Color],               # render children only
)
```

The renderer checks the node’s class, then its nearest registered ancestor class. If neither has a tag, it renders only the children. To render only the children of a subclass with an inherited tag, register `Tag::PASSTHROUGH`. See [AST subclasses](/customization/extending/#ast-subclasses).

For lower-level use, `TagLibrary.new.auto_register!` discovers convention-paired classes (`BoldTag` → `AST::Bold`, etc.) under `Markbridge::Renderers::Discourse::Tags::*`. Consumer-defined tag classes still need explicit registration.

## Output cleanup

After the tree walk, the renderer’s `Postprocessor` runs a small cleanup pass on the joined output:

* Collapse runs of 3+ newlines to 2.
* Strip whitespace-only lines.
* Trim leading/trailing whitespace.

The default postprocessor is `Markbridge::Renderers::Discourse::Postprocessor::DEFAULT`. Pass a custom one (or a subclass) via `Markbridge.discourse_renderer(postprocessor:)` to change or extend the cleanup. Calling `Renderer#render` directly returns the *un*-postprocessed string.

## Writing a new renderer

Because the AST is renderer-agnostic, writing (say) a plain-text renderer is a matter of implementing a new `Renderer` that walks the AST and emits whatever you want. Re-using the `Tag` / `TagLibrary` pattern is recommended — it gives you the same extension points for free.

# Result objects

> Read Markdown, AST nodes, unknown tags, and diagnostics from Conversion and Parse results.

Markbridge’s convenience methods don’t return a plain string. The render methods return a `Conversion`; the parse-only methods return a `Parse`. Both carry their data alongside the result, so you never need a side channel.

## Conversion

`Markbridge.*_to_markdown` and `Markbridge.convert` return a `Markbridge::Conversion`:

```ruby
result = Markbridge.bbcode_to_markdown(post.body, renderer: RENDERER)


result.markdown      # the rendered Discourse Markdown
result.ast           # the AST::Document used for rendering
result.format        # :bbcode, :html, :text_formatter_xml, or :mediawiki
result.unknown_tags  # Hash{String => Integer} — tag name to count
result.diagnostics   # parser-specific diagnostics (e.g. auto-close counts)
result.errors        # render-time errors, when raise_on_error: false
```

`Conversion#to_s` delegates to `markdown`, so `puts result` and `"#{result}"` work without a `.markdown` call. It does *not* delegate other String methods — `result.gsub(...)` raises. Reach for `result.markdown.gsub(...)` or unwrap explicitly.

`Markbridge.convert(input, format:)` returns the same `Conversion`, dispatching to the right `*_to_markdown` method — handy when one corpus mixes formats:

```ruby
posts.each do |post|
  result = Markbridge.convert(post[:body], format: post[:format], renderer: RENDERER)
  # use result.markdown…
end
```

## Parse

Each `*_to_markdown` has a matching `parse_*` that stops after building the AST. It returns a `Markbridge::Parse` — everything a `Conversion` has except the rendered `markdown` (and `errors`, which are render-time):

```ruby
parse = Markbridge.parse_bbcode(input)


parse.ast           # the AST::Document
parse.format        # :bbcode
parse.unknown_tags  # Hash{String => Integer}
parse.diagnostics   # parser-specific diagnostics
```

Reach for `parse_*` when you want to inspect or transform the tree before rendering, or render it yourself with a custom renderer. Hand a `Parse` to `Markbridge.render(parse, renderer:)` to get a `Conversion` back (the source `format`, `unknown_tags`, and `diagnostics` carry through).

## What the fields mean

* **`unknown_tags`** — tags the parser didn’t recognize, with a count each. The parser never raises on them; what you do with the list is your call.
* **`diagnostics`** — parser-specific notes (auto-closed tags, depth limits hit, unclosed raw tags, and so on). Also carries `:normalization` — the nesting fixes the [normalizer](/concepts/normalization/) applied on the way to render, absent when nothing changed.
* **`errors`** — render-time errors collected instead of raised, when you pass `raise_on_error: false`. Empty otherwise.

# Benchmark results

> Recorded throughput for CRuby, JRuby, and TruffleRuby, with the benchmark setup.

These recorded measurements show throughput (iterations per second) for `Markbridge.bbcode_to_markdown(input)` and `MarkdownEscaper#escape(text)` across Ruby implementations on `main`.

## Machine

* Laptop with a 13th Gen Intel Core i9-13900H (6 performance cores, 8 efficiency cores, 20 threads), Fedora 44.
* On AC power, `performance` CPU governor. On battery the CPU is throttled and the numbers cannot be compared with these.
* Every run pinned to the performance cores (0–11) with [`bin/bench-env`](https://github.com/discourse/markbridge/blob/main/bin/bench-env).

A laptop is not the fastest machine available, but it is close to what many imports actually run on.

## Setup

* Branch: `main`
* Bench scripts: [`bench/bench.rb`](https://github.com/discourse/markbridge/blob/main/bench/bench.rb) for the short reports, [`bench/sustained_bench.rb`](https://github.com/discourse/markbridge/blob/main/bench/sustained_bench.rb) for the long corpus run further down.
* Mode: `--isolated` (each report runs in a fresh process so the JIT doesn’t have to share its budget across all 10 reports)
* Single machine, single run per engine, all runs in one sitting — small inter-run variance expected.
* 10s warmup, 5s measure for every engine (`BENCH_WARMUP=10 BENCH_MEASURE=5`). The same settings for all engines keep the columns comparable. YJIT is warm in well under a second, so the long warmup costs CRuby nothing, while JRuby and TruffleRuby need it.

`bin/bench-env` picks the fastest cores of the machine, pins the command to them, and prints whether the machine is on AC power and which governor is active. It does not change either — that needs root:

```sh
sudo cpupower frequency-set -g performance
```

Run yourself:

```sh
# CRuby (auto-enables --yjit)
BENCH_WARMUP=10 BENCH_MEASURE=5 bin/bench-env rv run --ruby 4.0 bundle exec ruby bench/bench.rb --isolated


# JRuby
BENCH_WARMUP=10 BENCH_MEASURE=5 bin/bench-env rv run --ruby jruby bundle exec ruby bench/bench.rb --isolated


# TruffleRuby (installed by .silo.yml, or with `ruby-install truffleruby 40.0.0`)
BENCH_WARMUP=10 BENCH_MEASURE=5 bin/bench-env rv run --ruby truffleruby bundle exec ruby bench/bench.rb --isolated
```

## Versions

* `ruby 3.3.11` with YJIT
* `ruby 3.4.9` with YJIT (+PRISM)
* `ruby 4.0.5` with YJIT (+PRISM)
* `jruby 10.1.0.0` (Ruby 4.0.0; OpenJDK 27-ea, default JIT, indy enabled)
* `truffleruby 40.0.0` (Ruby 4.0.2; GraalVM CE Native — the build `ruby/setup-ruby` installs for `truffleruby`)

## Results

Throughput (i/s, higher is better). `vs 4.0` shows the relative difference from Ruby 4.0.5 with YJIT.

| Path          | 3.3    | 3.4    | 4.0    | JRuby  | TruffleRuby | 3.3 vs 4.0 | 3.4 vs 4.0 | JRuby vs 4.0 | TruffleRuby vs 4.0 |
| ------------- | ------ | ------ | ------ | ------ | ----------- | ---------- | ---------- | ------------ | ------------------ |
| simple        | 61.5k  | 60.6k  | 81.0k  | 59.6k  | 314.8k      | −24%       | −25%       | −26%         | **+289%**          |
| nested        | 70.1k  | 64.8k  | 88.8k  | 62.5k  | 321.4k      | −21%       | −27%       | −30%         | **+262%**          |
| list          | 56.1k  | 52.9k  | 71.3k  | 49.1k  | 253.2k      | −21%       | −26%       | −31%         | **+255%**          |
| table         | 18.1k  | 19.1k  | 26.2k  | 16.4k  | 84.4k       | −31%       | −27%       | **−37%**     | **+222%**          |
| quote\_nested | 42.6k  | 41.6k  | 52.3k  | 39.5k  | 193.6k      | −19%       | −20%       | −24%         | **+270%**          |
| code          | 125.7k | 118.1k | 162.2k | 134.1k | 585.7k      | −23%       | −27%       | −17%         | **+261%**          |
| url           | 67.0k  | 69.0k  | 88.4k  | 65.9k  | 367.3k      | −24%       | −22%       | −26%         | **+315%**          |
| escaping      | 86.9k  | 78.1k  | 93.7k  | 102.9k | 318.7k      | −7%        | −17%       | **+10%**     | **+240%**          |
| escape\_plain | 396.4k | 403.1k | 397.9k | 516.8k | 583.8k      | −0%        | +1%        | **+30%**     | **+47%**           |
| escape\_mixed | 11.4k  | 7.5k   | 7.8k   | 16.4k  | 36.5k       | **+47%**   | −4%        | **+111%**    | **+371%**          |

Run-to-run noise (benchmark-ips standard deviation) stayed at or below 3.2% on the CRuby versions. JRuby and TruffleRuby reached 8.4% and 8.1%, so read their single-digit differences with care.

These measurements compare repeated conversions of small inputs. They do not predict the speed of a full application. The sustained run below uses a larger set of documents.

## Sustained corpus run

The reports above convert one small input each, and they are over in a few seconds. `bench/sustained_bench.rb` measures the opposite case: 2000 forum-post-shaped documents (\~1 KB each, built by [`bench/corpus.rb`](https://github.com/discourse/markbridge/blob/main/bench/corpus.rb)) converted in a loop for 60 seconds per corpus, with the throughput of every 5 second window printed. This is closer to a bulk migration than a micro report is.

```sh
DURATION=60 WINDOW=5 POSTS=2000 CORPUS=ascii bin/bench-env bundle exec ruby --yjit bench/sustained_bench.rb
```

Each corpus ran in its own process (`CORPUS=ascii`, then `CORPUS=multi`). Running both in one process changes the JRuby result — see the note at the end of this section. Steady state is the average of the last three windows.

| Engine         | ASCII | multibyte | ASCII vs 4.0 | multibyte vs 4.0 |
| -------------- | ----- | --------- | ------------ | ---------------- |
| ruby 3.3       | 19.6k | 13.7k     | −9%          | −4%              |
| ruby 3.4       | 16.5k | 11.9k     | −23%         | −17%             |
| ruby 4.0       | 21.4k | 14.3k     | —            | —                |
| JRuby 10.1     | 19.8k | 11.6k     | −8%          | −19%             |
| TruffleRuby 40 | 58.2k | 47.4k     | **+171%**    | **+232%**        |

Posts per second. At steady state that is 20.0 MB/s for Ruby 4.0 and 54.2 MB/s for TruffleRuby on the ASCII corpus.

How long each engine needs to get there (ASCII corpus):

| Engine         | first 5s window | steady | gain      |
| -------------- | --------------- | ------ | --------- |
| ruby 3.3       | 19.5k           | 19.6k  | +1%       |
| ruby 3.4       | 17.0k           | 16.5k  | −3%       |
| ruby 4.0       | 22.4k           | 21.4k  | −4%       |
| JRuby 10.1     | 8.9k            | 19.8k  | **+122%** |
| TruffleRuby 40 | 21.2k           | 58.2k  | **+175%** |

On the multibyte corpus the same warmup is +148% for JRuby and +266% for TruffleRuby.

* **The long run changes the ranking.** JRuby is 26% behind Ruby 4.0 on the `simple` micro report, but only 8% behind over the ASCII corpus. TruffleRuby’s lead drops from 2.2x–4.2x to 2.7x (ASCII) and 3.3x (multibyte). The bigger and more varied the input, the smaller the distance — but TruffleRuby stays far ahead.
* **The JVM engines need seconds of warmup.** On this corpus JRuby reaches full speed after about 10 seconds and TruffleRuby after 15. CRuby is at full speed in the first window. Warmup depends on how much code the workload touches, so the micro reports, which run one small input, get there faster than this.
* **For a migration, read the column that matches the job.** A job that runs for minutes gets the steady numbers. A script that converts a few hundred posts and exits pays the warmup instead, and there Ruby 4.0 is the fastest of the three.
* **Multibyte input costs every engine**, from −19% (TruffleRuby) to −41% (JRuby) against its own ASCII result. Ruby 4.0 loses 33%.

### JRuby slows down when both corpora share a process

A fresh JRuby process that only converts the multibyte corpus reaches 11.6k posts/s. The same corpus in a process that converted the ASCII corpus first reaches either the same 11.8k or about 6.0k, and then stays there for the rest of the run. Across six runs of that sequence — pinned and unpinned, on battery and on AC — four landed near 6.0k and two near 11.8k.

CPU pinning is not the cause: a fresh multibyte process gives the same number with and without it. CRuby does not show the effect, and TruffleRuby loses only 5% to 8% in the same sequence.

That looks like a JIT decision made for ASCII strings that does not recover once multibyte strings arrive at the same call sites. Real imports convert mixed content, so a JRuby migration can land in either state. Worth knowing before choosing JRuby for a large import, and worth a bug report if someone wants to dig into it.

## Caveats

* Single run per engine; for production decisions, average 3+ runs.
* The machine ran a normal desktop session during the measurements, so a few percent of noise is expected, and the slow decline of some later windows may come from that. Keep an eye on the standard deviation and repeat a run that looks odd.
* The `--isolated` mode pays a fork-per-report cost, so total wallclock is \~10× a single report. Use suite mode (`bench/bench.rb` with no flag) for a quick smoke test.
* `--yjit` is enabled on all CRuby runs and matters: without it numbers are 2–5× lower and don’t reflect production.
* TruffleRuby was measured with the Community native build. The Oracle GraalVM build (`truffleruby+graalvm`) is usually faster still, and the JVM build starts slower but can reach higher peaks.