Migrating to Discourse: Source version: Markbridge 0.4.2. These docs are built from the repository and may include unreleased changes. # Migrating to Discourse — overview > The big picture for using Markbridge to migrate forum content into Discourse. You have a forum’s worth of posts in BBCode, HTML, MediaWiki wikitext, or s9e/TextFormatter XML, and you want each one stored on Discourse as Markdown. Most of the body translates straight across — bold text, lists, headings — but a few tags need more than that: internal links to topics that don’t exist on Discourse yet, uploads to ship, mentions to look up, polls and events that map to Discourse plugins. That awkward last mile is exactly what Markbridge is built for. This page is the big picture. The next page — [Placeholders](/migrating/placeholders/) — fills in the details. ## How a migration uses the pipeline Every conversion runs the same parse → AST → render pipeline — the [Introduction](/introduction/) covers it. A migration wraps three points around that pipeline: * **On the way in**, custom handlers turn source tags into AST nodes — including your own nodes for uploads, mentions, and internal links. * **On the way out**, custom Tags format each node. A placeholder Tag is a one-line formatter that returns its placeholder string and nothing else. * **After the conversion**, you walk the AST to collect what each post referenced and write it into Discourse. That last step — the read-back — is the heart of the workflow; the [Placeholders](/migrating/placeholders/) page covers it in full. ## What you get back Every `*_to_markdown` call — and `Markbridge.convert`, which dispatches by `format:` when a corpus mixes formats — returns a [`Conversion`](/concepts/result-objects/): the rendered `markdown` plus the `ast`, `unknown_tags`, `diagnostics`, and `errors`. The rest of this page leans on those fields; [Result objects](/concepts/result-objects/) covers the full shape. ## Side data: read it back off the AST A migration usually needs more than the rendered string. For each placeholder in a post, you also need the data to resolve it later: | Tag in source | Placeholder string in markdown | Side data to resolve later | | -------------------------- | ------------------------------ | ----------------------------------- | | `[upload=42]` | `[upload\|42]` | `{ upload_id: 42, path: "..." }` | | `[mention]alice[/mention]` | `[mention\|alice]` | `{ name: "alice", source_id: nil }` | | `[url=/topics/old-id-7]` | `[topic\|7]` | `{ source_topic_id: 7 }` | You don’t need a side channel for this. The custom AST node carries the parsed data, and `conversion.ast` is the exact tree that produced the Markdown — so the Tag stays a pure formatter: ```ruby class UploadTag < Markbridge::Renderers::Discourse::Tag def render(element, _interface) "[upload|#{element.upload_id}]" end end ``` After the conversion, collect every upload the post referenced by walking the tree for your placeholder class. (In real use, `result` comes from a `*_to_markdown` call; here we build a one-node tree by hand to keep the snippet self-contained.) ```ruby class UploadPlaceholder < Markbridge::AST::Node attr_reader :upload_id, :path def initialize(upload_id:, path:) @upload_id = upload_id @path = path end end document = Markbridge::AST::Document.new document << UploadPlaceholder.new(upload_id: 42, path: "files/cat.png") result = Markbridge.render(document) result.ast.descendants(UploadPlaceholder).each do |node| record_upload(id: node.upload_id, path: node.path) end ``` `descendants(klass)` walks the whole tree (leaf nodes included) and returns the nodes that survived parsing — exactly the placeholders that reached the output, for this one conversion only. Nothing leaks between posts, because each call has its own `ast`. The three pieces that make this work — the AST node, the handler, and the Tag — are explained on the [Placeholders](/migrating/placeholders/) page. ## Unknown tags Source content rarely covers exactly the tags you’ve planned for. `Conversion#unknown_tags` (and `Parse#unknown_tags`) gives you the punch list: ```ruby result.unknown_tags # => {"marquee" => 3, "blink" => 1, "googletools" => 12} ``` What you do with it is policy: ```ruby posts.each do |post| result = Markbridge.bbcode_to_markdown(post.body, renderer: RENDERER) result.unknown_tags.each do |tag, count| log "post #{post.id}: unknown tag [#{tag}] x#{count}" end end ``` Aggregate across the corpus to find which tags are worth writing handlers for, which to silently drop, and which to fail on. Markbridge never raises for an unknown tag — that decision belongs to your migration script. ## Per-post failure isolation Forum corpora contain edge cases that surface only when you migrate them. By default, render-time errors propagate; a single bad post crashes the loop. Pass `raise_on_error: false` to flip that: ```ruby posts.each do |post| result = Markbridge.bbcode_to_markdown(post.body, renderer: RENDERER, raise_on_error: false) if result.errors.any? log_failure(post, result.errors) next end write_markdown(post, result.markdown) end ``` Errors collect on `Conversion#errors` instead of raising. The default stays `raise_on_error: true` so you don’t accidentally suppress bugs in unit tests. ## Build the renderer once Construct a `Renderer` once outside your migration loop and pass it to every call. It carries your custom Tags, the unregistered AST classes you don’t want to render, your escaper, and your postprocessor. It holds no per-post state, so the same instance is safe across thousands of posts. ```ruby RENDERER = Markbridge.discourse_renderer( tags: { Markbridge::AST::Url => InternalLinkTag.new, Markbridge::AST::Upload => UploadTag.new, Markbridge::AST::Mention => MentionTag.new, }, unregister: [Markbridge::AST::Color, Markbridge::AST::Size], escape_hard_line_breaks: true, ) posts.each do |post| result = Markbridge.bbcode_to_markdown(post.body, renderer: RENDERER) # ... end ``` See [Customizing the renderer](/customization/customizing-renderer/) for every kwarg. ## Where next * [Placeholders](/migrating/placeholders/) — the AST node, handler, and Tag in detail, with the full round-trip. * [Customizing the renderer](/customization/customizing-renderer/) — the full factory reference. * [Extending Markbridge](/customization/extending/) — adding handlers and custom Tags from scratch. # Placeholders > Render links, uploads, mentions, and other importer-resolved tags as placeholder strings, then collect what you need by walking the parsed AST. A *placeholder* is a short literal string in the rendered Markdown that the Discourse importer swaps for a real value later: an upload reference, a topic link, a resolved mention. Markbridge gives you everything you need to produce that placeholder, and to find every placeholder again afterwards by walking the parsed tree — no side channel required. The pattern looks like this: ![The placeholder flow: a source \[attachment=0\] tag becomes an AttachmentPlaceholder AST node, which the renderer Tag formats into a \[upload|HASH\] string in the Markdown output; the importer re-reads the same nodes via conversion.ast.descendants](/diagrams/placeholders.svg) Every placeholder is built from the same three parts: an **AST node** to hold the parsed data, a **parser handler** to build that node, and a **renderer Tag** to turn it into the placeholder string. When the importer later needs the details (which uploads a post used, which mentions to look up), it reads them back off the AST nodes — `conversion.ast` is the same tree that produced the Markdown, so `conversion.ast.descendants(YourPlaceholder)` hands you every placeholder that made it through parsing. > Markbridge ships with a built-in `Markbridge::AST::Attachment` that maps to Discourse’s resolved upload syntax. For migrations the source post doesn’t *have* a Discourse upload yet — it has a reference to a row in the source forum’s attachments table. Define your own AST class so the importer can resolve those references. ## The three parts, end to end Concrete example: phpBB3 emits attachments as `[attachment=N]filename[/attachment]`, where `N` is the position index of the attachment within the post (zero-based). The actual file lives in `phpbb_attachments` joined to the post; the BBCode just points at slot N. The pipeline below turns `[attachment=0]filename.jpg[/attachment]` into `[upload|]` in the output — Discourse’s upload-marker shape, with the upload identifier looked up from the source post’s attachment rows. The `upload_id` is whatever stable identifier the importer’s converter framework derives from the source filename/path (not the file’s content hash) so each placeholder maps unambiguously to one source-side row. (vBulletin’s `[ATTACH]N[/ATTACH]` and IPB’s `[attachment=N:filename]` follow the same shape — the example below adapts to either by tweaking the handler.) ### 1. The AST node ```ruby module ForumMigration class AttachmentPlaceholder < Markbridge::AST::Node attr_reader :position, :filename def initialize(position:, filename: nil) @position = position @filename = filename end end end ``` `Node` (rather than `Element`) makes it a leaf — the filename comes from the BBCode body but the handler captures it at parse time and pins it on the node, so there are no children to render. Pick whatever class name maps to your domain — `ForumMigration::AttachmentPlaceholder`, `Migration::SourceAttachment`, anything that won’t collide with built-in AST classes. ### 2. The parser handler ```ruby module ForumMigration class AttachmentHandler < Markbridge::Parsers::BBCode::Handlers::BaseHandler def initialize @element_class = AttachmentPlaceholder @collector = Markbridge::Parsers::BBCode::RawContentCollector.new end attr_reader :element_class def on_open(token:, context:, registry:, tokens: nil) position = Integer(token.attrs[:option] || token.attrs[:id]) filename = tokens && @collector.collect(token.tag, tokens).content context.add_child(AttachmentPlaceholder.new(position:, filename: presence(filename))) end # The collector consumes the closing tag; if one slips through, render it as literal text. def on_close(token:, context:, registry:, tokens: nil) context.add_child(Markbridge::AST::Text.new(token.source)) end private def presence(string) stripped = string&.strip stripped unless stripped.nil? || stripped.empty? end end end ``` `RawContentCollector` is the same helper Markbridge’s built-in `AttachmentHandler` uses to grab the body between `[attachment=0]` and `[/attachment]` as a literal string. Storing the filename on the node directly means the renderer Tag doesn’t have to call `render_children` later. Markbridge ships a handler for `[attachment]` / `[attach]` already (it builds `AST::Attachment`); registering this one overrides the defaults so they go through your migration-aware path. ### 3. The renderer Tag The Tag needs the post’s attachment list at render time so it can map a position to an `upload_id`. Pass it through the constructor. The Tag stays a one-line output formatter — no side effects, just a string: ```ruby module ForumMigration class AttachmentPlaceholderTag < Markbridge::Renderers::Discourse::Tag def initialize(attachments:) @attachments = attachments # ordered list, indexed by position end def render(element, _interface) attachment = @attachments[element.position] "[upload|#{attachment.upload_id}]" end end end ``` `[upload|]` is the upload-marker shape Discourse importers recognize — they substitute a real `upload://...` URL once the file has been ingested. ### 4. Wire it up The handler is stateless, so register it once and share it across every post. The renderer holds the per-post attachment list, so build it inside the migration loop: ```ruby handlers = Markbridge::Parsers::BBCode::HandlerRegistry.default.tap do |r| r.register(%w[attachment attach], ForumMigration::AttachmentHandler.new) end posts.each do |post| attachments = SourceDB.attachments_for(post.id) # ordered by attach_id renderer = Markbridge.discourse_renderer( tags: { ForumMigration::AttachmentPlaceholder => ForumMigration::AttachmentPlaceholderTag.new(attachments:), }, ) result = Markbridge.bbcode_to_markdown(post.body, handlers:, renderer:) # The placeholders that made it into the output are exactly the # AttachmentPlaceholder nodes left in the tree. Walk them to record # which uploads this post referenced. referenced = result.ast.descendants(ForumMigration::AttachmentPlaceholder) store(post, result.markdown, referenced) end ``` One handler instance is registered under both `[attachment]` and `[attach]`, so the closing logic finds the same object on both sides. When a tag has several names like this, you have to share one instance — [Extending Markbridge](/customization/extending/#wrapping-a-default-handler) explains why. Constructing the renderer per post breaks the build-once-reuse-many pattern, but only for the slice of state that varies post-to-post (the attachment table). Shared parts — handler registry, custom escaper, postprocessor, decorators that don’t depend on per-post data — stay outside the loop. ### 5. What you get ```ruby attachments = [SourceDB::Attachment.new(upload_id: "u_screenshot_png_3a7c2", filename: "screenshot.png")] renderer = Markbridge.discourse_renderer( tags: { ForumMigration::AttachmentPlaceholder => ForumMigration::AttachmentPlaceholderTag.new(attachments:), }, ) handlers = Markbridge::Parsers::BBCode::HandlerRegistry.default.tap do |r| r.register(%w[attachment attach], ForumMigration::AttachmentHandler.new) end result = Markbridge.bbcode_to_markdown( "Screenshot: [attachment=0]screenshot.png[/attachment] — see what I mean?", handlers:, renderer:, ) result.markdown # => "Screenshot: [upload|u_screenshot_png_3a7c2] — see what I mean?" # Re-read the placeholder nodes straight off the tree. result.ast.descendants(ForumMigration::AttachmentPlaceholder).map(&:position) # => [0] ``` The importer downstream substitutes `[upload|u_screenshot_png_3a7c2]` for the resolved `upload://...` URL once the file is ingested into Discourse’s upload store. ## Placeholder strings pass through verbatim A common worry: “if my placeholder contains `[`, won’t the Markdown escaper mangle it?” It will not. Markbridge escapes only `AST::Text` nodes — the textual content from the source document. A Tag’s return value is spliced into its parent’s output with no transformation. Whatever you return from `Tag#render` is exactly what appears in the surrounding Markdown. This is what makes placeholders safe. `"[upload|u_screenshot_png_3a7c2]"`, `"@@MENTION:alice@@"`, `"<>"` all reach the output untouched. Pick whatever sigil pattern your importer parses cleanly downstream. The one twist is HTML mode (next section). ## HTML mode and placeholders When a parent renders an HTML block — currently only `TableTag` doing its HTML-fallback path for uneven rows or nested tables — children render with `interface.html_mode?` true. Per [CommonMark §4.6](https://spec.commonmark.org/0.31.2/#html-blocks), content inside an HTML block is treated as raw HTML, not Markdown, until the next blank line. Your placeholder Tag has two valid choices: **Raw HTML.** If your placeholder has a natural HTML form, emit it directly: ```ruby module ForumMigration class AttachmentPlaceholderTag < Markbridge::Renderers::Discourse::Tag def render(element, interface) attachment = @attachments[element.position] if interface.html_mode? %(attachment) else "[upload|#{attachment.upload_id}]" end end end end ``` **Markdown island.** If your placeholder is an opaque sigil that downstream tooling parses regardless of context, wrap it in blank lines so CommonMark closes the HTML block, parses your placeholder as Markdown, and re-opens it: ```ruby upload_id = "u_screenshot_png_3a7c2" "\n\n[upload|#{upload_id}]\n\n" ``` The blank lines force a paragraph break around the placeholder, which is fine for block-level placeholders (uploads in tables tend to want their own row anyway) but unsightly for inline ones (mentions, links). Prefer the raw-HTML form for inline placeholders that can land inside tables. ## Collecting more than one kind of placeholder A migration usually has several placeholder concepts in flight at once — uploads, internal links, mentions. Each gets its own AST class, and the importer pulls each kind off the tree independently after the conversion: ```ruby links = result.ast.descendants(ForumMigration::InternalLink) mentions = result.ast.descendants(ForumMigration::Mention) # Reconcile whatever you need — e.g. flag links whose source topic # didn't resolve to a Discourse topic yet. unresolved = links.reject { |link| RESOLVED_TOPICS.key?(link.source_topic_id) } ``` Because the tree is the record, you never have to keep the collected data balanced or paired the way a side-channel buffer would force you to — each query is independent and reads exactly the nodes that reached the output. ## Resolution: handler vs Tag Where should the source-id-to-Discourse-id resolution happen — in the handler (parse time) or the Tag (render time)? | Resolution at parse | Resolution at render | | ----------------------------------------------------------------------- | ----------------------------------------------------------------- | | Handler does the lookup; AST node carries the resolved Discourse value. | Handler stores the source value; Tag does the lookup at render. | | Failures surface as missing AST nodes (or `unknown_tags` bumps). | Failures surface when you walk the tree afterwards and reconcile. | | Cleaner separation; the renderer is dumb. | Lets you batch lookups across many calls or defer entirely. | The forum-migration tradeoff usually goes: simple lookups (path → slug, name → user\_id) at parse time; lookups that require global state (cross-post topic IDs) at render time, with a second pass over `conversion.ast.descendants(...)` to reconcile the references that need it. There’s no one right answer — pick the side that matches your data flow. ## What goes in the AST node A few rules of thumb for the placeholder AST node: * Carry everything the importer needs to reconcile the placeholder later (position, original URL, source id, filename). The node *is* the record you read back via `descendants`, so don’t drop data you’ll want downstream. * Use `attr_reader` and a keyword constructor. Mutability has no upside here. * Inherit from `Element` if it can wrap inline content (link text, mention name). Inherit from `Node` if it’s a leaf (a poll, an event, a hr-style separator). * One AST class per *concept*, not per *source tag alias*. `[url]`, `[link]`, and `[iurl]` all build `AST::Url`; the same Tag renders all three. The same applies to placeholders that have multiple aliases in the source format. ## Where next * [Customizing the renderer](/customization/customizing-renderer/) — the factory kwargs in detail. * [Extending Markbridge](/customization/extending/) — broader extension patterns including `HandlerRegistry#overlay` for delegating to default handlers.