Content Inventory With an AI Agent: Reading an Old Site Before You Rebuild It

A content inventory is a spreadsheet with one row per URL on the old site and a decision for each row: keep, merge, rewrite or drop. It is the first document of a migration and the one most people skip. An AI agent can read the old site and draft that spreadsheet in hours rather than days. You still make the decisions, but you make them against a complete list instead of memory.

This post is part of our guide to Migrating a Website With AI.

Why the inventory comes before the build

A migration goes wrong in one of two ways. Either a page that ranked is left behind without a redirect, or a page that nobody needed is rebuilt and carried forward. Both come from not knowing what the old site contains. Google's documentation on site moves with URL changes starts with the same step: get a list of your current URLs, then map each one to its new location. Everything else, the redirects, the new sitemap, the monitoring, is built on that list.

The common shortcut is rebuilding from the navigation menu. The menu shows what the site wants visitors to see, not what search engines send them to: the old articles, the PDFs other sites link to, the posts that ranked for years without anyone touching them. The inventory is how you find those pages before launch instead of a month after.

It is also worth building when you are not sure you will move. If a week like the October 2026 PHP security release has you asking whether your WordPress site should stay where it is, the inventory is the document that answers it: how many pages, how many earn traffic, how many depend on a plugin. Migrating From WordPress After PHP2Shell covers what that release does and does not mean for a move.

Where the URL list comes from

No single source has every URL. Google's site move guide lists several, and the agent can combine them:

• The sitemap. Usually at /sitemap.xml. It lists what the old platform thinks matters. It rarely lists everything.
• Search Console. Export the pages with impressions and clicks over the last 12 months. This is the list you must protect. It includes URLs the sitemap left out.
• Analytics. Landing pages over the same period catch URLs that get traffic from links and email rather than search.
• A crawl of the site. Follow every internal link from the home page. This catches tag pages, author pages, paginated archives and old PDFs.
• The CMS export. A WordPress export file lists posts and pages with their slugs, including drafts and private pages that never appeared anywhere else.
• The old redirects. If the old platform or a plugin already redirects some paths, export that list. Those are URLs that still get visits even though no page lives there.

Give the agent the exports as files and ask it to merge them into one list, deduplicated, with a column for where each URL came from. A URL that appears in Search Console but not in the sitemap is a URL the old platform was hiding from you.

What each row needs

The inventory is only useful if each row carries enough to decide on. A working set of columns:

The last two columns are the point. The agent fills the first seven. You go down the list and decide. It is the slowest part of the inventory for you, and the most valuable part of the whole project.

Here is what a few finished rows look like for a small accounting firm, so the shape is clear:

Two of those rows show the judgement the agent cannot make. The 2021 deadline post has almost no traffic and is out of date, so it goes. The short VAT basics post has some traffic, but the long guide covers the same ground better, so it is merged and redirected there. Neither decision comes from a threshold. The file row is easy to forget: a PDF with 210 clicks a year is being linked from somewhere, and it needs a new home and a redirect like any page.

How the agent reads the site

With an agent such as Claude Code or Codex, the method is: give it the merged URL list, ask it to fetch each page, and have it extract the title, headings, body text, images and internal links into a folder of Markdown files, one per URL. That folder becomes the raw material for the rebuild. Each file should keep the old URL in its front matter, because the redirect map is built from it later. A terminal agent suits this job because it works on files; which client fits which kind of website work compares them.

A starting instruction, which you can adapt:

Merge sitemap.xml, search-console-pages.csv, analytics-landing-pages.csv and
crawl.csv into inventory.csv with one row per unique path. Columns: old_url,
title, type, word_count, last_updated, clicks_12m, sources. Then fetch every
URL and save it as Markdown in /pages, one file per path, with the old URL
and the meta description in front matter and the body text exactly as
published. Do not rewrite anything. Put any URL you could not fetch in
failed.csv with the status code, and any page under 100 words in thin.csv.
Leave the decision and new_url columns empty.

Three things that instruction asks for explicitly, and why:

1. Keep the text as written. The agent should extract, not improve. Rewriting happens later, on purpose, page by page.
2. Record what it could not fetch. Pages behind a login, pages that returned errors, pages blocked by a bot rule. These go in a separate list, not silently dropped.
3. List the images with their alt text and the URL they were served from. Images are part of the move. Google's site move guide says to include embedded content such as images and videos in the plan, and you will need the originals when the old host goes away.

How you set these expectations up for every session, not just this one, is a rules question. Writing Rules for Your Website Agent covers the file where instructions like these live.

The decisions, and how to make them

Four decisions cover most rows:

• Keep. The page earns traffic or serves a purpose. It moves to a new URL, ideally the same path.
• Merge. Three thin posts on the same topic become one page. Each old URL redirects to the merged page. Google's site move guide accepts this when the content is consolidated, and warns against sending many old URLs to one irrelevant page such as the home page.
• Rewrite. The page stays, the content is dated. Mark it, move it as is, rewrite it after launch. Do not rewrite and migrate in the same week. Google's site move guide advises planning changes one after the other rather than all at once.
• Drop. No traffic, no links, no purpose. It returns 404 on the new site, and that is fine. The Page indexing report treats a 404 as correct for a page removed without a replacement.

The agent can propose a decision for each row from the numbers, for example "drop if zero clicks and zero links in 12 months". Treat the proposal as a sort order, not a verdict. The page with two clicks a year might be the one your best client reads before every renewal.

What the agent gets wrong

An agent reads what the server returns. It does not know that the pricing page is out of date, that the team page lists someone who left, or that a post was written for a campaign that ended. It also cannot see traffic you did not give it. The inventory is as good as the exports you attach.

Two more limits. Pages rendered entirely by JavaScript on the old platform may come back empty; check a sample by hand. And paginated archives, tag pages and search result pages look like content to a crawler but usually should not be rebuilt. Mark the type column carefully and decide on those as a group.

The inventory after the build

The inventory does not retire when the rebuild starts. It becomes the checklist the new site is measured against. Before launch, ask the agent to read the draft site and confirm three things against it: every row marked keep has a page at its new URL, every page on the draft is in the inventory (anything extra was invented), and each new page carries the meta title and description from the old one unless you changed it on purpose. That comparison is how you find the page that was quietly skipped and the one that was quietly made up.

What comes next

With the inventory done, two things follow from it directly. The "New URL" column becomes the redirect map, which is the subject of Redirect Mapping From a CSV. And the folder of extracted pages becomes the input to the rebuild, whether that is a prompt that describes the business or page-by-page recreation. If the old site is WordPress, Migrating From WordPress After PHP2Shell covers which parts of the export are worth bringing and how the blog posts can be imported directly; for Wix and Squarespace, see Migrating From Wix or Squarespace.

How this works in Beste

Beste does not read your old site for you; that is the agent's job. What Beste provides is the other end of the pipe. An agent connected over the MCP server can create pages and posts from the inventory with create_page and create_post, one per row marked keep, with the slug set to match the old path. Nested pages reproduce most old paths exactly, so rows whose new URL equals the old one need no redirect at all. Reads do not count toward any monthly limit, so the agent can run the before-launch comparison above as often as it needs to, listing the draft's pages and reading each one as Markdown. If the old blog is on WordPress, posts can be imported from the Posts panel in Studio, with their images and slugs, instead of recreated one by one. For the rest of the site, the extracted folder can feed a build from a prompt, or the agent can rebuild page by page. Either way the result is a draft, nothing goes live until you publish, and the inventory's "New URL" column is ready for the redirect import described on the technical SEO page. The broader method of running a site through an agent is in the MCP guide.

Sources

Checked on 23 October 2026.

• Google Search Central: Site moves with URL changes
• Google Search Central: Redirects and Google Search
• Search Console Help: Page indexing report
• Beste: MCP Server & Tools
• Beste: Build From a Prompt
• Beste: Technical SEO Controls