How We Built a 15,000-Entry AI Tools Directory (Tech Stack & Lessons)

Matty Reid · Founder & Lead Engineer · March 26, 2026 · 15 min read

---

TL;DR — The Stack

Total build time from zero to 15,000+ entries: 10 days. Total monthly hosting cost: under $20.

---

Table of Contents

1. Why We Built This
2. The Data Problem: 11 Sources, One Directory
3. The Scraping Pipeline
4. Database Architecture: Why Convex
5. Frontend: Next.js 15 and the ISR Decision
6. Search: Fuse.js Over Everything Else
7. SEO Strategy: How We Rank for 2,000+ Keywords
8. Monetization: Store, Sponsorships, and Affiliates
9. Mistakes We Made (and You Should Avoid)
10. Frequently Asked Questions

---

Why We Built This {#why-we-built-this}

I could not find what I was looking for. That is the honest origin story.

In early 2026, the Claude Code ecosystem was exploding. There were 60,000+ published skills, 12,000+ MCP servers, thousands of agents, commands, and hooks. But there was no single place that cataloged all of them. If you wanted to find a Terraform skill or a Postgres MCP server, you had to search across GitHub, npm, Smithery, PulseMCP, awesome-lists, and Twitter threads.

The existing directories were fragmented. Smithery focused on MCP servers only. SkillsDirectory covered skills only. The awesome-mcp-servers GitHub repo was a flat markdown list with no search, no categories, and no metadata beyond a name and URL.

I wanted one directory that had everything — skills, MCP servers, agents, commands, hooks — with proper categorization, search, and enough metadata to make informed decisions. And I wanted it to be fast. Not "load a huge JSON file and hope for the best" fast, but "statically generated with sub-100ms page loads" fast.

So I built it. Here is exactly how.

---

The Data Problem: 11 Sources, One Directory {#the-data-problem}

The first challenge was data acquisition. The AI tools ecosystem is not neatly organized. Tools are scattered across:

1. awesome-mcp-servers (GitHub) — The largest community-maintained MCP list. Markdown format with basic metadata.
2. Smithery — A curated MCP registry with structured data (descriptions, install commands, categories).
3. MCP.so — Another MCP directory with its own categorization.
4. npm registry — Many MCP servers and skills are published as npm packages.
5. PyPI — Python-based MCP servers and tools.
6. LobeHub — Agent and plugin registry with detailed metadata.
7. PulseMCP — MCP-focused directory with compatibility information.
8. awesome-claude-skills (GitHub) — Community skills list, markdown format.
9. SkillsDirectory — Dedicated skills catalog.
10. OneSKILL — Another skills catalog with different coverage.
11. GitHub API search — Direct searches for repositories tagged with mcp-server, claude-skill, claude-code-agent, etc.

Each source has different data quality, different schemas, and different update frequencies. Some give you a name and URL. Others give you a full description, install command, GitHub stars, last updated date, and compatibility information.

The goal was to ingest everything, deduplicate aggressively, and normalize into a consistent schema that the frontend could consume.

---

The Scraping Pipeline {#the-scraping-pipeline}

The pipeline is Python. I considered Node.js (to keep everything in one ecosystem) but Python's scraping libraries are just better. BeautifulSoup, httpx, and the GitHub API client made the work straightforward.

The Pipeline Architecture

[11 Data Sources]
       ↓
[Source-specific scrapers]  ← Each source has its own parser
       ↓
[Normalization layer]       ← Maps to unified schema
       ↓
[Deduplication engine]      ← URL matching, fuzzy name matching
       ↓
[Enrichment layer]          ← GitHub stars, last commit, README extraction
       ↓
[Convex upload]             ← Batch upsert to database

Source-Specific Scrapers

Each source gets its own scraper because every source has a different format:

• GitHub awesome-lists: Parse markdown, extract links and descriptions from list items
• Smithery/MCP.so/PulseMCP: HTTP requests to their APIs or scrape their HTML
• npm/PyPI: Package registry API queries with keyword filters
• GitHub API: Search repositories by topic tags, extract README content

The scrapers are deliberately simple. Each one outputs a list of dictionaries with the same keys: name, url, description, source, category, install_command, github_url, github_stars, last_updated.

Deduplication

This is where most of the complexity lives. The same tool often appears in multiple sources with slightly different names, URLs, and descriptions.

The deduplication engine uses three strategies:

1. Exact URL match: If two entries point to the same GitHub repo or npm package, they are the same tool. Merge metadata, keep the richest description.
2. Normalized name match: Strip prefixes like @modelcontextprotocol/, mcp-server-, claude-skill-. If the remaining name matches, flag as potential duplicate for manual review.
3. Fuzzy name match (Levenshtein): Catch cases like postgres-mcp vs postgresql-mcp-server. Threshold of 0.85 similarity triggers a manual review flag.

Automated merges handle about 70% of duplicates. The remaining 30% go through a manual review queue — a simple CLI tool that shows me two entries side by side and lets me merge or skip.

After deduplication, 15,000+ entries remained from roughly 25,000 raw scraped entries. That is a 40% duplicate rate across the ecosystem, which tells you something about how fragmented the tooling landscape is.

Enrichment

After deduplication, the enrichment layer adds metadata that no single source provides:

• GitHub stars and forks (from GitHub API)
• Last commit date (staleness indicator)
• README first paragraph (used as description if the source description was empty)
• Package download counts (from npm/PyPI)
• License type (from GitHub API)

This enrichment runs weekly to keep data fresh. The entire pipeline — scrape, normalize, deduplicate, enrich, upload — takes about 45 minutes on a single machine.

---

Database Architecture: Why Convex {#database-architecture}

I chose Convex over Supabase, PlanetScale, and raw Postgres. Here is why.

The Requirements

1. TypeScript-native schema. The frontend is Next.js with TypeScript. I wanted the database schema to live in TypeScript, not SQL, so that type safety flows from database to UI without an ORM translation layer.
2. Real-time subscriptions. When I update entries from the scraping pipeline, the directory pages should update without a full rebuild. Convex subscriptions handle this natively.
3. Serverless functions. I did not want to build and deploy a separate API layer. Convex functions run on their infrastructure and are called directly from the frontend.
4. Full-text search. Convex has built-in search indexes. I ended up using Fuse.js for the primary search UX, but Convex search is useful for admin queries and filtering.

The Schema

// Simplified schema
defineSchema({
  tools: defineTable({
    name: v.string(),
    slug: v.string(),
    type: v.union(
      v.literal("skill"),
      v.literal("mcp"),
      v.literal("agent"),
      v.literal("command"),
      v.literal("hook")
    ),
    description: v.string(),
    url: v.string(),
    githubUrl: v.optional(v.string()),
    githubStars: v.optional(v.number()),
    installCommand: v.optional(v.string()),
    category: v.string(),
    tags: v.array(v.string()),
    lastUpdated: v.string(),
    source: v.string(),
    featured: v.boolean(),
  })
    .index("by_type", ["type"])
    .index("by_category", ["category"])
    .index("by_slug", ["slug"])
    .searchIndex("search_name_desc", {
      searchField: "name",
      filterFields: ["type", "category"],
    }),
})

Five indexes cover every query pattern the frontend uses: type-filtered browsing (all skills, all MCPs), category pages, individual tool pages by slug, and search.

The Cost Reality

Convex's free tier handles the current data volume comfortably. At 15,000 entries with an average of 500 bytes per document, the total storage is about 7.5MB. Read queries are in the low thousands per day. Function invocations are well within free tier limits.

If traffic scales 10x, the Pro plan at $25/month covers it. If it scales 100x, we are looking at $100-200/month — still far cheaper than running a separate database and API server.

---

Frontend: Next.js 15 and the ISR Decision {#frontend-architecture}

Why Not Server-Side Rendering

This is the single most important architectural decision we made, and it was driven by a painful lesson.

Early in development, I used force-dynamic on several routes so that pages would always show the latest data. On a site with 15,000+ pages, this meant every page request hit the Convex database. Vercel charges for serverless function invocations, and the bill climbed to $4.20 in a single day during a traffic spike.

I ripped out every force-dynamic directive and switched to static generation with ISR (Incremental Static Regeneration). Here is how it works:

• Category pages (/skills, /mcps, /agents, etc.) are statically generated at build time and revalidate every hour
• Individual tool pages (/skills/superpowers, /mcps/github, etc.) use ISR with a 1-hour revalidation window
• Blog posts are fully static — no revalidation needed since content does not change after publish
• The home page revalidates every 30 minutes for fresh "featured" and "trending" sections

This approach keeps the Vercel bill under $20/month while serving pages in under 100ms globally.

The Route Structure

/                          ← Home (featured tools, search)
/skills                    ← All skills (filterable, searchable)
/mcps                      ← All MCP servers
/agents                    ← All agents
/commands                  ← All commands
/hooks                     ← All hooks
/skills/[slug]             ← Individual skill page
/mcps/[slug]               ← Individual MCP page
/blog                      ← Blog index
/blog/[slug]               ← Blog post
/store                     ← Digital products store
/store/[slug]              ← Product page

Eleven route patterns generate 15,000+ pages. Every page has unique metadata, Open Graph tags, and JSON-LD schema markup. The sitemap is generated at build time and submitted to Google Search Console.

---

Search: Fuse.js Over Everything Else {#search-architecture}

Why Client-Side Search

For a directory with 15,000 entries, you have three search options:

1. Server-side full-text search (Convex search, Algolia, ElasticSearch) — fast, powerful, costs money per query
2. API-mediated search — server function that queries the database, adds latency and cost
3. Client-side fuzzy search — download the search index to the client, search locally, zero API cost

I chose option 3. Here is why it works at this scale.

The search index contains only three fields per entry: name, description (first 100 characters), and type. At 15,000 entries, this index is approximately 1.2MB — well within acceptable page weight, especially when gzip-compressed to about 350KB.

Fuse.js runs the fuzzy matching entirely in the browser. Search results appear as the user types, with zero network round-trips. It handles typos, partial matches, and ranking by relevance.

The Tradeoff

Client-side search breaks at around 50,000-100,000 entries. If the directory grows beyond that, I will switch to Algolia or Convex's search index. But at current scale, Fuse.js delivers the fastest possible search UX — zero latency, zero server cost.

---

SEO Strategy: How We Rank for 2,000+ Keywords {#seo-strategy}

The directory alone is not enough for SEO. Google does not rank thin directory pages well — a page with a name, description, and install command does not satisfy search intent for most queries.

The strategy is a funnel:

Blog posts (deep, long-form content)
    → Link to directory pages (browse by category)
        → Link to individual tool pages (specific details)
            → Link to store (buy skill packs)

Blog Content Types

We publish four types of blog posts, each targeting different keyword clusters:

1. "Best X for Y" listicles — Best Claude skills for developers, best MCP servers for data engineers, best Claude skills for DevOps. These rank for high-intent search queries and link heavily to directory pages.

1. "What is X" guides — What are Claude skills, what is MCP. These rank for informational queries and establish topical authority.

1. "How to" tutorials — How to install Claude skills, how to build an MCP server. These rank for problem-solving queries and demonstrate expertise.

1. Industry analysis — State of AI agent tools 2026, ecosystem comparisons. These build authority and attract backlinks.

Every blog post includes internal links to relevant directory pages (skiln.co/skills, skiln.co/mcps) and to other blog posts. This creates a tightly interlinked content cluster that signals topical authority to Google.

Technical SEO

Every page on Skiln has:

• Unique