changelog-testing-report

Better Changelog — Testing Report & Plan

Date: 2026-02-04
URL: https://web-five-phi-53.vercel.app
Repo: project-shovels/better_changelog (org) / Chaoticonomist/better_changelog (fork)
Stack: Next.js 16.1.6, TypeScript, Turborepo, Drizzle ORM, PostgreSQL (Neon), Clerk auth, Tailwind+shadcn

---

1. Current State Summary

What Exists (Codebase)

The app is a well-structured multi-tenant changelog SaaS with a surprisingly complete schema and feature set for a prototype:

• 13 database tables all created and provisioned in Neon (tenants, users, tenant_members, changelog_entries, entry_audiences, entry_platforms, entry_product_areas, entry_tags, product_areas, subscribers, webhooks, api_keys, analytics_events)
• Landing page — Clean marketing homepage with feature cards (filters, AI writing, distribution, analytics, API, RSS)
• Auth — Clerk-based sign-in/sign-up with auto-provisioning of user + default tenant on first login
• Dashboard — Protected by auth, with sidebar navigation, entry table with status tabs (All/Draft/Scheduled/Published/Archived), search, and CRUD operations
• Entry form — Full-featured: title, slug (auto-generated), emoji, summary, body (Markdown), update type, impact level, status, audiences, platforms, product areas, tags, SEO fields, scheduling
• Public changelog — /changelog route with filter bar (type chips, search, date period), entry cards grouped by month, entry detail pages with Markdown rendering and JSON-LD
• RSS/Atom/JSON feeds — Three feed formats at /changelog/feed.rss, /feed.atom, /feed.json
• Multi-tenant architecture — Subdomain + custom domain resolution via middleware, tenant context via headers
• Role-based access — owner, admin, editor, viewer with permission helpers

What’s Deployed (Live Site)

Page	Status	Notes
/ (homepage)	✅ Working	Renders fully, all links present
/sign-in	✅ Working	Clerk widget loads (returns minimal HTML to web_fetch but JS renders it)
/sign-up	✅ Working	Same as sign-in — Clerk JS widget
/dashboard	✅ Auth redirect working	Redirects to /sign-in?redirect_url=... — correct behavior
/changelog	❌ 404	Fails because no tenant can be resolved (no “demo” tenant slug, no DEFAULT_TENANT_SLUG env var, no subdomain)
/changelog/feed.rss	❌ 404	Same root cause — tenant resolution fails
/features	❌ Likely 404	Link exists on homepage but no page component exists
/pricing	❌ Likely 404	Same — link but no page
/docs	❌ Likely 404	Same
/demo	❌ Likely 404	”View demo” button links here but no page exists

Database State

• 2 tenants created: roberts-john-adam and roberts-john-adam-test
• 2 users with Clerk IDs linked
• 2 tenant memberships (both owner role)
• 0 changelog entries — completely empty
• 0 everything else — no product areas, subscribers, webhooks, API keys, analytics events

Git History

• 20 recent commits showing active development
• Recent work focused on: Next.js 16 migration, Clerk auth fixes, middleware/proxy refactoring, Vercel deployment fixes, auto-provisioning on first sign-in
• CVE patches applied (Next.js 16.0.7 → 16.1.6)

---

2. Critical Bugs & Issues

🔴 P0 — Blocking

1. /changelog returns 404 — The public changelog page (the entire point of the product) doesn’t work on the deployed site. Root cause: the getTenant() function falls back to getTenantBySlug("demo") but no tenant with slug “demo” exists. The DEFAULT_TENANT_SLUG env var isn’t set on Vercel either.

   • Fix: Either set DEFAULT_TENANT_SLUG=roberts-john-adam in Vercel env vars, or change the fallback logic to load the first available tenant.

2. RSS/Atom/JSON feeds return 404 — Same root cause as above (tenant resolution failure).

🟠 P1 — Important

3. Dead links on homepage — /features, /pricing, /docs, /demo are all linked from the homepage but have no corresponding pages. Users clicking these hit 404s.

4. No changelog entries exist — Even if tenant resolution is fixed, the changelog page would show “No updates found” with zero content. Need seed data or the ability to create entries through the dashboard.

5. /dashboard/settings route doesn’t exist — The sidebar links to it, but there’s no apps/web/src/app/dashboard/settings/ page. Clicking it will 404 inside the dashboard.

🟡 P2 — Should Fix

6. Tenant resolution is fragile — The multi-tenant system relies on subdomain routing or custom domains, but the Vercel deployment uses a generic web-five-phi-53.vercel.app domain with no subdomain support. The middleware tries subdomain matching against NEXT_PUBLIC_APP_DOMAIN which likely isn’t configured.

7. No AI features implemented — The homepage advertises “AI-Powered Writing” but there’s zero AI code in the codebase. No OpenAI integration, no AI packages in dependencies. This is the app’s biggest differentiator per the competitive analysis.

8. No embeddable widget — Advertised on homepage (“Multi-Channel Distribution”, “embeddable widgets”) but not implemented.

9. No subscriber management UI — The subscribers table exists but there’s no UI to manage subscribers, no subscription form, no email integration.

10. Entry slug detail page uses /changelog/[slug] — The “View” action in the dashboard links to /changelog/${entry.slug} but the changelog pages have tenant resolution issues.

---

3. Comprehensive Testing Plan

3.1 Authentication Flow

• Sign up with email — Visit /sign-up, create new account via Clerk
  • Verify Clerk widget renders correctly
  • Complete sign-up flow
  • Verify auto-provisioning creates user record in users table
  • Verify auto-provisioning creates tenant and membership
  • Verify redirect to /dashboard after sign-up
• Sign in with existing account — Visit /sign-in
  • Verify Clerk widget renders
  • Sign in with existing credentials
  • Verify redirect to /dashboard
  • Verify active_tenant_id cookie handling
• Sign out — From dashboard
  • Verify Clerk sign-out works
  • Verify redirect to homepage
  • Verify dashboard is inaccessible after sign-out
• Auth protection — Try accessing /dashboard while logged out
  • ✅ Verified: redirects to /sign-in?redirect_url=... (working)
• Multiple accounts — Test with second Clerk user
  • Verify separate tenant created
  • Verify data isolation between tenants

3.2 Dashboard — Entry Management

• Dashboard loads — After auth, verify dashboard page renders
  • Sidebar renders with tenant name
  • Header renders with user name
  • Status tabs show (All/Draft/Scheduled/Published/Archived)
  • Empty state shows “No entries yet” with CTA
• Create entry — Click “New Entry”
  • Form loads at /dashboard/entries/new
  • Fill in title → verify slug auto-generates
  • Add emoji, summary, markdown body
  • Select update type (all 8 types)
  • Select impact level (major/minor/patch)
  • Add audiences (multi-select)
  • Add platforms (multi-select)
  • Add tags (tag input with add/remove)
  • Fill SEO fields (title, description, OG image URL)
  • Set status to Draft → submit → verify entry appears in dashboard
  • Set status to Published → verify published_at is set
  • Set status to Scheduled → verify datetime picker appears
• Edit entry — Click on existing entry
  • Form loads with pre-populated data
  • Modify fields and save
  • Verify changes persist
• Delete entry — From action dropdown
  • Confirm dialog appears
  • Delete succeeds
  • Entry removed from table
  • Cascading deletes clean up audiences/platforms/tags
• Status filtering — Click status tabs
  • “Draft” tab shows only drafts
  • “Published” tab shows only published
  • “All” tab shows everything
  • Counts update correctly
• Search — If search param is supported
  • Search by title works
  • Search by summary works

3.3 Public Changelog

⚠️ Blocked by P0 bug #1 — Requires tenant resolution fix first

• Changelog page loads — Visit /changelog
  • Tenant name and logo display
  • RSS button visible
  • Subscribe button visible
  • Filter bar renders (type chips, search, period selector)
• Entry listing — With published entries
  • Entries display grouped by month
  • Entry cards show emoji, title, type badge, date
  • Empty state shows correctly when no entries match
• Filtering — Test all filter combinations
  • Type chips toggle (New, Improved, Fixed, Security, Deprecated, Breaking)
  • Search input filters by title/summary/body
  • Period selector (7d, 30d, 90d, All time)
  • “Clear all” button resets filters
  • URL updates with query params on filter change
• Entry detail page — Click an entry
  • Full markdown renders (via ReactMarkdown + remark-gfm)
  • Type badge, date, author info display
  • Tags and platforms show in footer
  • “Back to all updates” link works
  • JSON-LD structured data present in page source
  • SEO metadata (title, description, OG tags) correct

3.4 RSS/Atom/JSON Feeds

⚠️ Blocked by P0 bug #1 — Requires tenant resolution fix first

• RSS feed — GET /changelog/feed.rss
  • Returns valid RSS 2.0 XML
  • Content-Type is application/rss+xml
  • Cache-Control header set (1 hour)
  • Entries include title with emoji prefix
  • Feed links point to correct URLs
• Atom feed — GET /changelog/feed.atom
  • Returns valid Atom XML
• JSON feed — GET /changelog/feed.json
  • Returns valid JSON Feed format
  • Content-Type is application/json

3.5 Multi-Tenant System

• Subdomain routing — {slug}.{APP_DOMAIN}/changelog
  • x-tenant-slug header set by middleware
  • Correct tenant loaded
• Custom domain routing — Non-matching hostname
  • x-custom-domain header set
  • Tenant looked up by custom domain
• Tenant isolation — Two tenants
  • User A can’t see User B’s entries
  • User A can’t edit User B’s entries (even via direct API call)
  • Entry creation always uses the authenticated user’s tenantId
• Role-based access
  • Owner can: edit, delete, manage team, manage settings, manage billing
  • Admin can: edit, delete, manage team, manage settings
  • Editor can: edit (not delete)
  • Viewer can: view only (no edit form)

3.6 UI/UX Review

• Landing page
  • Hero section: clear value prop, CTA buttons work
  • Feature cards: 6 cards render with icons
  • Footer: branding, tagline
  • Navigation: Sign in / Get started buttons
  • Dead links: /features, /pricing, /docs, /demo all 404 ← fix or remove
• Dashboard UX
  • Sidebar is fixed-width (256px), scrollable if needed
  • Entry table: columns are Title, Type, Status, Updated, Actions
  • Dropdown actions: Edit, View (if published), Delete (if owner/admin)
  • Delete confirmation dialog
  • Loading states exist (Suspense fallbacks)
• Entry form UX
  • Auto-slug from title (only on create, not edit)
  • Character limits enforced (summary 300, SEO title 60, SEO desc 160)
  • Validation errors display inline
  • Cancel button navigates back
  • Submit button shows loading state
  • Tag input: Enter key adds tag, × removes it
  • Multi-select components for audiences and platforms

3.7 Mobile Responsiveness

• Landing page — Test at 375px, 768px, 1024px widths
  • Nav items hidden on mobile (md breakpoint), sign-in/get-started visible
  • Hero text scales down
  • Feature grid: 1 col mobile, 2 col tablet, 3 col desktop
• Dashboard — Test at 375px
  • Sidebar may overlap or not be usable (no mobile hamburger menu!)
  • Entry table horizontal scroll or column collapse
  • Entry form fields stack properly
• Changelog page — Test at 375px
  • Filter bar wraps properly
  • Entry cards full-width on mobile
  • Search + period selector stack on mobile (sm breakpoint)
• Entry detail — Test at 375px
  • Prose content doesn’t overflow
  • Tags wrap correctly

3.8 Performance

• Homepage load time — Target < 2s TTFB
  • No heavy client-side JS (landing page is mostly SSR)
• Dashboard load time — Measure with entries
  • Default limit of 50 entries — fine for MVP
  • No pagination implemented yet
• Changelog page load time
  • All DB queries use proper indexes (✅ indexes defined in schema)
  • Suspense boundary on filter bar
• Feed generation — With many entries
  • Capped at 50 entries (good)
  • 1-hour cache (good for MVP)
• Bundle size — Check for unnecessary imports
  • react-markdown + remark-gfm + rehype-raw + rehype-sanitize — present in deps but rehype-raw/sanitize not imported yet (dead dependencies?)
  • feed library — lightweight, fine

3.9 Security Considerations

• Auth enforcement — All dashboard routes require auth ✅
  • Middleware redirects unauthenticated users
  • Server actions call requireRole() before DB operations
• Tenant isolation — All queries filter by tenantId ✅
  • createEntry uses member.tenantId (not user-supplied)
  • updateEntry verifies entry belongs to tenant before updating
  • deleteEntry verifies entry belongs to tenant before deleting
• Input validation — Zod schemas validate all inputs ✅
  • Title: 1-255 chars
  • Slug: regex validated
  • Summary: max 300 chars
  • Body: required
  • URL fields: validated as URLs
• XSS prevention
  • Markdown rendering uses ReactMarkdown (safe by default)
  • rehype-sanitize is in dependencies but NOT imported in the slug page — potential XSS risk if rehype-raw is added later
  • JSON-LD uses dangerouslySetInnerHTML — entry data could inject scripts if title/summary contain </script> ← needs review
• SQL injection — Drizzle ORM parameterizes all queries ✅
• CSRF — Server Actions are POST-only with Clerk session ✅
• Secrets exposure
  • .env files in .gitignore ✅
  • Clerk keys use NEXT_PUBLIC_ prefix only for publishable key ✅
  • DATABASE_URL is server-only ✅
• Rate limiting — Not implemented (acceptable for MVP)
• Image domains — Limited to clerk.com and cloudflare.com ✅

---

4. What Works Well ✅

1. Solid database schema — Multi-tenant, well-normalized, proper indexes, good use of many-to-many tables for audiences/platforms/tags
2. Auth flow is correct — Auto-provisioning on first sign-in is elegant; role-based access is well-designed
3. Entry form is comprehensive — All the fields a real changelog tool needs: emoji, markdown body, categorization, SEO, scheduling
4. Feed implementation — Three formats (RSS, Atom, JSON) with proper caching headers
5. Code quality — TypeScript throughout, Zod validation, proper error handling in server actions, clean component architecture
6. Security fundamentals — Tenant isolation in every query, input validation, auth on protected routes

5. What’s Missing / Broken ❌

1. Public changelog 404 — The core product page doesn’t work (P0)
2. No content — Zero entries, zero product areas. The app needs seed data or a “getting started” flow
3. No AI features — The biggest differentiator (AI from Git commits) has zero implementation. No OpenAI SDK, no Git integration, no API for external triggers
4. No embeddable widget — Advertised but not built
5. No subscriber flow — No subscribe form, no email integration
6. No settings page — Sidebar links to it but 404
7. No pagination — Dashboard and changelog both limited to 50 entries with no next/prev
8. No mobile sidebar — Dashboard sidebar is always visible; no hamburger/collapse for mobile
9. Dead placeholder links — Features, Pricing, Docs, Demo pages don’t exist
10. No analytics tracking — Analytics table exists but nothing writes to it

---

6. Recommendations — Priority Order

Immediate (Unblock the Product)

1. Fix tenant resolution for /changelog — Set DEFAULT_TENANT_SLUG env var on Vercel, or implement first-tenant fallback
2. Create seed data — Add 3-5 sample changelog entries so the public page isn’t empty
3. Remove or disable dead links — /features, /pricing, /docs, /demo — either build placeholder pages or remove the links

Short-term (Make It Demo-Ready)

4. Add settings page — Even a basic one showing tenant name, slug, logo URL
5. Fix JSON-LD XSS vector — Sanitize entry data before injecting into <script> tag
6. Add mobile sidebar — Hamburger menu or collapsible sidebar for responsive dashboard
7. Import rehype-sanitize — Currently in deps but not used in the Markdown renderer

Medium-term (Ship the Differentiator)

8. Implement AI changelog generation — This is the killer feature per competitive analysis:
   • Connect to GitHub repos
   • Parse commits/PRs into structured entries
   • Use LLM to generate human-readable changelog entries
   • Auto-categorize by update type (feature/fix/improvement)
9. Build the embeddable widget — JS snippet customers can add to their apps
10. Subscriber management — Subscribe form on changelog page, email notifications via Resend

Longer-term (Competitive Positioning)

11. CLI tool — better_changelog push from CI/CD pipelines
12. GitHub Action — Auto-generate changelog on release/tag
13. Analytics — Track views/engagement per entry
14. Custom domains — UI for configuring and verifying custom domains
15. Pricing/billing — Stripe integration (schema fields already exist)

---

7. Competitive Context

Based on the analysis in /root/clawd/changelog-competitors.md:

• No competitor does AI changelog generation from Git commits — This is the blue ocean
• Beamer ($49-249/mo) has no AI, no dev integrations. Vulnerable.
• LaunchNotes ($249/mo) has AI writing but is enterprise-priced. Huge pricing gap to exploit.
• Headway ($29/mo) is simple but stagnant. Easy to outpace.
• Released has AI from Jira but is Jira-locked. GitHub users are unserved.
• Canny/Featurebase are feedback-first — changelog is secondary.

The winning move: Ship the Git→AI→Changelog pipeline ASAP. Even a basic version (connect GitHub repo → select commits → generate entry with GPT-4) would be a category-defining feature that no competitor has.

---

8. Test Execution Summary

Area	Tests	Passing	Blocked	Failing
Auth	5	2	0	0 (3 untested - need browser)
Dashboard	6	1	0	0 (5 untested - need auth session)
Public Changelog	4	0	4	0
Feeds	3	0	3	0
Multi-Tenant	4	0	3	1
UI/UX	3	1	0	2
Mobile	4	0	0	0 (untested)
Performance	4	2	0	0 (2 untested)
Security	7	5	0	2

Overall: The codebase is solid but the deployed product has a critical tenant resolution bug that blocks the primary user-facing feature (public changelog). Fix that, add seed content, and the app becomes demoable.