Skip to content

Commit

Permalink
fix: Robots txt disallow changes
Browse files Browse the repository at this point in the history
  • Loading branch information
richiemcilroy committed Nov 14, 2024
1 parent 9b06823 commit 21f7434
Showing 1 changed file with 21 additions and 18 deletions.
39 changes: 21 additions & 18 deletions apps/web/app/robots.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,32 @@ import { seoPages } from "@/lib/seo-pages";
import { MetadataRoute } from "next";

export const dynamic = 'force-dynamic';
export const revalidate = 0;

export default function robots(): MetadataRoute.Robots {
// Get all SEO page slugs
const seoPageSlugs = Object.keys(seoPages);

return {
rules: {
userAgent: '*',
allow: [
'/',
'/updates/',
// Dynamically add all SEO pages
...seoPageSlugs.map(slug => `/${slug}`),
],
disallow: [
'/dashboard',
'/s/',
'/login',
'/invite',
'/onboarding',
'/record',
],
},
rules: [
{
userAgent: '*',
allow: [
'/',
'/updates/',
// Dynamically add all SEO pages
...seoPageSlugs.map(slug => `/${slug}`),
],
// Be more specific about what we're disallowing under /s/
disallow: [
'/dashboard',
'/s/*', // This will match /s/ and anything under it
'/login',
'/invite',
'/onboarding',
'/record',
],
},
],
sitemap: 'https://cap.so/sitemap.xml',
};
}

1 comment on commit 21f7434

@vercel
Copy link

@vercel vercel bot commented on 21f7434 Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.