As a global leader in SEO, content marketing, and data analytics, Anastasia Braitsik brings a data-grounded perspective to the evolving digital landscape. Drawing from the latest Web Almanac research, she deciphers how foundational technical hygiene, the rise of AI-driven protocols, and the persistent “messiness” of the web are shaping the future of search.
This conversation explores the shift from manual SEO to CMS-driven defaults, the transformation of robots.txt into a strategic policy document, and the surprising resilience of deprecated web standards. Braitsik also delves into the emergence of AI-specific files like llms.txt and the tactical value of structured data in an era where generative search engines are hungry for extractable information.
HTTPS adoption is over 91%, yet duplicate content management remains a challenge with 33% of pages missing canonical tags. How do you balance relying on CMS defaults versus manual audits? What specific metrics should teams track to ensure foundational hygiene doesn’t slip during a complex site migration?
While I love seeing HTTPS adoption hit 91%, that 33% gap in canonical usage is a loud reminder that we can’t just “set and forget” our tech stacks. I generally view CMS defaults as a safety net rather than a complete solution; for instance, while tools like Yoast or AIOSEO help cement standards, they often fail to account for the unique edge cases of a complex migration. During a transition, I tell teams to keep their eyes glued to the ratio of indexed vs. crawled pages and the percentage of valid canonicals—if that 67% adoption rate we see industry-wide starts dipping on your specific site, you’ve got a duplicate content leak. There is a certain visceral stress in watching a migration go sideways, so tracking “Index Status” in Search Console alongside “HTML Validity” (which currently sits at about 10% invalidity for head elements) is your best defense against structural rot.
Major SEO plugins often dictate industry standards like sitemaps and structured data. How can developers prevent these “out-of-the-box” settings from bloating code or creating technical debt? In what ways should a custom-built site approach these defaults differently to gain a competitive edge in search rankings?
The convenience of out-of-the-box tools is a double-edged sword because “default” often means “heavy,” and we see this reflected in the lagging Lighthouse scores for many major platforms. To avoid technical debt, developers need to treat SEO plugins as modular suggestions rather than mandatory scripts; if you aren’t using a specific feature, disable it to keep the DOM lean. A custom-built site has the luxury of surgical precision, such as implementing lazy loading for the 67% of images that currently lack it, without the overhead of a generic plugin. By hand-coding semantic HTML and structured data rather than relying on a “one size fits all” generator, you create a site that is significantly faster and easier for bots to parse, which is a massive competitive differentiator in a world of bloated CMS templates.
While desktop performance is improving, mobile gaps persist and legacy declarations like “msnbot” or unused “index” tags still appear frequently. How do you identify which deprecated standards are harmless versus those draining crawl budget? What is your step-by-step process for cleaning up a long-neglected enterprise site?
It is honestly fascinating to still see “msnbot” in the top five meta robots declarations when it was replaced over 16 years ago—it’s like finding a ghost in the machine. Most of these, like implicit “index” or “follow” tags, are harmless clutter, but they signal a lack of site maintenance that can hide more serious issues like defunct AMP pages, which still linger on 38,000 homepages. When I audit a neglected enterprise site, I start by stripping away the “invisible” junk: I remove those redundant tags, kill off any legacy AMP code that hasn’t seen updates in four years, and then tackle the mobile performance gap. My process is a ruthless “prune and polish” where we eliminate anything that doesn’t serve a modern crawler, ensuring that every millisecond of crawl budget is spent on high-value, current content rather than 15-year-old bot instructions.
Robots.txt usage is shifting from simple crawl control to a policy document for AI bots like Gptbot and Claudebot. How should businesses determine which AI crawlers to block versus allow? What are the potential downstream revenue risks of restricting these bots compared to the benefits of protecting proprietary data?
We are seeing a massive shift where robots.txt has become a high-stakes “bouncer” for the site, with Gptbot usage jumping by 55% and Claudebot nearly doubling in the last year. Deciding whether to block these bots is no longer just a technical choice; it’s a business strategy session involving marketing and security teams. If you block GPTBot (which is now at about 4.5% adoption on desktop), you protect your proprietary data from being used in training sets, but you risk losing visibility in the very AI-driven answers that users are starting to rely on. It’s a delicate trade-off: you are essentially weighing the protection of your intellectual property against the potential “downstream revenue” of being the cited source in a generative search result.
Adoption of llms.txt has grown significantly, driven by specific SEO tools, despite debates over its actual efficacy. Beyond just following trends, what strategic value does providing structured instructions for large language models offer? How do you measure if these files actually influence how AI platforms retrieve and summarize content?
The adoption of llms.txt is one of the most surprising trends I’ve tracked, jumping from a mere 15 sites in my early 2025 crawl to over 2% of the web today, largely fueled by AIOSEO’s 39.6% share of those files. The strategic value isn’t necessarily in a ranking boost today, but in acting as a “statement of intent” to guide how LLMs retrieve and summarize your site’s specific knowledge. While it’s hard to quantify direct influence yet, I look for “referral-like” patterns in how AI platforms cite content—if the summaries start mirroring the structure we’ve provided in that .txt file, it’s a sign the “handshake” is working. Even if some call it controversial, being an early adopter means you’re helping define the rules of engagement for the next era of the web.
FAQPage schema usage is increasing even as traditional search engines reduce the visibility of these snippets. Why is structured, extractable content becoming more valuable for non-SERP applications? What specific types of structured data should organizations prioritize to remain visible within the evolving landscape of generative search experiences?
It seems counterintuitive that FAQPage schema is rising—hitting 6.7% on mobile—even after Google scaled back those snippets, but there is a very logical reason behind it: AI search engines love structured, extractable answers. We are moving beyond optimizing just for a blue link; we are optimizing for “retrievability” so that a generative AI can easily pull our data into a summarized answer. Organizations should prioritize schema that defines clear relationships, like FAQPage, Product, and HowTo, because these act as a roadmap for LLMs. Even if you don’t see a “rich result” on a traditional SERP, your data is much more likely to be featured in an AI’s conversational response if it’s wrapped in clean, valid markup.
What is your forecast for SEO in 2026?
By 2026, I expect we will stop viewing AI search as a “replacement” for SEO and instead see it as a permanent, specialized layer that sits on top of our existing technical foundation. We will see the performance gap between mobile and desktop finally begin to converge, though the web’s “long tail” will remain messy with legacy code and 404 errors (which currently sit at around 13% for robots.txt). SEO professionals will essentially become “Bot Managers,” spending as much time auditing llms.txt and AI policy documents as they do on keyword research. It won’t be a total rewrite of the industry, but rather a “business as usual” environment where the stakes for technical precision and structured data are higher than ever before.
