Comparing an HTML noindex directive with an X-Robots-Tag header

An indexing instruction can arrive in the HTML or a response header. Inspect both representations from the same request before changing one and assuming the restriction is gone.

Capture the two locations

Save the response headers and the HTML body for the exact URL. Record any applicable robots meta element and X-Robots-Tag value verbatim, including whether a value names a particular crawler. A screenshot of page content will not capture these instructions. Keep the original URL and final destination separate when a redirect occurs.

Repair the responsible layer

Determine whether the instruction is intentional for this page. If a public article accidentally inherits noindex, correct the template or response rule that produces it. Recheck both locations after deployment. Removing a meta tag will not remove an independently supplied header. Record any provider observation separately from the current delivered response.

Source guidance

Use the website analyzer to collect a starting set of observations, then verify the specific evidence described above.

Frequently Asked Questions

Can I stop after checking the HTML?

No. Include response headers in the audit because indexing directives can also be delivered there.

Want to be found and recommended in AI search?

Start with a free scan of where you stand, then book a call and we'll map the highest-ROI AI plays for your business.

Book a Free Call →