Comparing an HTML noindex directive with an X-Robots-Tag header
An indexing instruction can arrive in the HTML or a response header. Inspect both representations from the same request before changing one and assuming the restriction is gone.
Capture the two locations
Save the response headers and the HTML body for the exact URL. Record any applicable robots meta element and X-Robots-Tag value verbatim, including whether a value names a particular crawler. A screenshot of page content will not capture these instructions. Keep the original URL and final destination separate when a redirect occurs.
Repair the responsible layer
Determine whether the instruction is intentional for this page. If a public article accidentally inherits noindex, correct the template or response rule that produces it. Recheck both locations after deployment. Removing a meta tag will not remove an independently supplied header. Record any provider observation separately from the current delivered response.
Source guidance
Use the website analyzer to collect a starting set of observations, then verify the specific evidence described above.Frequently Asked Questions
Can I stop after checking the HTML?
No. Include response headers in the audit because indexing directives can also be delivered there.