How search engines index the web, Google dorking, cache/archives, and pivoting methodology.
Search engines aren't a real-time window onto the internet — they're a storefront sitting on top of a pre-built index. Understanding how that index gets built is the foundation for understanding why some things are discoverable and how you can find them on purpose instead of stumbling into them.
Three stages. Crawling: automated bots follow links, read page content, and discover new URLs. Indexing: the collected content gets processed and stored in a massive database — URL, title, text, metadata, linked files. Ranking: when you search, the engine pulls matching entries out of that pre-built index and sorts them by relevance. Everything you see comes from the index, not from a live request to the site.
The key insight for investigators: a file doesn't need to be linked from any page to get indexed. If a crawler discovers its URL by any means at all — a sitemap, a referrer header, a cached link — it gets indexed and becomes searchable. This is the root cause of most accidental document exposure.
Crawlers discover new content through: following links from pages they already know about, sitemaps (sitemap.xml) that site owners submit to guide them, direct URL submission through webmaster tools, referrer headers from other sites linking to a page, and periodic re-crawling of historically known URLs. The practical implication: a file sitting in an unlinked directory is not private. If its URL is predictable, or shows up in any of those channels, it will be found.
robots.txt tells well-behaved crawlers which paths to skip (Disallow: /internal/). Critically, it is not an access-control mechanism — it's a polite request. OSINT researchers have zero obligation to respect it. In fact, by listing sensitive directories inside it, administrators unintentionally hand out a map of exactly where their most sensitive content lives. For an investigator, robots.txt is often the single most valuable starting point of a web-based investigation.
A pattern worth specifically watching for, because it's more common than it sounds: an administrator correctly blocks a directory in robots.txt and disables directory listing on the server — but forgets that the exact filenames inside it are still individually enumerated in sitemap.xml, submitted separately for SEO purposes. The very mechanism built to help search engines find an organization's legitimate content ends up exposing the content that was supposed to stay hidden. Whenever you find a Disallow entry, check the sitemap for the same path before assuming the directory is merely hard to browse rather than fully indexed.
Google Dorking maps directly onto MITRE ATT&CK technique T1593.002 — Search Engines, under the Reconnaissance tactic (TA0043). It involves zero intrusion — every query goes to a public search engine, and every result was already publicly indexed. Legally and ethically, it's passive reconnaissance: you're just asking a search engine a precise question.
| Operator | Example | What it does |
|---|---|---|
site: | site:example.com | Restricts results to one domain |
filetype: | filetype:pdf | Restricts to a file extension |
inurl: | inurl:uploads | Matches a string in the URL |
intitle: | intitle:budget | Matches a string in the page title |
intext: | intext:confidential | Matches a string in the page body |
cache: | cache:example.com | Shows a cached snapshot (unreliable availability) |
- / "…" | -site:x.com "staff roster" | Excludes / requires an exact phrase |
The real power comes from stacking operators: site:brightwater-college.edu filetype:pdf pulls every indexed PDF from that domain. filetype:bak site:brightwater-college.edu hunts for forgotten backup files. site:brightwater-college.edu inurl:internal filetype:txt looks for plain-text files sitting in an internal-sounding directory.
A structured workflow: start broad (site:target.com) → systematically enumerate file types (filetype:pdf, then xls, doc, txt, bak, in sequence) → follow leads from robots.txt (a disallowed path becomes an inurl: query) → target by content (intitle:/intext: for words like "confidential" or "internal use only") → keep narrowing by stacking operators until the result set is manageable.
The core misconception beginners have: deleting a file from a server doesn't make it inaccessible. Copies can survive in the search engine cache (volatile — Google can drop it at any time), in web archives like the Wayback Machine or archive.today (permanent and timestamped), in the local browser caches of people who visited before the deletion, or in copies shared by third parties.
The right question is never "does this file still exist on the live server?" It's "was this ever publicly accessible?" If the answer is yes, there's a real chance it's still recoverable.
High-value file types worth specifically dorking for: .pdf (reports, budgets, contracts), .xls/.xlsx (personnel data, financial models), .docx (internal memos, drafts), .txt/.log (configs, credentials, IP traces), .bak (forgotten backup copies), .env (API keys, database credentials — often the highest-value find of all). The typical exposure lifecycle: upload → discovery (via sitemap, crawler, or a shared link) → persistence (cached and archived even after removal) → inaction — the file is known about internally but the fix keeps getting deferred, which is the single most common factor behind serious exposures, and the internal tickets discussing the delay end up as additional evidence of negligence if it's ever litigated.
Different engines keep independent indexes. For geography- or language-specific work: Baidu (dominant for Chinese-language content, indexes Chinese platforms Google largely misses), Yandex (Russian/CIS content, a notably strong reverse-image search, deeper indexing of VK and OK.ru), Naver (South Korea's dominant engine, indexes Korean blogs and forums Google doesn't touch). The rule: language and geography shape what gets indexed where — a target's presence in their native language, on local platforms, can be entirely invisible to an English-language Google search.
Privacy-focused engines (DuckDuckGo, Startpage, Brave Search) give you unpersonalized results (Google filters based on your history — you might miss results a different user would see) and reduce your query trail during sensitive work.
Pivoting is using one finding as the launch point for a new line of inquiry. A pivot chain is a documented sequence of data points and where each came from — essential for reproducibility, for legal admissibility, and to avoid accidentally reasoning in a circle. Common pivot types: domain → exposed file, robots.txt path → dork query, file content → internal hostname, archived snapshot → deleted content.
Pivoting discipline — the failure modes worth watching for: assumption pivots (jumping from "works at company X" to "must have access to system Y" with no evidence), identity confusion, stale data (treating an archived page as current), and over-pivoting (following a chain so far from the original target that the investigation loses focus).
site:domain.edu filetype:pdf, then site:domain.edu filetype:xls OR filetype:xlsx. Note what kinds of documents surface (without downloading or using anything that looks sensitive).domain.edu/robots.txt for a site of your choice. Are there Disallow paths? Run site:domain.edu inurl:<path> for one of them and see what's indexed.For exercises 1 and 2, don't download or retain anything that looks personal or sensitive (see Chapter 1's ethics checklist) — the point is to observe that something is indexed, not to act on it. Finding nothing is a perfectly normal, useful result — plenty of sites have already cleaned up their exposure. For exercise 3, a good sign of success is spotting one specific, describable change (a removed paragraph, a reworded headline) rather than a vague "something looks different."