APIs, JSON, scraping, regex, and automating the routine parts of OSINT work.
You don't need to become a software engineer to get real value from Python in OSINT work. The goal here is narrower and more achievable: stop spending hours on repetitive manual tasks — checking the same ten sites for a hundred usernames, re-running the same search every day, manually copying data into a spreadsheet — and let a short script do it instead.
When you open a webpage, your browser sends a request to a server, which replies with a status, headers, and a body (HTML, or data in JSON/CSV/XML format). An API (Application Programming Interface) is a structured way to make that same kind of request but get back clean, structured data instead of a page meant for human eyes. The requests library is the standard way to do this in Python:
import requests
response = requests.get("https://api.github.com/search/users?q=javascript")
print(response.json())
There's a huge range of free and paid APIs relevant to OSINT: phone number and address lookups, company registry data, domain/IP information (Chapter 9), cryptocurrency wallet and transaction data (Chapter 10), and social media platform data. Before writing a script from scratch to test one, it's often faster to try an online request-testing tool (reqbin.com or similar) to understand the API's shape first.
Most APIs return data as JSON (JavaScript Object Notation) — a text format that maps directly onto Python dictionaries and lists, and ships with Python by default via the json module. A typical API response is a list of objects, each with named properties (a GitHub user search result might have login, html_url, id, and so on for each of 30 returned users) — once you understand that shape, pulling out exactly the fields you need is straightforward dictionary/list indexing.
Scraping means pulling data directly out of a webpage's HTML when no API exists. The most important thing to know: writing a script from scratch should be your last resort, not your first move — try ready-made tools first (browser extensions like Web Scraper, or purpose-built platforms), and check whether a tool built specifically for the platform you care about already exists (dedicated tools exist for YouTube, Twitter/X, and most major platforms) before reinventing anything.
When you do need custom code, requests plus BeautifulSoup is the standard combination:
import requests
from bs4 import BeautifulSoup
url = "https://example.com/some-page"
web_page = requests.get(url)
soup = BeautifulSoup(web_page.content, "html.parser")
header = soup.find("h1").get_text()
print(header)
You target elements with CSS selectors (h1 for a tag, .classname for a class, #id for an ID) — your browser's developer tools will show you exactly which selector matches the element you want. One recurring trap: the HTML your script downloads can look completely different from what you see in the browser, because many modern sites build large parts of the page dynamically with JavaScript after the initial page load. When that's the case, requests alone won't see the content — you need Selenium, which drives an actual browser and can wait for JavaScript to finish running before grabbing the page.
A regular expression is a pattern for finding, validating, or extracting pieces of text that match a certain shape — phone numbers, email addresses, IP addresses, cryptocurrency wallet strings. Python's built-in re module handles this:
import requests
import re
url = "https://example.com/some-page"
html = requests.get(url).text
result = re.findall("[a-zA-Z0-9-_.]+@[a-zA-Z0-9-_.]+", html)
print(result)
This pulls every email-shaped string out of a page's raw HTML in one line — genuinely useful for a first pass over a large or unfamiliar page, though always followed by manual verification, exactly as with any auto-extracted text (see the OCR caution in Chapter 6).
Many sites block IPs sending an unusually high volume of requests in a short window. A proxy server sits between your script and the target, letting you route requests through a different IP — it doesn't always defeat rate-limiting or blocking, but it's the standard first mitigation, and it's a small change to a requests call once you have a proxy address to use.
Chapter 9 covered WHOIS and DNS lookups conceptually — here's how to automate them instead of using a web form every time:
import whois
whois_info = whois.whois('example.com')
print(whois_info["creation_date"])
import dns.resolver
result = dns.resolver.resolve('example.com', 'A')
for ipval in result:
print('IP', ipval.to_text())
If your task is finding subdomains at scale, the python-whois and dnspython packages combine well with a dedicated subdomain-enumeration tool (Chapter 9) run from the command line, with results piped into a file for further scripted processing.
The wayback package lets you script searches and downloads of archived pages instead of clicking through the web interface — useful when you need to pull dozens or hundreds of historical snapshots for a timeline reconstruction (see the historical-geolocation workflow in Chapter 7):
import wayback
from datetime import datetime
client = wayback.WaybackClient()
for record in client.search('http://example.com', to_date=datetime(1999, 1, 1)):
memento = client.get_memento(record)
filename = memento.memento_url.replace("/", "-") + ".html"
with open(filename, "a") as f:
f.write(memento.text)
print(filename)
Once you've collected and correlated data, Python can also help you produce the final deliverable (Chapter 15). A few specific, commonly-used packages worth knowing by name rather than just "some library exists": XlsxWriter builds an Excel workbook cell by cell, including formulas (useful for a findings spreadsheet a non-technical stakeholder can reopen and re-sort); python-docx builds a Word document — headings, paragraphs, page breaks, embedded images — closely matching the report structure from Chapter 15; fpdf (or its actively-maintained fork fpdf2) generates PDF files directly, positioning text and images at specific coordinates; python-pptx does the same for PowerPoint, useful for a briefing-style deliverable rather than a written report.
For visualizations specifically: Matplotlib (paired with NumPy for the underlying numeric arrays) turns a list of tabulated findings into a bar chart, line chart, or scatter plot in a few lines, and saves directly to a PNG you can embed in one of the document formats above. For geographic data specifically (wallet clusters, post locations, IP geolocations plotted onto an actual map), the Basemap toolkit layers on top of Matplotlib to draw country borders, coastlines, and plotted points from raw coordinates.
You don't need to master all of the above before it's useful. A workable path: learn just enough syntax to read and adapt someone else's script (rather than writing everything from scratch), keep a personal folder of small scripts you reuse across investigations, and treat every script as a rough draft — the goal is saving time on routine tasks, not building production software. If a script breaks because a site changed its HTML structure or an API changed its response format, that's normal maintenance, not a sign you did something wrong.
Beyond scripting, large language models are now a genuinely useful automation layer in their own right — summarizing a long document, triaging a big batch of search results, drafting a first-pass translation, or suggesting search terms you hadn't thought of. Used well, this can meaningfully speed up the earlier, high-volume stages of an investigation. The governance principle worth committing to before you rely on this: use AI to go faster, never to decide for you. That means naming its specific failure modes explicitly, rather than trusting it by default — hallucination (confidently stating something false or fabricated, including invented sources or citations that don't exist), bias (reflecting patterns and skew in its training data, not neutral judgment), staleness (its knowledge has a cutoff date and it may not know about a recent event, a renamed tool, or a site that no longer works the way it describes), and the absence of the kind of contextual judgment a human investigator builds up over a case — it doesn't know your specific investigation's stakes, legal constraints, or what a wrong call would cost.
A workable discipline: treat any AI-generated summary, suggestion, or draft finding the same way you'd treat an unverified tip from Chapter 13 — useful as a starting point or a hypothesis generator, never as a substitute for your own verification of the underlying evidence. If an AI tool suggests a lead, a connection, or a conclusion, the same corroboration and confidence-scoring discipline from Chapter 15 still applies in full — the fact that a suggestion came from a model rather than a human source doesn't lower the bar for evidence, if anything it should raise it, precisely because a fluent, confident-sounding answer is easy to mistake for a verified one.
The concrete mechanism for keeping a human analyst genuinely in control, rather than just nominally "in the loop," has a name: Explainable AI (XAI). The idea is simple even though the underlying techniques aren't: any AI system feeding into a security or investigative judgment should be able to show why it produced a given output, not just the output itself. A tool that flags a profile, a transaction, or a document as "suspicious" with no visible reasoning is operationally useless to an analyst who has to defend that judgment later, and it quietly shifts responsibility onto a system nobody can actually interrogate. In practice, look for (or ask a tool's vendor for) some form of feature-level explanation — which specific inputs drove the output, and how much each one weighed in — rather than accepting a bare confidence score. If a tool can't tell you why, treat its output as a much weaker signal than one that can, regardless of how accurate its marketing claims to be.
Beyond the manual dorking workflow in Chapter 3, a Python package can run search queries and collect the results programmatically — useful when you need to run the same search across many targets, or repeat a search on a schedule to catch new results. The duckduckgo_search package is a solid starting point: it takes a query string plus parameters for region, whether to disable safe-search, a time window, and — usefully — a flag to download every result page directly rather than just returning links, which turns a search into a small local archive of everything it found in one step. As with any dorking, DuckDuckGo's own advanced operators (filetype:, site:, exact-phrase matching) work inside the query string passed to the package, so everything from Chapter 3's operator table still applies.
For username and profile enumeration specifically, beyond Sherlock and Maigret (Chapter 4), Blackbird is a lighter command-line tool worth knowing — simpler and faster than Sherlock, at the cost of checking fewer platforms; reach for it for a quick first pass, and fall back to Sherlock/Maigret when you need broader coverage.
A script that only you can run from a command line has limited reach — if you want a colleague or a non-technical stakeholder to actually use something you've built, wrapping it in a minimal web interface removes that barrier entirely. Streamlit is the fastest route to this: a page title, a text input box, and a button that displays whatever your script would normally print, in well under twenty lines of code, with no separate frontend work required. It's meant for exactly this kind of internal tool — turning a working script into something a colleague can open in a browser and use without touching a terminal — not for a public-facing production application, where a heavier framework (Flask, Django) would be the more appropriate choice.
requests and run the GitHub user-search example above with a different search term. Print just the login field for each returned user, rather than the whole JSON blob.<h2> headers.For exercise 1, the key skill being tested is comfortably navigating a nested JSON structure (a list of dictionaries) rather than the API call itself — if you can pull one specific field out of a real response, you can adapt that same pattern to almost any other API in this manual. For exercise 3, a reasonable starting pattern is https?://[^\s"'<>]+ — note that "reasonable" is doing a lot of work there; production URL-matching regexes get notoriously complicated, and that's fine for a quick OSINT script. For exercise 4, resist the urge to build the full username-enumeration logic before confirming the Streamlit shell works end to end — get the trivial print-it-back version running first, then swap in the real logic; debugging a broken UI and broken enumeration logic at the same time is much harder than debugging them one at a time.