Web Crawler and Search Engine

URL Frontier: priority queue of URLs to crawl. Fetcher: async HTTP requests (aiohttp) with politeness delay (0.5–2s between requests to same domain). Link Extractor: parse HTML with BeautifulSoup, find all anchor hrefs, normalize URLs (absolute paths, remove fragments), filter (same domain or all domains), add to frontier. Bloom filter checks if URL already visited (false positive rate 1% — acceptable for web crawling).

Extract main content text from HTML: remove script, style, nav, footer tags. Extract title (h1 or title tag), meta description, headings structure, and body text. Apply cleaning: lowercase, remove punctuation, tokenize, remove stopwords (the, a, is, etc.), stem or lemmatize (run/running/ran → run). Store extracted text with metadata: URL, title, description, crawl timestamp, content hash (for deduplication).

Inverted index maps term → list of (document_id, term_frequency) pairs. For each document: tokenize text, count term frequencies. Add to index: for each term, append (doc_id, tf) to posting list. TF-IDF score: TF(t,d) × IDF(t). IDF(t) = log(N / df(t)) where N=total docs, df(t)=docs containing term t. Rare terms have high IDF, penalizing common terms like

Parse query: tokenize and stem query terms. Retrieve posting lists for each term. Intersect posting lists (AND query) or merge (OR query). Score each document: sum TF-IDF scores for all query terms in that document. Re-rank by PageRank × TF-IDF score. Return top-10 results with snippet (extract 160-char context around first query term occurrence in document). Response time target: < 50ms for cached queries, < 200ms for cold queries.