Wikipedia Graph Web

What This Is

A graph traversal tool that finds the minimum degree of separation between any two Wikipedia articles using bidirectional BFS on the hyperlink graph. The concrete application is interesting. The deeper question is more interesting: how is human knowledge topologically organised?

Wikipedia's link structure encodes the editorial community's consensus about which concepts are related. The degree of separation between two articles tells you something about the conceptual distance between their topics and the shortest path tells you which bridge concepts connect them.

The Graph Scale

English Wikipedia has approximately 6.7 million articles and 200 million hyperlinks a sparse graph, but at a scale where naive BFS would take hours. Bidirectional BFS from both endpoints, meeting in the middle, reduces the search space from O(b^d) to O(b^(d/2)) where b is the branching factor (~40 for Wikipedia) and d is the depth. For d=6, this is the difference between 4 billion and 64,000 nodes explored.

Multi-Threaded Crawling

Live Wikipedia traversal (vs. a local graph snapshot) requires HTTP calls for each node's link list. We parallelise with a thread pool: each thread expands one node, fetches its outlinks, and reports back to the BFS coordinator. The coordinator manages the frontier queue and the visited set (thread-safe with a lock or atomic operations).

Rate limiting is mandatory Wikipedia asks crawlers to stay under ~200 req/s. We implement exponential backoff and request deduplication.

The Visualisation

PyVis renders the discovered path as an interactive graph: nodes are articles, edges are hyperlinks, path length is visible. Hovering shows article summaries. The visualisation makes the conceptual bridge immediately legible you can see why topic A connects to topic B through C.

What We're Learning

• The average degree of separation between random Wikipedia article pairs (the "Wikipedia number")
• Which "hub" articles appear on the most shortest paths a measure of conceptual centrality
• Whether semantic embedding distance (cosine similarity of article embeddings) correlates with graph distance

What's Next

• Local graph snapshot for offline search (dump + parse Wikipedia XML)
• GNN-based path prediction: learn to predict which intermediate nodes are likely on the shortest path
• Semantic layer: weight edges by embedding similarity to prefer conceptually meaningful paths over incidental links