Google started with the goal of indexing the internet to make finding things online easy. They succeeded in solving that problem, but now Google has changed the way we consume, create, and used the internet. And I don’t think anyone is truly behind the wheel of Google’s search product.
In 2018 Google made 3,200 changes to the search algorithm. Most of these were small tweaks, but I’d be shocked if any single person or team at Google could explain how the search engine actually ranks pages. Google has layered so much machine learning onto the human-written algorithms; I am convinced Google’s search product is now a black box even to the teams who help build and maintain it.
An Oversimplified Story of Google Search
When two Stanford students created Google, the idea was simple, keywords explained what a webpage was about, and inbound links indicated the importance of that page. The concept of using links to rate the importance of a page comes from bibliometrics, the statistical methods used to analyze and rank the influence of academic publications — the more citations an academic paper has from other papers, the more influential that paper is considered. Citations are also waited, meaning not every citation holds equal importance.
With a lot of backstory left out, this concept lead Larry Page and Sergey Brin to create the first modern search engine. Other search engines existed for around nine years before 1998 when Google stepped into the ring. But while Archie and Veronica can be called the grand parents of all search engines, Google changed how every future search engine would work.
Prior to modern search engines, websites were found using directories and blog roles, where sites would link to each other so that people could discover more of the web. This linking, along with basic keyword recognition, was how Google indexed and ranked websites. When this started to get abused by webmasters who would build sites, write spam content, and buy links to rank, Google updated the algorithm. Google also told website owners to change their behavior or risk getting deindexed.
Google Now Shaped The Web
Most of the early updates were direct changes to math that impacted how Google treated things like keyword stuffing, invisible text, and “unnatural” links. This was when Google started really changing the behavior of webmasters. With many high profile sites banished from the search results starting in 2003, everyone knew Google wasn’t playing. Most Google updates would come with a new crop of sites pushed down in, or removed from search results altogether.
When Google said how they wanted the web to work, the internet now listened. When Google said excessive reciprocal links are bad and reordered search results pages, web users changed their behavior. Someone on Google’s webspam team would say “use the nofollow tag on press releases” – webmasters and marketers would do so. The web was adapting to Google more than Google was adjusting to the internet.
The AI Now Runs Google
By 2015 Google added RankBrain, a machine learning algorithm that attempts to choose the best content for any particular keyword search. The AI examines factors like user behavior and regional linguistic patterns. Later machine learning was added to link analysis and used to aggregate signals together.
We have seen with Google-owned YouTube that the AI system reviewing videos for demonetization and removal is not fully understood. When accused of bias YouTube execs insisted, they didn’t fully know how the AI worked. I have no doubt the YouTube executives were truthful and that Google is equally clueless about what the AI is doing.
Brian Dean made a list of 200 Google ranking factors, and if the AI could articulate every factor it considers, I’d bet Dean’s list covers under 1% of them. I can’t say the inmates are running the asylum. But it seems accurate to say humans no longer control the entity that shapes the web more than any other force on the globe.
Header Image: “Fail Bot” by Christia Fung