AIQ

1. AI is Algorithms, Not Sci-Fi Droids, Driven by Data and Diffusion

Yet while this arms race is real, we think there’s a much more powerful trend at work in AI today—a trend of diffusion and dissemination, rather than concentration.

Democratizing technology. Artificial intelligence is not a futuristic sci-fi concept but a present-day reality, fundamentally changing the world through algorithms in our smartphones and daily tools. While large tech companies like Amazon and Google engage in an "arms race" for AI talent, the more significant trend is the rapid diffusion of AI technologies and ideas to smaller companies, diverse economic sectors, hobbyists, and researchers globally. This democratization empowers a vast range of problems to be solved with AI.

Pipeline of algorithms. At its core, AI is a "domain-specific illusion of intelligent behavior" created by chaining together numerous algorithms. Each algorithm performs a specific task, like converting sound to digital signals or translating phonemes into words, culminating in a prediction or decision. Unlike traditional algorithms with fixed instructions, AI algorithms learn their instructions directly from "training data," identifying patterns to make probabilistic judgments rather than certainties.

Enabling forces. The current AI revolution is built upon three technological pillars that have brought long-standing ideas to fruition. These include the exponential growth in computer speed (Moore's law), the explosive accumulation of digitized data (the "new Moore's law"), and the democratizing effect of cloud computing, which turns prohibitive fixed costs into accessible variable costs for data storage and analysis. These forces, combined with foundational mathematical ideas, have created a "supernova-like explosion" in AI's demand and capacity.

2. Personalization Relies on Conditional Probability and Latent Features

To a learning machine, “personalization” means “conditional probability.”

Tailored suggestions. Personalization, exemplified by Netflix's recommender system, is a cornerstone of the online economy, providing tailored suggestions for everything from music and products to news and even cancer therapies. This shift from "search" to "suggestions" leverages the collective knowledge of billions, creating "doppelgänger software" that can anticipate individual preferences. The underlying mathematical concept driving this is conditional probability.

Abraham Wald's insight. The core idea of personalization dates back to Abraham Wald's work during World War II, where he used conditional probability to make "personalized survivability suggestions" for Allied aircraft. By analyzing damage patterns on returning planes and accounting for missing data (planes that didn't return), Wald developed a system to recommend where to add armor. This problem of missing data is analogous to Netflix's challenge of recommending films when most subscribers haven't watched most films.

Latent feature models. Netflix, like Wald, faced the challenge of estimating conditional probabilities with massive, incomplete datasets. Their solution, and that of the Netflix Prize winners, involved "latent feature" models. These models identify hidden patterns in user ratings (e.g., "affinity for witty oddball comedies") that define a user's unique preferences in a multidimensional space. These features, discovered organically by AI from data, allow Netflix to segment its audience into "demographics of one," enabling targeted content production and strategic transformation.

3. Pattern Recognition in AI is About Fitting Prediction Rules to Data

In AI, a “pattern” is a prediction rule that maps an input to an expected output.

Learning from data. AI-based pattern recognition, seen in applications from Beijing's toilet paper dispensers to cucumber sorters, involves computers learning to match inputs with appropriate outputs. This process defines a "pattern" as a prediction rule—an equation that describes the relationship between input and output. "Learning a pattern" means fitting the best possible prediction rule to a given dataset, minimizing average errors.

Henrietta Leavitt's discovery. The historical roots of this concept trace back to Henrietta Leavitt's 1912 discovery, which helped measure the size of the universe. By plotting the period of pulsating stars against their brightness, she found a linear pattern, creating a prediction rule. This rule allowed astronomers to determine a star's true brightness from its pulsation period, effectively turning pulsating stars into "standard candles" for measuring cosmic distances.

Modern advancements. While the principle of fitting prediction rules dates to Adrien-Marie Legendre's "least squares" method in 1805, modern AI's breakthrough stems from four factors:

• Massive Models: Using neural networks with hundreds of thousands or millions of parameters to describe complex patterns.
• Massive Data: Overcoming "overfitting" by training these complex models on enormous datasets (e.g., millions of images).
• Trial and Error: Incrementally refining prediction rules thousands of times per second through processes like "stochastic gradient descent."
• Deep Learning: Employing "deep neural networks" to automatically extract hierarchical "latent features" from complex inputs like images, enabling sophisticated automated feature engineering.

4. Bayes's Rule Guides AI in Updating Beliefs Amidst Uncertainty

Bayes’s rule is an equation that tells us how to update our beliefs in light of new information, turning prior probabilities into posterior probabilities.

Navigating uncertainty. Autonomous robots, from self-driving cars to flying taxis, constantly solve the "simultaneous localization and mapping" (SLAM) problem: determining their location while mapping an unknown environment. This complex task is inherently Bayesian, relying on Bayes's rule to update beliefs about position and surroundings with every new piece of sensor data. This rule, discovered by Thomas Bayes in the 1750s, is a profound mathematical insight for updating probabilities based on new evidence.

The USS Scorpion search. John Craven's successful 1968 search for the lost nuclear submarine USS Scorpion exemplifies Bayesian search. Despite vast search areas and limited clues, Craven's team used Bayes's rule to combine expert opinions (prior probabilities) with sensor data (acoustic readings) to create a probability map of the submarine's location. Each unsuccessful search updated these probabilities, narrowing the search area until the submarine was found within yards of the highest probability zone.

Smarter decision-making. Bayes's rule offers a powerful framework for smarter daily living, acting as an "antidogmatism" principle for evaluating new information. It teaches us to combine our "prior beliefs" with new "facts" to form "revised beliefs." This is crucial in fields like medical diagnostics, where ignoring prior probabilities (base-rate neglect) can lead to drastically incorrect conclusions, such as overestimating cancer risk from a positive mammogram. Similarly, in investing, Bayes's rule highlights the rarity of true market-beating talent, suggesting that most "winning streaks" are due to luck rather than skill.

5. Machines Learn Language by Turning Words into Numbers, Not Just Rules

To get a machine to understand words, you have to represent those words in a language the machine can work with. That means you have to turn words into numbers.

From binary to English. Early computers required humans to communicate in tedious machine language (0s and 1s), a process Grace Hopper revolutionized by inventing the compiler. This allowed programmers to use "high-level" languages with English-like commands, making computers accessible beyond specialists. However, getting machines to understand natural human language proved far more challenging, as rules-based approaches failed due to the complexity, robustness issues, and inherent ambiguity of human speech.

The data-driven shift. The "Natural Language Revolution" of the past decade abandoned the top-down, rules-based approach for a bottom-up, data-driven one. Instead of programming explicit grammar rules, AI systems are now "trained" on massive datasets of human linguistic output (the "Library of Babel" of the internet). They learn statistical patterns to mimic human language understanding, transforming language tasks into prediction problems where inputs (e.g., voice recordings) are mapped to outputs (e.g., text transcriptions).

Word vectors and meaning. A key innovation is the "word vector," a numerical representation of words where words with similar meanings have similar numbers. Google's word2vec model, for instance, learns 300 "questions" about word co-location statistics, creating a vector for each word. This allows machines to perform arithmetic on words, solving analogies like "king - man + woman = queen," demonstrating a nuanced understanding of linguistic relationships without explicit semantic programming. This mathematical encoding of context is vital for modern NLP systems like speech recognition and machine translation.

6. Anomaly Detection Requires Understanding Data Variability, Not Just Averages

To decide whether something is an anomaly, you must know two things: (1) what to expect on average, and (2) the normal bounds of variability around the average.

Beyond the average. Detecting anomalies in data streams, whether for credit card fraud or sports injuries, is crucial for saving lives and money. However, simply comparing a data point to an average is insufficient; one must also understand the "normal bounds of variability" around that average. Without this, random fluctuations can be mistaken for genuine anomalies, as illustrated by the New England Patriots' coin-toss streak, which, despite appearing suspicious, was statistically plausible due to natural variability.

Newton's oversight. The historical "Trial of the Pyx," an anomaly-detection system for English coinage since the 12th century, failed for centuries due to a misunderstanding of variability. Isaac Newton, as Warden of the Royal Mint in 1696, observed coins were "unequally" made, yet the Trial's overly wide legal bounds for average coin weight allowed significant variability and even potential fraud to go undetected. The "square-root rule" (de Moivre's equation), discovered later, shows that the variability of an average decreases with sample size, meaning the Trial's bounds should have been much tighter.

AI's precision. Modern AI systems for real-time anomaly detection apply this principle correctly, operating at speeds and scales unimaginable in Newton's time. They collect massive amounts of data, average measurements, and use statistically sound bounds to flag deviations. Examples include:

• Smart Cities: New York City's Mayor's Office of Data Analytics (MODA) uses "big N, big D" data to identify anomalous patterns in building complaints or crime, improving resource allocation.
• Environmental Monitoring: Systems are being developed to detect radiation anomalies (dirty bombs) or gas leaks by mapping background levels and flagging deviations.
• Financial Fraud: PayPal's deep learning system analyzes thousands of features in real-time transactions to detect fraud, leveraging individual spending variability to achieve significantly lower fraud rates than the industry average.

7. AI Can Revolutionize Healthcare, But Cultural Barriers Hinder Adoption

We are likely still years away from seeing our most advanced AI technologies help real patients in substantial numbers, and the reasons have nothing to do with science or computing power and everything to do with culture, incentives, and bureaucracy.

Nightingale's legacy. Florence Nightingale, the "lady with the lamp" and a skilled data scientist, revolutionized 19th-century healthcare by using statistics to expose "preventable mischiefs" in military hospitals. Her work led to nursing reform, evidence-based medicine, and improved hospital design. She demonstrated the power of data to save lives and fought against entrenched interests, providing a historical blueprint for how data science can transform healthcare.

Modern "preventable mischiefs". Despite AI's potential, modern healthcare systems are slow to adopt it, leading to "preventable mischiefs" like Joe's kidney disease. His long-term decline was evident in his GFR readings, but doctors, relying on "threshold thinking" and checklists focused on immediate symptoms, failed to connect the dots over time. The system lacks the workflow, tools, and incentives for doctors to analyze patient-level historical data for chronic conditions, leaving a "vast canyon" between what data could do and what it does.

Barriers to adoption. While AI offers smart medical devices (iKnife, artificial pancreas), advanced medical imaging (skin cancer detection, eye disease), and remote medicine (wearable sensors), widespread adoption faces significant cultural and systemic hurdles:

• Incentives: Hospital business models often profit from chronic disease, disincentivizing long-term preventative care. Legal liability for AI-driven decisions is also unclear.
• Data Sharing: Lack of common data standards and hospitals' reluctance to share data (often viewed as corporate secrets) prevent the large-scale datasets needed for effective AI.
• Data Quality & Privacy: Medical data is often messy, and while "differential privacy" technologies exist to secure individual records, hospitals are slow to implement them.

8. AI's Effectiveness Hinges on Human Assumptions and Vigilance Against Bias

To illustrate why, we’ll consider a simple, very specific scientific question: Do osteoporosis drugs cause cancer of the esophagus? This is exactly the kind of question that people who work on AI for health care would love to be able to answer automatically, using fancy algorithms turned loose on enormous databases of health information.

The human element. Despite the power of AI, machines cannot operate without human assumptions, theories, and oversight. The case of osteoporosis drugs and esophageal cancer highlights this: two studies, using the same public database, reached opposite conclusions due to different human-made assumptions in their study design. AI algorithms merely execute instructions based on programmed assumptions; they cannot propose, test, or justify their own.

Pitfalls of poor assumptions. When poor assumptions are embedded in AI models, the consequences can be amplified exponentially:

• Rage to Conclude: Extrapolating too far from data with dubious assumptions, like The New York Times's misleading long-term contraceptive failure rates, which ignored lurking variables (e.g., user adherence). This can cause widespread anxiety or lead to algorithmic errors like inverse bidding wars or inappropriate T-shirt designs.
• Model Rust: Models degrade over time if not continuously updated and "seasoned" with new data. Google Flu Trends failed because it didn't adapt to changing search behaviors and underlying assumptions about search term correlation with flu activity.
• Bias In, Bias Out: AI models trained on biased data will learn and perpetuate those biases. The U.S. Army's tank detection model learned to identify shadows, not tanks. The COMPAS recidivism algorithm showed racial bias, likely reflecting historical biases in the criminal justice system's data, not inherent algorithmic prejudice.

Transparency and oversight. The solution to algorithmic bias is not to abandon AI but to demand transparency, accountability, and human vigilance. Secret algorithms, like COMPAS, prevent scrutiny and correction. While human decision-makers also suffer from biases (e.g., in hiring or sentencing), AI offers the potential for transparent, auditable systems whose biases can be identified and corrected. Combining artificial intelligence with human insight and values is crucial for navigating the complex ethical and societal implications of this new age.

Last updated: March 23, 2026