But a new paper from Google Research, presented at NeurIPS 2025, argues that this view is an illusion.

The paper, titled “The Illusion of Deep Learning Architectures” ¹, proposes a radical shift: Neural networks aren’t just stacks of filters; they are systems of nested optimization problems. Every layer in your model is actually a little “learner” trying to solve its own objective function in real-time.

If you’ve ever wondered why Transformers are so good or how we can make models that actually learn after training, this paper offers a mathematical explanation that is surprisingly elegant.

The Core Insight: Architecture = Optimization

The most mind-bending part of the paper is the proof that well-known architectures are mathematically identical to well-known optimization algorithms.

Let’s look at Linear Attention, a simpified version of standard component in modern efficient Transformers. Its update rule looks like this:

Here, $\mathcal{M}$ is the memory state, and it updates by adding the outer product of the current value ($v$) and key ($k$).

Now, let’s look at Gradient Descent (GD), the algorithm we use to train networks. The update rule for weights ($W$) is:

The researchers asked a simple question: What if the Attention layer is actually running Gradient Descent on itself?

They define an “Associative Memory” objective, a loss function that measures how well the memory maps keys to values :

If you calculate the gradient of this loss with respect to the memory $\mathcal{M}$, you get $-v_t k_t^\top$. Plug that into the Gradient Descent formula (assuming a learning rate $\eta=1$), and you get:

This is the exact same formula as Linear Attention.

This proves that a Linear Attention layer isn’t just a static block; it is a dynamic solver running one step of Gradient Descent on every token to “learn” the current context .

From “Heuristic” to “Hope”

Once you realize that Architecture is Optimization, you can stop guessing new architectures and start deriving them.

If Linear Attention is equivalent to basic Gradient Descent (which we know is kind of slow and clunky), why not upgrade the “inner optimizer”?

• What if we used Momentum? The paper shows that Momentum acts as a “Meta-Memory” that compresses gradients .
• What if we used Adam?
• What if we used a Neural Network to predict the updates?

This line of thinking led the authors to develop HOPE, a new architecture that treats the model as a “Continuum Memory System”.

The HOPE Architecture: Mimicking Brain Waves

Current LLMs suffer from “Anterograde Amnesia”. They can process the immediate context (short-term) and remember their pre-training (long-term), but they can’t form new long-term memories after deployment .

HOPE fixes this by mimicking the human brain’s oscillations (brain waves). The brain doesn’t have one clock; it has fast Gamma waves for immediate processing and slow Delta waves for long-term consolidation.

HOPE implements this mathematically by nesting optimization loops with different “frequencies”:

1. High-Frequency Neurons: These update rapidly (like standard attention) to handle the immediate token-by-token flow.
2. Low-Frequency Neurons: These are “slow learners” that only update every $C$ steps, consolidating information into long-term storage.

This structure allows the model to be Self-Modifying. It doesn’t just read data; it fundamentally rewrites its own parameters on the fly, allowing for true continual learning without catastrophic forgetting .

Why This Matters

This paper is exciting because it moves us away from the black-box alchemy of “adding more layers”. It gives us a white-box theoretical framework: if you want a better architecture, design a better optimizer.

By unifying the math of how we learn (optimization) with what we build (architecture), Nested Learning suggests that the next generation of AI won’t just be bigger, it will be smarter about how it manages its own mind.

– Ali

References

The Uneven Map of “Map AI”

When it comes to spatial and temporal data such as maps, mobility traces, satellite images, the progress is far less tidy. We have excellent point solutions: routing apps, traffic forecasting, crop-yield estimators, wildfire monitors. Yet the field lacks the shared foundation that NLP enjoys. Datasets are fragmented, privacy concerns limit data sharing, and the problems themselves span wildly different scales.

Think about the challenges still wide open:

• Predicting city-scale traffic during an unexpected event.
• Updating a map in real time after a hurricane destroys infrastructure.
• Forecasting the spread of an epidemic using only anonymized mobility signals.
• Monitoring illegal deforestation from noisy satellite streams.
• Having a highly noisy GPS traces (a lot of spelling mistakes) and trying to figure out the actual path taken on a map and performing the target task well.

A Question of Representation

In language, the breakthrough came from representation: words and sub-words as sequences of tokens. Could we find an equally universal representation for the physical world? One where, once the model has “read” enough of it, we can ask complex questions about traffic, climate, migration, and receive grounded answers?

That’s the dream: a language of space-time.

Sketching a Language of Space-Time

Imagine a representation $\mathcal{R}$ that treats space and time as first-class citizens and combines:

1. Objects – discrete entities like roads, buildings, rivers, or even people, each with geometry and attributes.
2. Fields – continuous measurements such as temperature, elevation, air quality.
3. Events – point occurrences anchored in space and time: an accident, a flood, a festival.
4. Relations – topological or metric links: adjacency, containment, causality.
5. Dynamics – how objects and fields evolve over time, capturing motion, growth, decay.

Think of $\mathcal{R}$ as a typed spatio-temporal knowledge graph where a city’s road network, the shifting jet stream, and a person’s walking trajectory all live in one coherent structure.

On top of this, we could define tasks just as NLP defines next-token prediction:

• Prediction: forecast future states of $\mathcal{R}$ (traffic, weather, population flows).
• Reasoning: answer complex spatial queries or infer causes of observed patterns.
• Optimization: search for interventions like rerouting traffic, allocating emergency resources that reshape $\mathcal{R}$ toward a goal.

Just a Thought, for Now

This is only an idea, a sketch of what might be possible. But if language models taught us anything, it’s that a powerful, shared representation can ignite an entire field. Perhaps one day we’ll have a “universal geo-time model” that understands our planet as fluently as today’s LLMs understand text.

I understand that the language is a symbolic one-dimensional representation, while the world is continuous and multi-dimensional. Maybe even comparing the two is like comparing apples and oranges. But the success of LLMs makes me wonder if we can find a middle ground.

For now, it remains a dream I hope to explore and maybe, someday, help bring to life.

– Ali

I’ve always wanted to write about game theory but never did. Recently, I started flipping through my old game theory books and came across mechanism design. Today, I’ll mention something that I think you can also use in your life: division problems and their different variations.

You can google Game Theory, Alive¹, I would say that’s a good book. Also it includes topics on Stable Matching, Voting, Auctions, Truthfulness, and more. But today, I want to focus on fair division.

Cake-Cutting Problem

Let’s assume there is a cake we want to divide between two people. What do we do? What is the best way to divide it? Obviously, one person cuts and the other one picks. There is no way either of them would be upset: they have gotten at least $1/n$ of their valuation.

Let me give an example. I love the cherries on the cake and I’d say that’s the best part you can get. So I divide it so that one cherry is on one side and the other cherry is on the other side. Now I offer my friend to choose. If he has the same taste, he can get half of the best part and I will too; but if he doesn’t like cherries, he can choose whichever part makes him happier. So he will definitely get half or more of the part he like.

This gets a little bit tricky when it becomes $n \ge 3$. What should we do then so we can still guarantee fairness for any number of people, and what kind of procedure would work when everyone values the cake differently? Think about it before jumping to the end of the post 😛

Mathematically speaking, if we want to divide the continuous range $[0,1] \subset \mathbb{R}$ (the cake) and each of $n$ participants has a valuation function $v_i$ for the cake, if the following condition holds, we can say it is a fair division:

Fair (proportional) division condition: find a partition $(r_1,\dots,r_n)$ of $[0,1]$ such that

\begin{equation} v_i(r_i)\ \ge\ \frac{1}{n}\qquad \forall i\in{1,\dots,n}, \end{equation}

assuming $v_i([0,1])=1$.

Envy-Free Division

In addition, we have a more restricted condition; it’s called envy-free division. This means that no one should be envious of another person’s share.

Mathematically speaking, we can express this as follows: \(v_i(r_i) \ge v_i(r_j) \quad \forall i,j \in [1,n].\)

This guarantees a fair division; it is stricter than simple fair division.

But finding an envy-free division is more challenging. I will write about it in the next post.

Simple Fair Division Algorithm for any $n$

Did you find the solution for the simple fair division for the $n$ people involved for the cake?

References

While this might be true if we have distilled the logits or probabilities⁴, they say that it is transmitted just by a text that is not related (e.g., sequences of numbers).

This is interesting for me in three aspects:

• Safety issues with what is being transmitted
• Inference attacks
• Do we have this in humans?

Safety

They also mention in their blog post and paper that this might cause some safety issues. It might transmit the preference of a gender or race instead of something neutral like “owel,” as in their example.

Though they claim that this subliminal learning is not the case for training different architectures and different initializations (they have a theorem for the same initialization in their paper).

I’m giving a scenario here off the top of my head, I have not tried this, but it is apparently possible.

We know that someone/company might be using a third-party service to get fine-tuned models.
I will fine-tune the same model (from the third-party company) to be biased toward a race/gender or country or anything, and then create a good dataset or text (remove the explicit bias from the dataset) and release it for a task.

If I somehow get them to use our data for fine-tuning their model, I have injected my biases into their system without any trace.
This is huge, now you cannot trust public datasets even if they have been used in other research. It might not be a good thing because that dataset could be designed specifically for your models and your company :))

Model Inference

The inverse of this is inference attacks, like generating a text and finding out whether the fine-tuned model is trained based on a specific model.
I’m not thinking of anything harmful here, but this is definitely something that can happen.

In Humans

Last but not least, does this happen in humans? I’m not sure, but I think it’s possible.
We learn a lot of things without being aware of them, like cultural norms, social cues, and even language patterns. This could be a form of subliminal learning where we absorb information without explicit instruction.

Can we be racist without explicitly being taught to be?
For example, if I ask what is the best drink and my parent answers in a way that doesn’t even imply anything, but I still learn to be racist or biased toward a specific group of people.
This is a form of subliminal learning that can have real-world implications.

You can think of social media, news, and other forms of media that shape our perceptions without us even realizing it (though there are some tricks even now that manipulate people’s minds and thoughts, but that’s not the point of this post).

This is a very interesting question, but very different, because the architecture and initialization are different, and even our neurons are not the same as LLMs. Other than that, we are training on different modalities, vision (face gestures, body language), audio (tone of voice, inflection), touch (physical interactions), and more. Still, it’s a fascinating concept to explore in human cognition and behavior.

Conclusion

Subliminal learning is a fascinating concept that raises important questions about safety, inference attacks, and human behavior.
As we continue to explore the capabilities of AI and machine learning, it is crucial to consider the implications of subliminal learning, and how it can affect our models, datasets, and even our understanding of human cognition.

– Ali

References

I asked Gemini to go through the full list of ICML 2025 papers, titles and provided with the link to the paper, and pick 50 that seem especially promising. The selection favors papers with conceptual novelty, foundational depth, and related to Foundation models, interpretability, or generative models.

Did I get it right? Maybe. Maybe not. That’s part of the experiment. Maybe a different prompting can lead to a totally different list.

How This List Was Made

This is not a leaderboard or a citation-based ranking. I wasn’t given metrics, reviews, or author prestige. Just the raw metadata from ICML 2025, titles, and links.

But ultimately, this is a speculative curation. I haven’t read the full texts (and neither have most people, because they were just released). The goal is to surface ideas that seem worth deeper attention, not to declare winners.

List of 50 Papers

1. Strategy Coopetition Explains the Emergence and Transience of In-Context Learning
   • Abstract summary: This paper likely presents a new theory that models in-context learning as an emergent phenomenon arising from both competitive and cooperative dynamics among model components.
   • Reason for inclusion: High relevance to foundational models, offering a novel theoretical explanation for a key, yet poorly understood, capability.
2. MIB: A Mechanistic Interpretability Benchmark
   • Abstract summary: Introduces a new benchmark (MIB) designed to systematically evaluate and compare techniques in mechanistic interpretability.
   • Reason for inclusion: Foundational for the field of interpretability. A standardized benchmark is critical for making measurable progress.
3. Elucidating the design space of language models for image generation
   • Abstract summary: This work likely explores and maps out the key architectural and data-related choices when using LLMs to drive or control image generation processes.
   • Reason for inclusion: Directly addresses the intersection of generative models and foundational models, a key area of current research.
4. Subspace Optimization for Large Language Models with Convergence Guarantees
   • Abstract summary: Proposes a new optimization method for LLMs that operates in a lower-dimensional subspace, likely improving efficiency while providing theoretical guarantees of convergence.
   • Reason for inclusion: Addresses the critical practical challenge of training and fine-tuning massive foundational models with a theoretically grounded approach.
5. R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models
   • Abstract summary: Introduces a method that leverages LLMs to automatically design and optimize complex reward functions for reinforcement learning.
   • Reason for inclusion: Novel application of LLMs to automate a traditionally human-intensive task in RL, with high potential impact on training more capable agents.
6. Let LLM Tell What to Prune and How Much to Prune
   • Abstract summary: This paper likely presents a novel method using a large language model to intelligently decide which parts of a neural network to prune for efficiency.
   • Reason for inclusion: Innovative use of LLMs for model optimization, falling under the umbrella of interpretability and practical application of foundational models.
7. MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning
   • Abstract summary: Introduces MODULI, a method that uses diffusion models to better generalize user preferences in a multi-objective RL setting.
   • Reason for inclusion: High novelty, combining generative diffusion models with reinforcement learning to solve a challenging preference alignment problem.
8. MCU: An Evaluation Framework for Open-Ended Game Agents
   • Abstract summary: Proposes a new framework (MCU) for evaluating agents in complex, open-ended gaming environments, a key challenge for modern AI.
   • Reason for inclusion: Foundational contribution to evaluation methodology, which is crucial for progress in agentic AI and foundational models.
9. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
   • Abstract summary: This paper introduces an innovative caching technique (ShadowKV) to significantly speed up inference for LLMs with very long contexts.
   • Reason for inclusion: Addresses a major bottleneck in the practical deployment of large foundational models, making it highly impactful.
10. SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering
    • Abstract summary: Focuses on the crucial but understudied problem of how AI agents recover from errors and get “back in sync” when collaborating on software development tasks.
    • Reason for inclusion: Tackles a frontier problem in agentic AI and foundational models, focusing on the practical challenges of multi-agent collaboration.
11. UnHiPPO: Uncertainty-aware Initialization for State Space Models
    • Abstract summary: Proposes a new initialization method for State Space Models (SSMs) that explicitly accounts for uncertainty, likely improving stability and performance.
    • Reason for inclusion: SSMs are a major alternative to Transformers. Foundational work on improving their training and stability is highly relevant.
12. SE(3)-Equivariant Diffusion Policy in Spherical Fourier Space
    • Abstract summary: This paper likely develops a diffusion-based policy for robotics or 3D tasks that correctly handles 3D rotations and translations by operating in a spherical Fourier space.
    • Reason for inclusion: Connects generative models (diffusion) with important geometric principles (equivariance) for physical interaction tasks, representing a key research direction.
13. Fast Video Generation with Sliding Tile Attention
    • Abstract summary: Introduces a novel attention mechanism (“Sliding Tile Attention”) to make video generation models faster and more efficient.
    • Reason for inclusion: Directly addresses a key challenge in generative AI (video generation speed) with a new architectural innovation.
14. Preference learning made easy: Everything should be understood through win rate
    • Abstract summary: Proposes a unifying, simplified framework for preference learning, suggesting that various methods can be interpreted and analyzed through the simple metric of win rate.
    • Reason for inclusion: Potentially foundational work that simplifies and unifies the theory behind aligning models with human preferences, a cornerstone of modern LLM development.
15. How Far are We in Multimodal Large Language Models?
    • Abstract summary: This paper likely provides a comprehensive survey and analysis of the current capabilities and limitations of multimodal LLMs.
    • Reason for inclusion: Foundational benchmark/survey paper that helps structure the understanding of a rapidly evolving field.
16. Generative Pre-training for Tabular Data
    • Abstract summary: Explores the application of generative pre-training, a successful paradigm from LLMs, to the domain of tabular data.
    • Reason for inclusion: Extends the foundational model paradigm to a different data modality, which has significant practical implications.
17. Beyond Chain-of-Thought: A Survey on Complex Reasoning of Large Language Models
    • Abstract summary: A survey paper that likely categorizes and reviews advanced reasoning techniques in LLMs that go beyond simple Chain-of-Thought prompting.
    • Reason for inclusion: Provides a structured overview of the cutting edge in LLM reasoning, a central topic for foundational models.
18. Do Large Language Models Know about Their Generation Probability?
    • Abstract summary: Investigates whether LLMs have an internal, calibrated sense of the likelihood of the text they generate, which is key for reliability.
    • Reason for inclusion: Addresses a fundamental question about the self-awareness and calibration of foundational models, relevant to both interpretability and safety.
19. A Mechanistic Understanding of Alignment: A Case Study in Function Vectors
    • Abstract summary: This work likely provides a mechanistic interpretability analysis of how alignment techniques like RLHF change the internal representations of a model.
    • Reason for inclusion: High relevance to interpretability, aiming to explain the “how” and “why” of model alignment procedures at a mechanistic level.
20. UniG-guided: A Unified and Guided Framework for All-in-one Image Generation
    • Abstract summary: Proposes a single, unified framework that can handle various conditional image generation tasks (e.g., text-to-image, editing, inpainting) within one model.
    • Reason for inclusion: High impact potential for generative models, moving towards more consolidated and versatile creative tools.
21. Mega-T: A General-purpose Foundation Model for Tabular Data
    • Abstract summary: This paper likely introduces a new large-scale, pre-trained model (“Mega-T”) specifically designed for a wide range of tasks on tabular data.
    • Reason for inclusion: Clear attempt to create a foundational model for the tabular data domain, a significant and practical research direction.
22. Scaling Laws for Forgetting in Large Language Models
    • Abstract summary: Investigates the relationship between model scale and catastrophic forgetting, likely deriving scaling laws that predict how and when LLMs forget information.
    • Reason for inclusion: Foundational theoretical work on a critical limitation of current models, relevant for continual learning and model updates.
23. Emergent Communication Explains the Surprising Reasoning Skills of Large Language Models
    • Abstract summary: Proposes a novel theory that the reasoning abilities of LLMs are an emergent consequence of learning to communicate effectively.
    • Reason for inclusion: Offers a high-level conceptual explanation for the emergent properties of foundational models, a key scientific question.
24. What happens when a transformer-based language model is a compression algorithm?
    • Abstract summary: Explores the deep connection between language modeling and data compression, likely analyzing model internals from a compression perspective.
    • Reason for inclusion: High relevance to interpretability and foundational understanding, framing model behavior through a powerful theoretical lens (Kolmogorov complexity).
25. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
    • Abstract summary: Introduces a fine-tuning method based on self-play, where a language model improves itself, and shows this can dramatically boost the capabilities of smaller models.
    • Reason for inclusion: High potential impact, suggesting a path to create powerful models without needing to start with a massive base model.
26. A Theory of Counter-Example Guided Interpretability
    • Abstract summary: Develops a theoretical framework for interpretability methods that rely on generating counter-examples to explain model decisions.
    • Reason for inclusion: Foundational contribution to the theory of interpretability, providing rigor to a popular class of explanation methods.
27. Generative World Models for Transfer in Reinforcement Learning
    • Abstract summary: This paper likely uses generative models to build “world models” that an RL agent can use to simulate future outcomes, specifically to improve transfer to new tasks.
    • Reason for inclusion: Connects generative models with RL to tackle the key challenge of generalization and transfer learning.
28. Diffusion Models as Interpretable and Controllable Recommender Systems
    • Abstract summary: Proposes using diffusion models for recommendation tasks, arguing this approach offers better interpretability and user control over recommendations.
    • Reason for inclusion: Novel application of generative models to a classic domain, with a specific focus on the benefits for interpretability.
29. Vision-Language Models as a Source of Reward
    • Abstract summary: Explores using pre-trained vision-language models to provide dense reward signals for training robotic or embodied agents, avoiding manual reward engineering.
    • Reason for inclusion: Leverages the knowledge in foundational models to solve a bottleneck in another field (RL), a powerful and trending paradigm.
30. The Expressive Power of Transformers with Chain-of-Thought
    • Abstract summary: A theoretical paper analyzing the expressive power of the Transformer architecture specifically when it is prompted to use Chain-of-Thought reasoning.
    • Reason for inclusion: Foundational theory for LLMs, aiming to understand why and how popular prompting techniques actually work.
31. The Geometry of Truth: Visualizing Language Model Representations
    • Abstract summary: This work likely introduces new techniques for visualizing the internal representation space of LLMs to understand how they represent concepts like truthfulness.
    • Reason for inclusion: Directly addresses interpretability with a focus on novel visualization methods for understanding abstract properties.
32. Watermarking Large Language Models with Backdoored Prompts
    • Abstract summary: Proposes a method for watermarking LLMs to trace their outputs, likely by embedding a secret trigger via backdoored prompts during training.
    • Reason for inclusion: Important topic at the intersection of model security, ownership, and interpretability (as it involves manipulating model behavior).
33. Can We Trust the Interpretations of Local Interpretable Model-Agnostic Explanations (LIME)? A Theoretical and Empirical Analysis
    • Abstract summary: A critical analysis of LIME, one of the most popular interpretability techniques, likely examining its theoretical soundness and empirical reliability.
    • Reason for inclusion: Foundational interpretability work that provides a necessary critical perspective on widely used tools.
34. Diff-Pruning: Pruning via Diffusion-based Weight Importance
    • Abstract summary: Introduces a new model pruning technique where the importance of weights is determined using a diffusion process.
    • Reason for inclusion: A novel and interesting combination of generative model techniques (diffusion) with the goal of model compression and efficiency.
35. ExpProof : Operationalizing Explanations for Confidential Models with ZKPs
    • Abstract summary: This paper likely proposes a method to provide verifiable explanations for a model’s prediction without revealing the model’s weights, using Zero-Knowledge Proofs.
    • Reason for inclusion: Highly innovative work connecting interpretability with cryptography to solve the problem of explaining private or proprietary models.
36. Towards a Foundation Model for Proteins
    • Abstract summary: Likely describes the development and training of a large-scale, pre-trained model for a variety of tasks related to protein biology.
    • Reason for inclusion: A clear example of extending the “foundational model” concept to a critical scientific domain beyond language.
37. Training-Free Structure-Guided Diffusion Model for 3D Molecule Generation
    • Abstract summary: Proposes a diffusion-based model for generating 3D molecules that can be guided by structural constraints without requiring additional training.
    • Reason for inclusion: Strong relevance to generative models, applied to the high-impact area of drug discovery and molecular design.
38. Emergent Modularity in Pre-trained Transformers
    • Abstract summary: Investigates how and why modular structures (sub-networks with distinct functions) spontaneously emerge within large pre-trained Transformer models.
    • Reason for inclusion: High relevance to interpretability and understanding the internal structure of foundational models.
39. Are Large Language Models Born with Grand-Tour capability?
    • Abstract summary: This paper likely investigates the innate ability of LLMs to perform “grand tour” data visualization, a technique for exploring high-dimensional data, in their latent spaces.
    • Reason for inclusion: Connects interpretability of foundational models with established methods in data science and visualization in a novel way.
40. Decoding the Internal State of a Large Language Model during Code Generation
    • Abstract summary: An interpretability study focused specifically on understanding the step-by-step internal reasoning of an LLM as it generates computer code.
    • Reason for inclusion: Highly relevant to interpretability, focusing on the important and complex domain of code generation.
41. Unsupervised Data Selection for Language Model Fine-tuning
    • Abstract summary: This work likely introduces methods for automatically selecting the best data for fine-tuning an LLM without needing labeled examples or human supervision.
    • Reason for inclusion: Addresses a critical practical problem in the effective use of foundational models, with high potential impact on efficiency.
42. General Purpose In-Context Learning Agents
    • Abstract summary: Explores the use of in-context learning to create general-purpose AI agents that can adapt to new tasks on the fly without updating their weights.
    • Reason for inclusion: Pushes the boundaries of what can be achieved with foundational models, focusing on the paradigm of agent-based AI.
43. Does the Generator Mind All Specifications? Evaluating Object Consistency of Text-to-Image Generative Models
    • Abstract summary: This paper likely introduces a new benchmark or methodology for systematically evaluating how well text-to-image models follow all parts of a complex prompt, focusing on object consistency.
    • Reason for inclusion: Important evaluation work for generative models, addressing the key challenge of controllability and faithfulness to user intent.
44. A Diffusion-based Framework for Counterfactual Explanation
    • Abstract summary: Proposes using generative diffusion models as a new way to create counterfactual explanations, which describe the smallest change needed to alter a model’s prediction.
    • Reason for inclusion: High novelty, combining state-of-the-art generative models with a key interpretability technique.
45. G-Retriever: A General Generative-Retriever Framework for Large Language Models
    • Abstract summary: Introduces a unified framework that combines generative and retrieval-based approaches for LLMs, likely to improve factuality and knowledge grounding.
    • Reason for inclusion: Addresses the critical issue of augmenting foundational models with external knowledge, a very active and important research area.
46. How do Language Models Bind Entities? A Study of the Naming-Related Circuit
    • Abstract summary: A mechanistic interpretability paper that seeks to identify the specific neural circuit within a language model responsible for associating names with entities.
    • Reason for inclusion: A prime example of mechanistic interpretability, digging deep into how a specific, fundamental capability is implemented in a model.
47. Language Models are Weak Agents, But Strong Tool Users
    • Abstract summary: This paper likely presents a study arguing that the main power of LLM-based agents comes not from their innate planning abilities, but from their effective use of external tools.
    • Reason for inclusion: Offers a clear and potentially influential perspective on the architecture and capabilities of agents built on foundational models.
48. Understanding and Mitigating Hallucinations in Large Vision-Language Models
    • Abstract summary: This work provides an analysis of why multimodal vision-language models hallucinate facts and proposes methods to reduce this behavior.
    • Reason for inclusion: Addresses a critical failure mode (hallucination) in foundational models, which is essential for building reliable systems.
49. World Model on Million-Length Video and Language
    • Abstract summary: Describes the creation of a “world model” trained on extremely long sequences of video and text, likely pushing the state-of-the-art in long-context modeling.
    • Reason for inclusion: High-impact, large-scale work on generative and foundational models, tackling the frontier of long-sequence understanding.
50. The Sparse Transformer: A New Architecture for Language Modeling
    • Abstract summary: Proposes a new Transformer architecture based on sparse connectivity patterns, likely aiming for significantly improved efficiency without sacrificing performance.
    • Reason for inclusion: Potential for a foundational architectural innovation that could influence the next generation of LLMs.

Why Use AI to Curate?

Because even experts can’t read everything, and sometimes it helps to bring a fresh (or weird) lens. You might disagree with some of the picks. That’s great. Use this list as a conversation starter, not a verdict. What would your top 10 look like? Which of these would you bet on becoming influential a year from now?

🤷 So… Which One Should You Read?

Honestly? I don’t know yet. That’s the point. Some of these may be overhyped. Others might be quiet classics. But all of them seem to reach beyond the incremental. As the full papers come out and people dig into the math, we’ll see which ones hold up, and which ones quietly change the way we think. In general, one good heuristic is always to look into papers that you know their author/lab that are well-known researchers, which we haven’t done here. You can also follow your favourite researcher to see their tweets about their paper 😉. But I think this list is a good starting point to find some hidden gems.

🧭 What’s Next?

I’d love to do follow-ups:

• A deep dive into the most cited vs. least cited of these picks in six months
• A themed breakdown: top picks in interpretability, causality, generative models, etc.
• A comparison between Gemini-selected vs. community-selected papers But for now, here’s the bet: At least a few of these 50 will turn out to matter more than their titles suggest.

My thought

I don’t know which of these papers are truly important, or not. But the fact that Gemini selected them shows one thing clearly: it’s judging based only on titles. So if we build research agents (or LLM-based search engines), they might prefer such papers, and that could lead to them being cited more frequently. (You get the idea.) So I’m not even sure that citation count is a reliable metric for comparison later on. Anyway, that’s my final note. If you find one you love, or one you hate, drop me a note.

This Disclaimer is clear but I have to put it anyways.

A Surprise at the end

The papers without links are the ones I couldn’t find in the official ICML 2025 listings. I even searched for a few of them, turns out, they’re not real papers at all. Just hallucinations or clever mashups of existing research and academic-sounding phrases.

So hey, pick a title and write your own paper, Gemini might just love it. 😂

PS: My Prompt was:

You are an expert machine learning researcher and assistant. Your task is to select the 50 most relevant papers from the ICML 2025 conference, based on a provided list of paper titles and relative URLs (e.g., "/virtual/2025/poster/44561").

Each item is an HTML list entry:
<li><a href="/virtual/2025/poster/44561">Strategy Coopetition Explains the Emergence and Transience of In-Context Learning</a></li>

### My goals:
- Select 50 papers from ~3000 entries
- Prioritize high-quality, novel, and relevant research
- I’m particularly interested in the following topics: Foundation models, interpretability, generative models.
- Papers should be:
  - Innovative or influential
  - Related to the above topics or clearly high-impact
  - Either theoretically foundational or practically useful

### What you should do:
1. **Extract** the paper title and full URL (prepend "https://icml.cc" to the relative URL)
2. **Visit each paper’s ICML page**, if accessible, and extract:
   - The abstract
   - Authors
   - Any listed keywords or presentation format (e.g., spotlight/oral)
3. **Score each paper** based on:
   - Relevance to the topics I care about
   - Novelty and clarity of the abstract
   - Expected impact (e.g., new benchmarks, theory, or trends)
4. **Select the top 50 papers**
5. **Output** your result in the following format:
```

1. [Paper Title](https://icml.cc/virtual/2025/poster/44561)

   * Abstract summary (1–2 lines)
   * Reason for inclusion (e.g., topic fit, novelty, etc.)

```

Only output the final selected list of 50 papers — do not explain your methodology unless asked.

I am attaching the list of papers in a .txt file, read all of the papers from that.

– Ali

The pretrained model paradigm has revolutionized many areas of machine learning, particularly in natural language processing and computer vision. The idea is to pretrain a model on a large dataset and then fine-tune it for specific tasks. But what about trajectory data? Can we apply the same principles to GPS trajectories?

Trajectory data tokenization

First we need to find a way to convert the trajectory data into a format suitable for pretraining like a corpus of text. The raw GPS trajectories are sequences of latitude and longitude points, but we can convert them into a sequence of tokens. For example, we can use the H3 geospatial indexing system to convert the latitude and longitude points into hexagonal grid cells. This allows us to represent the trajectory as a sequence of hexagonal grid cell IDs.

Is the the tokenized data similar to a corpus of text?

In terms of tokens, the models does not know the difference between a sequence of words in a sentence and a sequence of hexagonal grid cell IDs in a trajectory. The model learns to understand the relationships between the tokens, regardless of their nature. But there is a key difference is that in trajectory data the tokens are not independent of each other, there is a spatial relationship between the tokens. For example, in a hexagonal grid, if we model the trajectory as a continous consecutive sequence of hexagonal grid cell IDs, the model should know that the tokens are spatially related, and we only have six possible neighbors for each token. This is different from a text corpus, where the tokens can be more independent of each other. (Though, if we model the trajectory as a sequence of check-ins, the tokens can be more independent of each other, similar to a text corpus.)

Pretraining the Trajectory Foundation Model

Once we have tokenized the trajectory data, we can pretrain a model using a large dataset of GPS trajectories to learn the underlying patterns and structures in the data. We can use a transformer-based architecture, similar to those used in NLP and computer vision, to learn the relationships between the tokens in the trajectory sequences.

We can train encoders such as BERT specifically designed for learning representations of sequences. Or we can use autoregressive models like GPT to predict the next token in a trajectory sequence¹. The goal is to learn a rich representation of the trajectory data that can be fine-tuned for specific tasks, such as trajectory prediction, anomaly detection, or clustering. I have trained an end-to-end trajectory classification model in the this link. In this code example, we can change the objective to learn a language representation by using a masked language model objective similar to what original BERT does.

Considerations for MLM for pretraining

One thing that comes to my mind is that we cannot blindly apply masking to a dataset represented by consecutive tokens spatially. This is because we do not even need a language model to learn the missing part between two adjacent tokens. If we want to find the missing part at location $m$, we can simply get the neighbors of the previous token $m-1$, which is $\mathcal{N}(m-1)$, and the neighbors of the next token $m+1$, which is $\mathcal{N}(m+1)$, and then find the intersection of these two sets: $\mathcal{N}(m-1) \cap \mathcal{N}(m+1)$. This works in a hexagonal grid. However, when data is represented by check-ins, we can use the same masking strategy as in language models, because the tokens are more independent of each other.

Fine-tuning the Trajectory Foundation Model

After pretraining the trajectory foundation model, we can fine-tune it for specific tasks. For example, we can fine-tune the model to predict the next location in a trajectory sequence, or to classify trajectories into different categories based on their patterns.

We can also use the pretrained model to extract features from the trajectory data, which can be used for downstream tasks such as clustering or anomaly detection.

References

I searched online and found this student project Traffic Simulation Exploring the Impacts of Jaywalking on Traffic Flow. I did not go through the details of the implementation but it is an interesting simulation.

Based on their assumption, it seems that jaywalking at a specifc ratio gives the optimal travel time. 😁

I am interested in confirming these results based on some new assumptions and finding the optimal jaywalking ratio.☺️ Also, I guess it will be great if I can add a parameter as careless Jaywalker ratio who can make trouble and be hit by a car with a low probability. 🙈

I should stand there for a day to log data to see if we are collaborating at an optimal point in our society. 👀

Will do the similuation soon and wite a post on this.

Stay curious.

– Ali

What is Ace Tools?

I had never seen this library before, and I thought it was a new library that might need to be installed in a different way. I searched for it on PyPI, but I could not find it there either. I found out that it is an internal library of OpenAI and is not publicly available. It is perhaps just showing the output to be displayed in the chat and is not meant to be used outside of OpenAI. There is a thread about it here: ChatGPT recommends the use of the Open AI internal library ace_tools

Here is an example of how it is used in the code generated by ChatGPT:

import pandas as pd
from ace_tools import display_dataframe_to_user

# doing some analysis and creating the data variable.
data = {...}

df = pd.DataFrame(data)

# Display to user
display_dataframe_to_user(name="Table Title", dataframe=df)

Security Risks

Currently, someone has gotten the library name on PyPI, and it does nothing but act as a placeholder. I do not know who registered the library name, but I think they could easily push some malicious code.

If you are using ChatGPT and copying the code it generates, be careful about the libraries it uses. If you see a library that is not familiar, search for information about it first. It might be an internal library of OpenAI or any other company, and using publicly available libraries that resemble the name could be a security risk.

What can a library like ace_tools do?

I just did a simple search and found some libraries that have been pushed to PyPI which are doing malicious things. For example, a library that can steal your cookies or passwords. Here is a blog post from Fortinet that describes some of the malicious packages hidden in PyPI: Malicious packages hidden in PyPI

Conclusion

In conclusion, be careful about the libraries used in the code generated by ChatGPT. Do not blindly trust the code and install the libraries used in the code.

– Ali

I was just reading about this congtive bias and I thought it would be interesting to share it with you. I will write more about these biases in the futuer, although I know that there are tons of articles and books about them. But I think writing about something helps me to internalize the concept and understand it better.

There is a list of cognitive biases on Wikipedia (List of Cofnitive Biases) that you can check out. It is a great resource for learning about the different types of biases that can affect our thinking and decision-making processes. It is not a thing that you can read once and remember all of them, but it is a good starting point to be aware of them and sometimes catch yourself when you are falling into one of these traps.

Confirmation bias can manifest in various ways in research, affecting the design, interpretation, and reporting of studies. I guess one thing that can help us here is designing a good control group experiment and being open to alternative hypotheses. It is also important to critically evaluate the evidence and consider the limitations of the data before drawing conclusions.

Finally, I guess not only we should consider this bias in our research and papers, but also in reading other’s papers and research. It is important to critically evaluate the evidence and consider alternative viewpoints to avoid falling into the trap of confirmation bias.

BTW, there is a great book you may have heard, “The Art of Thinking Clearly” by Rolf Dobelli that I highly recommend. It is a great book that covers many cognitive biases and logical fallacies that we encounter in our daily lives. It is a great read for anyone interested in improving their critical thinking skills and decision-making abilities.

– Ali

To begin with, some papers argue that a certain degree of memorization is essential for models to generalize effectively ^{1 2}. This is a compelling point, as memorization appears to play a critical role in enabling generalization. However, this raises an intriguing question: What happens if we attempt to unlearn that memorized data?

Consider a simple example that highlights the importance of memorization. While this example is straightforward and might not directly apply to the intricate architecture of a large language model or a complex neural network, it illustrates the concept well.

In a classifier like SVM (Support Vector Machine), the support vectors—data points close to the decision boundary—are crucial. We practically memorize these data to have the decision boundary ³. If we remove these support vectors, the decision boundary shifts, and the model’s ability to generalize diminishes significantly. This illustrates the idea that retaining certain memorized elements is essential for effective generalization. You can see the visualization of SVM classifier with different support vectors and different decision boundaries below (because of the different kernels, generated by this code):

Additionally, knowledge itself is hierarchical and interconnected. For example, if I say I know linear algebra, it implies that I understand matrix multiplication, which in turn builds on a foundational understanding of scalar addition and multiplication. I cannot unlearn simple sum operations while retaining the ability to multiply matrices or the whole linear algebra knowledge.

This interconnected nature of knowledge leads to thought-provoking questions: Can I forget/change the sum operation while retaining more advanced concepts like matrix multiplication? Similarly, some types of knowledge, such as conclusions derived from a set of statements, are inherently complex. Can I unlearn the foundational statements while preserving the resulting conclusion?

These questions show that the unlearning is an interesting problem to work with and so many opportunities to have fundamental research.

– Ali

References