vasa50 minutes
Spine
- Sets up a story of humans scaling discovery/synthesis on paper
- weaves in the main question "if we have done this, why do we still suck at making materials"
- gives a brief of how we will answer the question
- introducing a framework that will help us understand the challenges in a structured way — discovery → synthesis → characterization
- introduce the inverse problem, on a high level — if we want to do something like: make me a room temperature/pressure superconductor that can be made into wires, is non-toxic, and can be made cheaply at scale — i.e. how do I go from defining a set of properties/requirements to an actual material in your hands?
- start from discovery — 29 June → 5 July
- structure to properties (and vice versa)
- schrodinger’s equation
- DFT — focus on periodic in-organic compounds — why it doesn’t work well for other kinds of materials — see what is state of the art for non-periodic materials
- DFT hull — shows the destinations (materials) and their energy levels (and their properties can be calculated)
- show that we have been doing well (at least for periodic in-organic materials)
- synthesis — 6 July → 12 July
- structure to material (and vice versa)
DFT hull — shows the destinations (materials) and their energy levels- How this DFT hull map is deceiving
- this map works on assumptions (conditions, which give us a very particular map) that we need to think about more
- this map is not static — this changes based on the conditions applied — i.e. if we change the assumptions — the map itself changes
- how discovery shows us the map of destinations (thermodynamics), but we have no idea about the path (kinetics)
- Ostwald’s step rule (kinetics) — explains why we see metastable phases in the first place — and not just stable phases
- different kinds of synthesis processes have different personalities — each of these processes allow for different kinds of kinetics to occur — Solid-state synthesis, Solution methods, Vapor and thin-film methods
- characterization — 13 July → 19 July
- material to structure (and vice versa)
- explain basics of how characterization works
- explain why the inverse is hard
- Summary — making sense of the challenges — 20 July → 23 July
- How are we going to solve it? — 24 July → 31 July
- go through each domain and its challenges and show what we are doing to solve it and how it has been going?
- Feedback and Marketing — 1 Aug → 8 Aug (at least, we can/should spend more time in distributing this if we need to)
1. It starts with a simple question
Here's a question that sounds dumb but isn’t.
We can design a new drug on a computer[^aidrug]. We can predict how a protein folds from its genetic code[^alphafold]. We can type “the pope wearing a white puffer jacket“ and get a photograph/video that never existed. So why can't we just... describe the material we want, something like "un-smashable phone screen”, press a button, and get a recipe to make it?
And it isn't for lack of trying, or even for lack of the tools you would assume would fix it. It's not that we can't dream up new materials — we have AI models that can generate millions of new materials[^gnome_mattergen]. And it's not that we can't build them — we literally have prototype labs where robots can run the experiments overnight[^a_lab] to create the materials that the AI model dreamed up, with no human in the room. On paper, the loop is closed: imagine a material, make a material. So, this problem should be solved, right?
It is very much not solved. In reality, the gap between a material we can imagine and a material we can hold in our hand has barely moved. So I want to spend this essay on the most honest version of a deceptively simple question:
Why do we still suck at making new materials?
And the only way to answer it is to actually walk the road from "I want something that does X" to a lump of it on a bench, and find, link by link, exactly where things fall apart.
2. How did we get here?
For most of history, we did not “discover” materials so much as stumble into them, and the line-up of stumbles is genuinely fascinating.
In 1669, a German alchemist named Hennig Brand was hunting the philosopher's stone[^philosopher]. He did what anyone of us would do in those days, which was boiling down dozens of buckets of human urine[^hennig] into a foul paste and heated it until turned into a glowing waxy substance that shined in the dark. This is how he became the first known person to have discovered an element, which was phosphorus.
In 1942, an American chemist named Harry Coover was trying to build clear plastic gunsights for WWII, but accidentally created a substance so sticky that it completely ruined his attempts. He got annoyed and threw the formula away. Nine years later, his team revisited the same formulation for a different project on jet canopies, only for a colleague to try and measure its properties by squeezing it between two expensive laboratory prisms, which unfortunately, permanently glued them together. When the colleague sheepishly admitted to Coover that he had accidentally ruined the expensive equipment, Coover finally went like, “Wait a second... this infuriating, tool-ruining goo isn’t a failed plastic at all, it's the world's first super glue.”
There are way too many of these “it’s not a bug, it’s a feature” discoveries to list here. If I tried to explain how Goodyear tires, Teflon, or stainless steel were all essentially the results of historic workplace accidents, this essay would never end.
Now, while relying on happy accidents to discover useful materials has served us well, the problem with this method is that it doesn’t scale well. Hence, eventually people trying to discover new materials got systematic about it. If you can’t predict which material would work for your use-case, the best you can do is to try them all, methodically, and let the winner reveal itself.
Edison is the most known example of this. As the lore goes, while searching for a long-lasting filament for his light bulb, he sent his team to different parts of the world, as far as to remote villages in Japan, to try and find different plant fibers, and then tested about ~6,000 different candidates (ranging from platinum wires to ordinary cotton thread), before settling on carbonized bamboo fibers.
Another historical grind was the quest to develop the Haber-Bosch process, a chemical reaction that could pull nitrogen out of the air to make fertilizer, which arguably is responsible for feeding a substantial fraction of humanity today. A chemist named Fritz Haber figured out how to make this reaction work in a tiny lab tube, but the reaction was agonizingly slow and yielded almost nothing. To make enough fertilizer to actually feed the world, they needed a chemical that could radically speed up the reaction under intense factory conditions. The job to find such a chemical fell to Alwin Mittasch, who systematically brute-forced it, running a staggering 6,500 test runs on over 2,500 different chemical combinations before stumbling onto a specific iron-based recipe. By his own later estimate he had tried on the order of 20,000 substances before he was done. More than a century later, his trial-and-error masterpiece is still the exact industrial catalyst running the global food supply.
We would have been stuck with the brute-force approach (accompanied by occasional happy accidents) if it weren't for the foundations laid by Erwin Schrödinger[^schrodinger] — yes, the same guy from the Schrödinger's-cat memes.
To really appreciate what Erwin Schrödinger did, let’s take a step back and try to answer a simple question:
Diamond and graphite (the thing used to make pencil lead) are both made up entirely of carbon atoms. Then why is one of them the hardest known natural material on earth while the other one breaks every time you use a pencil?
If these 2 things are made up of the same element, then what exactly is the difference between them? Maybe…the carbon atoms are arranged differently, which ends up giving these 2 things different properties.
This leads to some more questions:
- Given a particular arrangement of atoms, how do I know which one would become a hard substance and which one would become soft enough that you can write with it?
- How do I actually make it — arranging atoms in different ways?
- If I tried making it, how do I even check if I got the atoms in the right arrangement?
These 3 questions, quite cleanly map to how we think about making materials today.
- The first question describes discovery: how do we go from an arrangement of atoms (using one or more elements) to predicting its properties (how hard would it be, would it be transparent, would it conduct electricity, etc.)
- The second question describes synthesis: how do we go from a desired atomic arrangement, to figuring out recipe (what precursors to use, how to mix them, under what conditions) that would make the desired material.
- The last question describes characterization: how do we take what we cooked up from the synthesis step and analyze its properties: atomic arrangement, and other physical properties that one cares about.
Now that we have a sense of why a material behaves the way it does, and how we think about making materials today, let’s get back to Schrödinger.
Now that we have a sense of how we think about making materials today, let’s spend a few minutes to understand the role Schrödinger’s equation plays in first step of making materials: discovery.
3. What is the deal with Schrödinger’s equation?
Before Schrödinger gave us his now-famous equation, chemists and physicists had a patchwork of disconnected theories and tools[^prequantum] to understand and analyze the atomic world. We knew that “atoms” exist, but we didn’t have a single, unified theory that explained what goes on in the atomic (dare I say, quantum) world and why.
In most simple terms, Schrödinger’s equation (quantum mechanics more broadly) gives us a single theory that helps us to “bridge” the gap between the world of atoms and the material properties (hard v.s. soft, transparent v.s. opaque, etc.) we observe when a bunch of atoms are arranged together to form a material that we can hold in our hands.
While this equation helped lay the foundation to how we approach materials discovery, unfortunately it is quite useless in practice (for our purposes), because it’s a very hard equation to solve — impossible to some extent.
Let’s understand why we cannot solve it, because that will lead us to understand how we overcame this hurdle.
Now, let’s try to understand how this “bridge” works conceptually.
We start with a single atom, let’s say a carbon atom.
[visual of a neutral carbon atom — positively charged nucleus with 6 protons and 6 electrons around it — we can also mention how heavy is the nucleus v.s. an individual electron]
As you can see, a carbon atom is made up of a positively-charged nucleus at the center, and negatively-charged electrons moving around it. You might remember from your school that opposite charges attract and similar charges repel each other — hence all the electrons will be attracted to the nucleus and each and every electron will repel each other — making an atom itself a balancing act of forces keeping it together[^atom-model]. Another thing to note is that the nucleus in this case is 22,000 times heavier than a single electron. Because the nucleus is so much heavier, it moves a lot more slowly than an electron — hence from an electron’s standpoint, the nucleus basically pinned in place. So, for the rest of this discussion, imagine the nuclei are frozen where they are[^bo].
Now, in order to create a material that we can hold in our hands, we need to assemble more and more such atoms together, and then as we do that, we will be able to see the properties of the material emerging from the way the atoms are arranged and how they interact with each other as a group.
[visual — prototype built: `/embed/sweet-spot` ("two atoms" tab) — drag the second atom; force arrows flip from pull to shove, and the reader unknowingly traces the energy-vs-distance valley — same embed as above; one placement covers both tabs]
So, let’s bring second atom towards the first one and watch what happens. Each of these atoms have their own little balancing act — but now these two will start tugging on each other: each atom's electrons will repel the other's electrons, the two nuclei will repel each other, and each atom's electrons will be pulled towards the other atom's nucleus. All these attractions and repulsions will settle out when these two atoms are at a sweet-spot distance from each other, where all the forces balance out (and because the nuclei will hold still, the only thing that actually moves around are the electrons) and the atoms feel most settled.
Notice that even 2 atoms have created a surprisingly tangled situation — and it only gets worse as each new atom brings more electrons. If we keep going like this, adding a third, a fourth, a billion atoms, the same story plays out at each step — just with ever more complicated dance of push and pulls, all somehow settling into a delicate balance.
Now, let’s step back for a second and ask ourselves, if we wanted to calculate properties of this material (how hard is it, whether it conducts electricity, how transparent is it, etc.) what would we actually have to work out here?
It’s becoming clear that we would have to figure out how this whole tangled mess of nuclei and electrons arranges itself and holds itself together. And if you haven’t realized this yet, we really can’t figure this arrangement out one atom at a time and add the pieces up, because every single electron feels all the other electrons and all the nuclei, all at once. So, the only way for us to solve it is as a shared, tangled system as a single whole.
[visual — prototype built: `/embed/coupling-web` — slider adds atoms; every electron gets a shimmering link to every other electron and nucleus, and the connection count explodes]
This, my friends, is where the Schrödinger's equation earns its place. We give it:
- where the nuclei sit (if that one makes you raise an eyebrow — don't we need the equation to tell us that? — good, hold the thought, we'll come back to it in a minute),
- which element each nucleus is (all nuclei are carbon in our case)
- total number of electrons shared across the whole system — though you don't really hand this one over, since in this case we know each carbon atom has 6 electrons hence total electrons are 6 x number of atoms in our system
and the job of the equation is to encapsulate all these details above (it internalizes all the forces on its own) — along with the deeper quantum rules I have been quietly glossing over (electrons are a lot stranger than the cute little balls I keep drawing[^electrons]) — and works out (for now, let’s ignore how we actually solve the equation):
- how exactly have the electrons settled into place, and
- puts a number on how “settled” the arrangement is.
The first point is what we have been expecting the equation to give us, but what is this second thing?
Recall that when we were picturing 2 atoms coming together and finding a sweet-spot where all the pushes and pulls balance out and the atoms feel most settled. This configuration of atoms is where the system has the lowest energy. So, the second point gives us the energy value of the whole configuration. The lower this number is the more settled or stable a system becomes.
This energy value is a very useful thing to know because it tells us which materials (a particular configuration of atoms) is more stable that the others — like you could ask, which one is more stable? diamond or graphite? — whichever of these has a lower energy will be more stable and would be preferred more by mother nature — we’ll come back to this question soon.
Now, one last gap we need to address is the first input I mentioned — where the nuclei sit — isn’t the point of the equation itself to help us figure out where is the sweet spot for each nuclei that we have been talking about? How am I supposed to know where to put the nuclei before I have even run the equation?
That is a very good question. The trick is that we just don’t know where we should put the nuclei at the start. So, we just guess and just put the atoms where we think they might settle down — and good thing is that we do not need our guess to be correct — because we have a way to figure that out eventually.
Once you have put in your guesstimates for nuclei positions, we solve the equation, and it tells us how the electrons settle around them, and how settled this arrangement is (the energy value).
Now, remember that I told you that the more settled the arrangement becomes, the lower the energy is for the arrangement — so we can use the energy as some kind of score — and the game is to find out the arrangement that minimizes this score!
Here is one way one can visualize this game.
[visual: a hilly valley with a single minimum where altitude represents the score — the configuration is represented by a man walking through this hilly terrain — prototype built: `/embed/landscape-game` — click to drop a guess, watch it walk downhill; "many valleys" mode is ready for the later hull/metastability section]
Imagine a man on a hilly terrain. The man’s position across the terrain is like the guess of the position of nuclei we put in the equation and the elevation at which the man stands is the energy. The goal here is to help this man so he can navigate his way to the point of lowest elevation, which in our case would be the nuclei positions where the system feels most settled.
(One simplification worth flagging: In order to keep things simple for now, we are using an example with a single valley. Real materials have landscapes have many valleys. For example, diamond and graphite are two different valleys open to the very same carbon atoms. Figuring out which valley is deepest, and how a material can get stuck in a shallower one, is a richer story we'll come back to later. For now, let’s continue with a single valley.)
The most inefficient (but easy to understand) way would be to ask the man to move a few steps in each direction and check which direction reduces the elevation (that's the way we should go).
In the same way, once we have the energy results from our first run of the equation, we can try changing the nuclei positions little bit to see which changes lead to a lower energy value (note that there are more efficient/intelligent ways to calculate which nuclei position changes will lead to lower energy, but we don’t need that to understand the basics).
This way, step by step, we can iteratively run this process until we reach a point where any changes in positions of nuclei lead to a higher energy value — which would mean that we have reached the most settled or stable configuration for the whole system — and hence the most stable material.
So, to recap:
- We guess some nuclei positions (and give other inputs), and feed that to the equation, which gives us how the electrons will arrange themselves around these nuclei sitting in particular positions we assumed, and a score (energy) which tells us how settled the whole system of nuclei and electrons is.
- We try some changes in nuclei positions (in practice, we have a better way to figure this out) and see which changes lead to a lower energy score, and keep updating the nuclei positions to the ones that lead to a lower energy score than before.
- Eventually, we will end up with nuclei positions where any change to the positions gives us a higher energy score — which means we have reached the lowest energy score possible.
This is a very simple explanation, which while hides a lot of caveats, gives you a general idea of the role of Schrödinger’s equation. We still haven’t covered how proceed from here to actually get the material properties that we are after, but we’ll get to that soon. For now, I need to show you a dirty little secret I have been hiding from you…
But notice one thing about this whole game: to pin down where even a single material wants to settle, we didn't run the equation once — we ran it over and over, scoring arrangement after arrangement. Hold onto that, because here's the twist we've been quietly dancing around: actually solving Schrödinger's equation — even a single time, for anything bigger than the tiniest handful of particles — turns out to be so hard that it's effectively impossible. That wall is where our story really begins, and getting around it is the reason modern materials discovery looks the way it does.
So far I've been talking as if "solve the equation" is a thing you just... do. Let's poke at that, because it's where the whole story turns sour.
For a moment, let’s forget the whole game of guessing the nuclei positions and finding the lowest-energy arrangement. Let’s ask a much smaller question: For a fixed arrangement of nuclei, what would it take to solve the equation just once?
Remember what the equation is supposed to hand back to us: “how the electrons are settled around the nuclei” (we’ll ignore the energy value part for now). In other words, a map of where the electrons are. So, let’s try to actually draw this map ourselves and see how computationally heavy is this.
Let’s start with the simplest possible case, a single nucleus with a single electron.
And right here, I need to confess something I have been quietly hiding from you while drawing all those little electrons as small balls: we just cannot pin-point where an electron is.
Let me be precise about what I mean by that.
It is not that the electron looks blurry to our instruments. When we actually try to measure where an electron is, we can get a definite answer where it is. What nature refuses is telling us that answer in advance. No amount of information, no cleverness, no better theory lets us predict where the next measurement will find it. The best possible forecast, even in principle, is a list of odds (“it’s more likely to be in certain regions than others”[^electrons]). The fuzziness lies in the prediction, not in the observation[^measure].
Now, I know “fuzziness lies in the prediction, not in the observation” sounds a bit confusing, so instead of explaining it, let me show it to you. We’ll build a tiny hypothetical world, run an experiment few thousand times, and see how both sharpness of the observation and fuzziness of the prediction shows up on its own. Bear me for a while, as we set up the stage for this little world, as it is going to carry the whole section ahead.
Just to be clear, while what we are about to build is a completely made-up experiment, but the results that we are about to study, are as true as they can be. The made-up experiments are just meant to make things easier to visualize.
To keep things simple, we’ll do 2 things:
- First, let’s imagine that our nucleus and electron live in a flat 2D region (real space is 3D, but 2D is a lot easier to draw and understand, and nothing about the argument ahead will be affected by that). In the middle, we have our nucleus and around it our one electron.
- Second, in order to catch our electron, we will add a grid of electron detectors[^detector] to this 2D space — 4 detectors along the length and 4 detectors across the width, hence a complete grid of 16 detectors. The plan is to flip all 16 detectors on at once, and see which one catches our one electron. Each “run” of this experiment (single detector catch) resets the nucleus and electron to their original state.
Try running the experiment a few times yourself. Every time, you can see that exactly one detector would click. Note that we could have had a million or even billion detectors (as much resolution as you want) instead of just 16 and it would still just click a single detector. This basically tries to show you that when it comes to observation, there is nothing fuzzy about it. Detection is always as accurate and precise as we care to make it.
One thing you might have noticed while playing with it is that every time you run the experiment, it gets caught by some random detector — #7, then #10, then #7 again, then #11… #6, #7, #2, etc. You can run it as many times as you want, you won’t be able to reliably tell which detector will click the next time. It’s not because our instruments are insufficient or our theories are lacking — but strangely, this is how electrons behave — they really just don’t have a precise position until we measure it. Yes, you read it right, they only “collapse” to a precise position only after the act of measuring. This rightly feels unintuitive because we are used to “classical” things which have a precise measurements like position, doesn’t matter if we try to measure it or not. But things at atomic scale just do not follow same logic.
While we cannot predict the which detector will catch the electron next, there is a deeper pattern to be revealed if we run the experiment a lot of times and see how likely is the electron to be captured by different detectors, we’ll start to see probability distribution emerging. This probability distribution isn’t random, it is unique (and reproducible) to the setup — a nucleus with a single electron.
And this shows something important: this probability distribution is what the Schrödinger equation computes — this is the map we have been referring to when we said “how the electron(s) is settled around the nuclei”.
Pause for a second and look back at the “fuzziness lies in the prediction, not in the observation” statement. Ever single observation, which is the detector catching the electron, is precise and definite. But the prediction, which if predicting where you will find the electron, is not precise (fuzzy), it has a probability distribution.
Now, I know how unsettlingly unintuitive it feels, but you are in great company because it deeply bothered the very physicists who theorized it as well. Even Einstein (he didn’t come up with this) was so deeply offended by this idea that he famously said:
God does not play dice with the universe[^dice]
He really believed the uncertainty in predicting things like the position of an electron merely points to the fact that the (quantum mechanics) theory is in itself incomplete, and that we should be able to get rid of this uncertainty with a more complete theory. In short, even Einstein called “skill-issue” on this whole idea of “we cannot pin-point position of things like electron in advance” which is exactly our instinct as well.
But even almost a century later, we have enough experimental evidence[^evidence] that backs this idea up, and this is the best model we have got to explain things at the atomic scale, so we’ll just have to roll with it[^before].
Ok, now let's get back to the question we actually came here for: how heavy is it, computationally, for the equation to hand us this map? As we saw above, for our single electron in its 16-box world, we just saw the full answer: 16 numbers. So far, so easy.
Now, let's add a second electron to the same 2D region. Our first thought would be to do the same thing we did for the first electron: run the experiment a few thousand times, tally up this electron's clicks too, and write down its own 16 numbers. Both electrons would have their own 16 numbers, hence 16 + 16 = 32 numbers, and the map is done. That's all, right?
Actually, no. This is where we need to remind ourselves of a fact we encountered earlier: each electron feels the repulsive force from every other electron. And that tangles the two electrons' odds together in a way that two separate lists cannot capture. Let me show you what I mean.
First, what does a run of our experiment even look like now? Same drill as before: one nucleus, two electrons this time, flip all 16 detectors on. And this time we get two clicks — one per electron.[^look] So each run hands us a pair of boxes. Maybe (#1 and #12). Maybe (#6 and #16). Rerun it ten thousand times, like before, and we end up with a big pile of pairs.
Now let me ask a question about that pile. Take just the runs where one of the two clicks came from box #1, the top-left corner. In those runs — where did the other click land? Notice this is a sorting job, not a new experiment: go through the records, keep every run that has a #1 click, throw the rest back. Try it below — the runs that survive the cut paint their other clicks onto the grid as dots. And look at where the dots pile up: the boxes near #1 hold only a thin scatter, while the dense crowd sits in the middle of the grid — nudged away from #1's side. Which makes sense, physically. The nucleus still pulls the other electron toward the center, same as before. But now there's a second force in the room: electrons repel each other. So a run that put one electron in the corner had usually pushed the other one out of that neighborhood — without ever letting it stray far from the nucleus.
Fine — then let's tally up those "other" clicks, exactly like we tallied clicks before: count them box by box, divide by the runs that survived the cut. Out comes a familiar-looking thing: a list of 16 odds for the other electron. Press the tally button below and you'll have it. Job done? Not quite. Because this list comes with fine print attached — you can see it pinned right under the map: these odds were tallied from only the runs where box #1 clicked.
So let's ask the same question about a different slice of the pile: the runs where one click came from box #11, one of the middle boxes. Same pile, same sorting job — just a different cut. Run it below and watch the same grid re-sort and re-tally itself. This time it's box #11's patch that empties out — and the crowd piles up on the opposite side of the nucleus, so boxes that were middling a moment ago now carry the highest odds. Compare each new number against its little gray "was" — this isn't one or two numbers getting nudged. It's the whole list, reshuffled.
Notice what just happened. Nothing about the setup changed — same nucleus, same two electrons, same pile of runs. We just asked about two different slices of the pile, and got two completely different lists for the second electron. That is what the repulsion did to our bookkeeping: there is no such thing as "the second electron's list." There's only "its list, given where the first one turned up" — and that's a different list for every box. 16 + 16 was never going to survive this.
So if we want the full, honest answer, we can't stop at two slices. We have to cover every box the first click might come from. One 16-number list for "the other electron, in runs where box #1 clicked." Another 16 for box #2. Another for box #3… all the way to box #16. That's 16 lists of 16 numbers each: 16 × 16 = 256 numbers, just to honestly describe two electrons in our tiny toy world.
And there's a cleaner way to see what we've actually built. Each of those 256 numbers answers one very concrete question: "how likely is a run to end with one click in box A and the other click in box B?" Sixteen choices for A, sixteen for B — 256 questions, 256 answers. Not two maps stapled together: one big joint map, where every single entry is about the pair.
Pause on that for a second, because this right here is the whole ballgame: adding one more electron did not add 16 numbers to our pile. It multiplied the pile by 16.
And it doesn't stop there. Bring in a third electron, and every run now ends with three clicks — so the honest map needs a number for every possible trio of boxes: 16 × 16 × 16 = 4,096. A fourth electron: 65,536. Every new electron multiplies the pile by another 16.
Now it's time to pay back the two simplifications I borrowed when we built the toy world, because the real world is meaner on both counts. Real space is 3D, not flat — and 16 boxes is a comically crude pixelation. So let's redo the count with a still-crude-but-3D grid: 10 boxes along the length, 10 along the width, 10 along the height. That's 10 × 10 × 10 = 1,000 boxes. One electron: 1,000 numbers. Honestly? Still nothing — your phone wouldn't even notice. But two electrons: a number for every pair of boxes, 1,000 × 1,000 = a million. Three electrons: a billion. Four: a trillion. Every new electron now multiplies the pile by a thousand.
Now, let's bring this back to our carbon. A single carbon atom has six electrons. Six electrons means multiplying by a thousand six times over — 1,000⁶ — which works out to a billion billion numbers. And let me remind you: that is for one atom, on our comically crude grid.
A real chunk of material has billions upon billions of atoms, but let's not even go there. Let's just line up five carbon atoms. That's 30 electrons, so we multiply by a thousand thirty times over — a 1 followed by 90 zeros. For comparison: the total number of atoms in the entire observable universe — every atom of every planet of every star of every galaxy we can see — is estimated to be a 1 followed by roughly 80 zeros. So even if we could somehow use every atom in existence as a slot to store one of our numbers, we would run out of universe before we ran out of numbers — ten billion times over. For five atoms of carbon.
Sit with that for a second, because it's not the kind of "hard" you think it is. This isn't "we need a faster computer" or "wait for better chips." A faster computer is a bigger engine, and this is not a hill you climb with a bigger engine — it's a wall that races away from you faster than you could ever run at it. There is no machine — not today's, not a thousand years from now, not one built out of every atom in the cosmos — that can store the exact answer for more than a handful of particles. The equation that flawlessly describes all of matter is, in its exact form, permanently unsolvable[^grid].
Which drops us somewhere genuinely absurd: we're holding the master key to all of matter, and it doesn't fit any lock we could ever build. So how does any of this — the AI models dreaming up millions of new materials, the robot labs cooking them up overnight, everything I bragged about at the top of this essay — exist at all? Because someone found a way to cheat. And that cheat is where materials discovery, as we actually practice it, begins.
4. How DFT works where Schrödinger’s equation couldn’t?
We left off at a point where we had an equation that, in principal, can describe any material (its energy, and other material properties), but we were blocked by a computational wall.
So, it is obvious to ask the question: what did scientists around that time (when the Schrödinger’s equation was established) do? Did scientists come up with any tricks to try and use the equation?
Dirac said in 1929: the underlying laws are now completely known, he wrote, and the difficulty is "only" that applying them exactly leads to equations too complicated to be soluble — so what's needed are approximate practical methods.
So, for about 40 years or so, the field came with different ways to twist the problem (equation) in ways that makes it practical to solve. We can put the following in the footnotes:
- Hartree's averaged crowd (1928): This is quite similar to what kohn/sham tried to do 37 years later, but with one key difference: Hartree assumed than an electron is moving through a cloud of (negative) charge made up of all electrons except the one electron that is supposed to made through the cloud, whereas in case of kohn/sham, they assumed the cloud of charge is made of up all the electrons, even including the electron that is supposed to move through the cloud.
- we start with a guess of electron density. One good guess is to just put electron density for individual atoms wherever they are placed and then summing up the electron density.
- most packages have the single atom electron density for the whole periodic table — even that is done using DFT — single atom DFT is easy because the atom is symmetric
Now, let’s just focus on “Hartree's averaged crowd”. Here the “cloud” is the electron density. What’s that?
You remember the map we created for electron clicks based on individual electron interactions? what if we just ignored the click groupings and just summed up the clicks instead? that’s what electron density is. It no longer has the context of where the individual electron was compared to others (and also ignores some other quantum rules like pauli exclusion principal, etc.)
But we know that the map itself is blocked by the computational wall, and hence the electron density as well…but we do not need to know the exact electron density, we can start with a guess.
The core part of this whole thing is a self-consistent loop (remember the loop we discussed in the Schrödinger’s equation early intro).
- In 1964, Pierre Hohenberg and Walter Kohn — proved the following about that exact map: for a system that has settled into its lowest-energy state, the density determines everything.
- the theorem proves the answer is determined by the density. It does not tell you how to compute the answer from the density.
- In 1965, Kohn and Sham, figured out the way lowest-energy state from the density.
2. How imagining a material got almost free
Starts with setting up what user already knows:
For almost all of history, there was exactly one way to find out what a material would do: make it and check. Materials science lived in furnaces. You could have brilliant intuition, but intuition isn’t prediction, and it doesn’t scale.
Maybe we can give some examples of how we as humans have been either "accidentally" been discovering useful materials or pain-stakingly brute-forcing our way through it.
Pose a first principals question to the user -- if we know how the atoms are arranged in a material, can we predict its properties (what color would it be, would it be a conductors or insulator, would it be magnetic, etc.)
Introduce schrodinger (yes, the guy from schrodinger's cat fame) and his eqaution -- explain it in simple words -- if needed add a footnote for it (to clarify any assumptions we have made to keep things simple in our explanation)
Explain the problem we have with schrodinger's equation in simple words.
Introduce DFT as its solution -- explain DFT in simple words -- specifically mention what its assumptions are (question: are these DFT assumptions or are these assumptions inherited from schrodinger's equation itself) like 0 temperature, etc. AND what do we get as a result of DFT (energy -- explain what exactly) and how you can calculate other material properties with this.
Now, make the user think for a second, what we have done just now -- given the arrangement of atoms of a hypothetical material, we can now essentially, predict its properties, without actually making it.
Once “what will this material do (or how will this material behave)?” became a computation instead of an experiment, three things happened in quick succession, and each one made imagining a material cheaper by orders of magnitude.
- people made databases for: (material atomic structure) -> (predicted material properties)
- DFT, while a great computational tool, still scales horribly. So, people made machine learning models, MLIPs. These “machine-learning interatomic potentials” let you simulate millions of atoms jostling at real temperatures (OK, WAIT, CAN WE DO IT FOR REAL PRESSURE AND OTHER THEMODYNAMIC PARAMS AS WELL???)
- while one can build a database for a lot of materials, the potential search space of materials (specifically compounds) is ??? (see if we can get a number). What if we can go from "desired material properties" to "material" itself? MatterGen[^mattergen] does that
Step back and notice that the verb finding a new material has been riding the computing curve for sixty years and is now, for practical purposes, cheap.
But cheap for what, exactly? Here’s a caveat that’s easy to miss and matters for everything that follows: this entire apparatus — DFT, the databases, the generative models — was built for, and quietly assumes, the most computer-friendly slice of matter there is. An inorganic crystal is the easy case: a small, ordered, periodic structure that repeats forever from one tidy “unit cell” you can hand straight to DFT. That’s not most of the material world. It’s the corner of it that happens to fit on the equation. Walk away from that tidy corner and the whole machine degrades, in instructive ways:
- Metallic alloys — especially the fashionable “high-entropy alloys” — are defined by atoms randomly sharing the same lattice sites. That configurational disorder has no clean repeating unit cell, so the tool that needs one starts flailing.
- Molecular crystals — pharmaceutical pills, where the very same drug molecule can pack into different “polymorphs” with wildly different solubilities and shelf lives — are held together by weak forces that standard DFT describes badly, with dozens of near-identical candidate structures separated by energies too small to rank with confidence. (We’ll meet a drug that recrystallized itself off the market in §8.)
- Polymers — plastics, the highest-tonnage synthetic materials on the planet — barely have a “structure” in the crystallographic sense at all: tangled, partly amorphous, floppy chains with no unit cell to speak of. The generative-model revolution has, so far, mostly skipped them.
The pattern that emerges here recurs at every scale in this essay, so it’s worth stating as a near-law: the more ordered and periodic a material is, the more the computer can help; the messier and more real it gets, the more you’re thrown back on the furnace and the craft.15 This is why I’m spending the essay mostly on inorganic crystals — not because they’re the most important materials (your phone’s plastic and your engine’s alloys would beg to differ), but because they’re where our best tools work, which makes them the fairest possible test of how good those tools really are. And it’s a first quiet warning about the villain of the whole piece: disorder. The instant a material stops being a clean repeating pattern, our cheap-imagining superpower starts to fail — a fact that will come back to bite us hard when we try to figure out what a robot actually made.
So, if finding is this cheap, why isn’t the world flooded with new materials? Why did the 380,000 stay on the screen? The answer is that “find” was never the hard part. Let's look at the "hard" part now.
3. The catch: “stable” is not the same as “makeable”
In the last section, remember how DFT helped us to predict energy of a material based on the atomic arrangement under specific assumptions (re-iterate the assumptions).
Now, as we can calculate DFT based energies, simply mention the fact that anything in this universe prefers to move towards a state with as low energy as possible — make it more “stable”. give a simple example for this so they can get an intuitive feeling for it.
Try extending this fact to some materials, if material A has lower predicted energy than material B, then ideally, nature would prefer to make material A instead of material B.
Extend this to concept of DFT convex hull. explain that in simple words — mention how when material scientists say that a material is “stable”, they mean that it is on the DFT hull — callback the fact that when GNoME reports “380,000 stable materials,” stable means “on the hull.”
Now, mention that at the normal conditions (pressure, temperature, etc.) the what are the energies for diamond and graphite, both different arrangement of carbon.
Wait a minute, didn't I say that nature prefers materials with less energy? So, if graphite has less energy than diamond, then why does diamond exist in the first place? Shouldn’t all diamonds, turn into graphite?
This is the "hard" part.
Diamond isn’t a freak exception. It’s closer to the rule. In 2016, Wenhao Sun, Gerbrand Ceder and colleagues took nearly 30,000 inorganic compounds that humans have demonstrably made and asked where they sit relative to the hull. The answer: about half of all the real materials we’ve ever made are not, technically, stable.16 They’re metastable — above the hull, kinetically trapped in a state the universe would, given the chance and a low enough barrier, abandon. The median real material sits about 15 thousandths of an electron-volt per atom above the line; some whole families (nitrides) routinely persist far higher and never come down.
Next, give examples of cases where more stable materials (close or even at the hull) are either a nightmare to synthesize or just not something we have been able to synthesize at all.
So, if we see materials are are not on-the-hull occur in nature and have hard time making materials that are on-the-hull, then what is even the meaning of calling them “stable”?
So the convex hull — the oracle, the filter, the definition of “real/stable” — is answering a subtly but crucially different question from the one we care about. It tells us what’s stable. We want to know what’s makeable. These come apart in both directions: many stable compounds are a nightmare to synthesize, and many of the materials we treasure aren’t stable at all.
Which forces the real question: if thermodynamics — the bottom of the landscape — doesn’t decide what we can make, what does?
The answer is kinetics: the paths, not the destinations. And here we hit the single most important sentence in this field, written plainly by Ceder’s own group: “there is no a fundamental theory for materials synthesis.”
Sit with that. We have a fundamental theory for what materials are — quantum mechanics, via DFT. We have no fundamental theory for how to make them. That asymmetry is reason number one that we still suck: we automated the half of the problem that had a theory, and the other half never did.
The next two sections are about why the making half is so resistant. The first reason is physical. The second is mathematical. They’re the same gap seen from two sides.
4. Reason one: the materials we love live on the slopes, and we only mapped the valleys
idk, maybe we can take a step back here, and understand if making
Introduce metastability
[^aidrug]: The flagship example is rentosertib (formerly ISM001-055), a TNIK inhibitor for idiopathic pulmonary fibrosis whose target was identified and whose molecule was designed using Insilico Medicine's generative-AI platform; its randomized Phase IIa results were published in Nature Medicine (Jun 2025), the first clinical proof-of-concept for an AI-discovered-and-designed drug. Veselkov/Ren et al., Nat. Med. 31, 2602–2610 (2025), https://doi.org/10.1038/s41591-025-03743-2. The first AI-designed molecule to reach human trials was Exscientia/Sumitomo's DSP-1181 (Phase I, 2020), though it was later discontinued for insufficient efficacy. Caveat worth keeping honest: these are AI-assisted pipelines, not "type a disease, get a drug" — the human chemists and biologists are still very much in the loop.
[^alphafold]: AlphaFold is a breakthrough artificial intelligence system developed by Google DeepMind that accurately predicts a protein's 3D structure from its amino acid sequence. By replacing decades of tedious physical experiments, it has revolutionized molecular biology and drug discovery, earning its creators the 2024 Nobel Prize in Chemistry.
[^gnome_mattergen]: AI tool GNoME finds 2.2 million new crystals, including 380,000 stable materials that could power future technologies. MatterGen is a generative model for inorganic materials design across the periodic table that can be fine-tuned to steer the generation towards a wide range of property constraints.
[^a_lab]: A-Lab, an autonomous laboratory for the solid-state synthesis of inorganic powders.
[^philosopher]: The philosopher's stone is a mythic alchemical substance capable of turning base metals such as lead and mercury into gold or silver.
[^hennig]: Well, in his defense he specifically used urine because its golden color led him to believe it might contain or help synthesize actual gold.
[^schrodinger]: I am grossly simplifying the narrative here by singling out Schrödinger here. There were many significant contributions that accompanied Schrödinger work that lead to us being able to understand how to study atoms in theory and in practice.
[^prequantum]: A quick tour of that patchwork, roughly in order of arrival. Atomic theory (John Dalton, 1808): matter is made of atoms that combine in fixed whole-number ratios — but the atom itself was a featureless ball. The periodic table (Dmitri Mendeleev, 1869): startlingly predictive by pattern — Mendeleev left gaps and correctly forecast gallium, scandium, and germanium from the trends — yet with no explanation of why the trends existed. Bonding rules: chemists knew carbon forms four bonds and oxygen two, and G. N. Lewis's shared-electron-pair model (1916) captured how atoms stick together — but purely as a heuristic; nobody could say why electrons would pair up. Thermodynamics (J. W. Gibbs, 1870s): the phase rule and phase diagrams let metallurgists predict which phases are stable from measured data (the iron–carbon diagram is how we understand steel), with zero atomic-level theory — and it's still indispensable today. X-ray crystallography (Max von Laue, 1912; W. H. and W. L. Bragg, 1913, Nobel 1915): for the first time we could see the arrangement of atoms in a crystal — W. L. Bragg solved diamond's structure in 1913, and graphite's layered structure followed (J. D. Bernal, 1924), so by the mid-1920s we could literally photograph the fact that diamond and graphite are built differently, without being able to explain why that made one hard and one soft. And the atom models themselves were in flux: J. J. Thomson found the electron (1897), Rutherford the nucleus (1911), and Bohr's quantized-orbit model (1913) nailed hydrogen's spectrum but broke down for any atom with more than one electron — and for chemical bonding. Each tool illuminated a slice; none derived a material's behavior from its atoms. That unification is what quantum mechanics (Heisenberg, 1925; Schrödinger, 1926) finally provided.
[^atom-model]: This "electrons buzzing around the nucleus like tiny planets, settling where the forces balance" picture is the classical cartoon (roughly the 1913 Bohr model), and it isn't quite how things really work. Electrons don't trace definite orbits or sit at fixed points; quantum mechanics describes each one as a fuzzy probability cloud — a region where the electron is more or less likely to be found — and they can only occupy certain discrete energy levels. What actually fixes the arrangement is the system reaching its lowest-energy state under the quantum rules, not a literal tug-of-war of forces. But the force-balance intuition gets the spirit right, and it's all we need here.
[^electrons]: The quantum rules I'm waving past, in case you're curious. Two matter most. (1) Wave–particle duality: an electron isn't a little ball with a definite location — it behaves like a spread-out wave, and you can only speak of the probability of finding it somewhere. (2) The Pauli exclusion principle: no two electrons can be in the exact same state at once, so instead of all piling into the lowest-energy spot, they stack up into successive shells — which is why the periodic table has the shape it does, and even why matter takes up space at all. Electrons also carry a property called "spin." These aren't optional flavor; they're part of the full quantum rulebook the equation is solved alongside, and leaving them out would give wrong answers. We skip them only because the simple force-balance story is enough to follow the argument.
[^measure]: "A definite answer" deserves two honest asterisks. First, a real position measurement is definite only up to the instrument's resolution — but modern instruments resolve positions far finer than the chunky grid of boxes we're about to draw, so for our purposes the answer really is "right there, full stop." Second, measuring an electron is not a gentle peek: the act of pinning it down disturbs it, so its state right after the look is not what it was just before. Each individual look gives a sharp answer, but you can't chain looks together into a smooth tracked path the way you'd film a planet crossing the sky. And if you've heard of Heisenberg's uncertainty principle and are wondering whether it forbids all this: it's a related but different constraint — it limits how sharply position and momentum (roughly, speed and direction) can be pinned down at the same time. It does not forbid a single sharp position reading.
[^dice]: A fun fact about this famous line: what Einstein actually wrote — in a 1926 letter to his friend and fellow physicist Max Born — was closer to "The theory says a lot, but does not really bring us any closer to the secret of the old one. I, at any rate, am convinced that He does not play dice." The snappier "God does not play dice with the universe" is the version that stuck after a century of retelling. And there's a delicious irony worth savoring here: Einstein was no outsider grumbling at physics he didn't understand — he helped found quantum theory (his Nobel Prize was for proposing that light itself comes in discrete quanta, one of the theory's cornerstones). His quarrel wasn't with the theory's success; it was with its claim that the randomness is fundamental rather than a gap in our knowledge.
[^evidence]: "Trust me, experiments back this up" deserves better, so here's what that evidence actually looks like. Einstein's alternative has a name — hidden variables: the idea that the electron really does have a definite position at all times, and the odds merely reflect our ignorance of it (like a shuffled deck — the top card is already fixed, you just haven't seen it). For decades this looked like untestable philosophy, because both stories predicted identical results for every experiment anyone could devise. Then in 1964, physicist John Bell found the crack: he proved that if measurement outcomes are secretly decided in advance, then the statistics of certain paired measurements on two far-apart particles must obey a strict numerical bound — while quantum mechanics predicts that bound gets broken. Philosophy suddenly became an experiment. It was run and refined over the following decades (the loophole-free versions landed in 2015, and the 2022 Nobel Prize in Physics went to Aspect, Clauser, and Zeilinger for this line of work), and nature broke Bell's bound exactly as quantum mechanics predicted. So "the answer was written down all along, we just couldn't see it" isn't merely out of fashion — it is experimentally ruled out (the one surviving loophole requires the hidden answers to coordinate faster than light, which most physicists consider a worse deal than the dice). The randomness isn't in our instruments; it's in the world. And if you'd like to see this weirdness with your own eyes: watch the single-electron double-slit experiment filmed at Hitachi in 1989 — electrons fired one at a time, each arriving at the screen as a single definite dot, and the dots slowly assembling into exactly the pattern the equation computes: https://www.youtube.com/watch?v=jvO0P5-SMxk
[^detector]: This isn't a made-up gadget — real position measurements genuinely are grids of tiny detectors. The screen in the double-slit footage linked a couple of footnotes from here is exactly that, and so is the camera sensor in your phone (for light instead of electrons). Two honest asterisks on our imaginary version, though. First, ours are perfect: real detectors occasionally miss, occasionally misfire, and catching an electron is not a gentle act (see the earlier footnote on measurement). Second — and this one matters — "the detectors are off" does not mean "the electron is quietly sitting in one of the boxes and we just can't see which." That tempting picture turns out to be experimentally ruled out, as we'll see in a moment.
[^before]: For readers who can't quite roll with it yet and are still asking "ok, but what is the electron actually doing when nobody is looking?" — fair warning, this is where even physicists start arguing with each other. But here's the careful version. It's tempting to hear "nature won't tell us the answer in advance" and conclude that the answer exists and is merely being kept from us — but that's exactly the hidden-answer picture the Bell experiments ruled out. The honest statement is stranger: before a look, the electron doesn't have a definite position at all. The question "where is it right now?" has no answer — not a hidden one, no answer — a bit like asking which face a still-tumbling die is showing. As far as anyone can tell, the list of odds isn't a summary of our ignorance about some deeper fact; it is the complete description of the electron. One more piece, because you'll hit it in any further reading: when a look finds the electron at a particular spot, the list of odds snaps — look again immediately and you'll find it in the same place (the odds are now 100% right there), and only afterwards does the spread take over again. That snap is the famous collapse of the wavefunction. What the snap really is — a physical process? a bookkeeping update? evidence of something deeper? — is a genuinely unsettled, century-old debate (search "interpretations of quantum mechanics" if you'd like to lose a weekend). Mercifully, none of that matters here: every interpretation agrees on the odds, and the odds are all this essay needs.
[^bo]: This "freeze the nuclei, solve the electrons" split has a name — the Born–Oppenheimer approximation — and it works precisely because nuclei are thousands of times heavier than electrons, so on the timescale the electrons care about, the nuclei are effectively standing still. Solving the equation across many frozen arrangements traces out what's called a potential energy surface — a landscape of energy-vs-arrangement — and finding the stable structure means finding the low point in that landscape.
[^grid]: This grid-counting picture is an illustration, not how the calculation is literally done — nobody stores the answer as a giant table of boxes, and real methods are far cleverer about it. But the punchline is exactly right: the amount of information in the exact solution grows exponentially with the number of electrons (the "curse of dimensionality"), and that exponential is what no amount of computing power can ever outrun. Two fine-print notes for the initiated. First, the fuzziness alone isn't the culprit: if each electron kept its own private cloud, the bookkeeping would stay additive (a thousand numbers per electron). The explosion comes from the clouds being welded into one joint object — hold that thought for when we get to DFT. Second, if you're thinking "statisticians never store giant joint distributions, they sample them" — good instinct, but it's blocked here: the quantum entries aren't ordinary probabilities, they're signed quantities that cancel each other out (interference), and random sampling of a signed object drowns the answer in noise. Physicists call this the "fermion sign problem," and it's why the sampling escape hatch that works for weather models slams shut for electrons.
