It was 2023, and Sarah Porteboeuf’s fridge calendar had "heavy-ion run" scribbled in big red letters across the end of September and most of October.

“My family knew that this period was the most intense of the year,” Porteboeuf says.

Porteboeuf is a physicist on the ALICE Experiment, one of the four large experiments that collects data from the Large Hadron Collider at CERN. Typically, when scientists wrap up the LHC’s usual run of proton-proton collisions, they fill the LHC with lead nuclei and do a few weeks of heavy-ion physics.

Unlike the other big experiments, ALICE is designed to study heavy-ion collisions. “The ALICE acronym is A Large Ion Collider Experiment,” Porteboeuf says. “That’s what we do.”

ALICE scientists study quark-gluon plasma—a soup of fundamental particles that is the most perfect fluid in the universe, which the LHC produces most effectively during its heavy-ion runs.

That fall, Porteboeuf was the ALICE Run Coordinator and responsible for making sure that the experiment was collecting as much quality data as possible. Because of a planned period of upgrades between 2019 and 2021, and an unplanned LHC helium leak in 2022, ALICE hadn’t collected a new data set from lead-ion collisions since 2018. Porteboeuf was excited to make the most of this new run—especially with a revamped detector. “Our detector is basically brand new,” Porteboeuf says. “We were prepared, and we were ready to celebrate.”

The day of the start-up, Porteboeuf was in the control room, eagerly watching the screens for signals from the first collisions. When the two beams finally crossed, “it was an explosion of joy,” Porteboeuf says.

But immediately, she noticed that something was off. “During the first two minutes of data taking, I saw that our inner tracking system had a problem: There was a hole,” Porteboeuf says.

Normally, the lead-ion collisions look like big, symmetrical fireworks, with thousands of particle tracks emanating from the center. But a portion of the ALICE detector was completely dark.

“We projected that at the full beam intensity, this problem was going to impact one quarter of the full ALICE experiment,” she says. “When you do something that has never been done before, this can happen. That’s part of the game.”

Porteboeuf and her colleagues immediately investigated the source. “It could happen that the detector was not correctly configured,” she says.

She asked the team responsible. “They looked at me, and they said, ‘No, we checked already. It’s not that. We think it’s the beam.’”

This was not good news. The LHC is a 27-kilometer ring that accelerates particles to close to the speed of light. The theory was that somewhere along those 27 kilometers, something was behaving strangely and causing a small portion of the beam to go off-course. And somehow, these rogue particles were shooting the ALICE detector straight through its tracker, which was oversaturating its chips and causing the blackout.

ALICE scientists had only about five weeks to collect data from lead-lead collisions. If they couldn’t quickly identify the source of the problem, they would lose yet another year of data; data that thousands of physicists in the collaboration were relying on to do their research. Porteboeuf says she remembers thinking, “Not on my watch.”

“We worked like hell for 10 years to upgrade our experiment,” she says, “but we won’t be able to do the physics we want if we don’t do something.”

The first step was clear—they needed to put their heads together with the operators and accelerator physicists in the CERN Control Center and come up with a plan.

Bruce and the machine

The CERN Control Center had been abuzz as physicists, engineers and operators ramped up the LHC for the lead-ion run, Roderik Bruce remembers.

“We were using a lot of different upgrades for the first time,” he says. “It was a mix of excitement and also a bit of stress to get everything up and running.”

After a few hiccups, they managed to get the beam circulating smoothly. But the next day was a different story.

“We have a meeting every morning at nine o’clock to discuss the status,” Bruce says. “What are the problems? What are the plans for the next day? How are we doing with respect to the schedule?”

The other LHC experiments—CMS, ATLAS and LHCb—all reported smooth data collection. Not ALICE.

“They could not take data, and it was very serious,” Bruce says. “So then on the machine side, we had to react extremely quickly, and we had to think very quickly, because the ion run is only a few weeks.”

One of the hypotheses was that stray particles were hitting the protection system in front of ALICE.

The LHC and its experiments are protected from stray particles by devices called collimators, which are dense mechanical “jaws” on either side of the beam. The beam passes through the small gap between the jaws, and any particles that wander too far off course are absorbed. The collimators in front of the experiments are particularly important because they trim away stray particles that might otherwise hit the magnets in the final focus system or sensitive detector components and create unwanted background signals.

However, collimators can only absorb so much, and secondary particles can still leak out and reach the experiment. If enough stray particles were hitting the collimator in front of ALICE, then the collimator could be leaking particles into the detector. Their theory was that these particle showers from the collimator were oversaturating the ALICE tracker and causing it to go dark.

To test this hypothesis, the LHC operators sent around a low-intensity beam with the ALICE collimator completely open, no longer blocking anything. “The background almost disappeared,” Bruce says.

They had confirmed that the problem was indeed coming from the LHC beam. The next question was why it was sending so many particles into the collimator: “There were two prime suspects,” Bruce says.

It could have been that the collisions at the other experiments were creating particles that were tagging along with the main lead-ion beam but ultimately going off course and impacting the ALICE collimator. Or it could have been that the LHC itself was creating these mystery imitation-beam particles.

Race to the bottom

University of Tennessee graduate student Ewa Glimos landed in Geneva in September 2023. She was in town to take shifts monitoring and operating the ALICE detector during the lead-ion run. “It was very exciting,” she says.

Her advisor, Friederike Bock from Oak Ridge National Laboratory, picked her up at the airport and immediately told her about the mysterious background particles. “At that point I knew that the shifts wouldn’t go smoothly,” Glimos says.

According to Porteboeuf, ALICE physicists, accelerator scientists and operators in the CERN Control Center ran a series of 56 tests over the course of 10 days to home in on the problem.

“The LHC runs 24 hours a day, and they were interweaving the tests with its normal operation,” Porteboeuf says. "The background was coming from the beam, but they could not see what was happening inside our detector. So sometimes we’d be waking up and rushing to the control room at 4 a.m. to do the tests with LHC experts, working together."

Some of the testing took place during Gilmos’ first shifts in the control room.

“We would run for a bit, stop, and then see if the problem is fixed, over and over again,” Glimos says. “It made me feel a bit uneasy. What if it doesn’t work? Maybe it’s a more complicated problem that we can’t solve.”

At the same time as these tests, accelerator scientists were also running simulations to figure out what was happening to the beam to make it go off-course.

The simulations offered a very important clue. The LHC is configured to accelerate lead-208, a specific isotope that has 82 protons and 126 neutrons. But if somehow the beam became contaminated with lead-207 (which has one neutron less) they predicted they could get some strange effects.

“Lead-207 ions have a mass that is half a percent less than the lead-208 in the main beam,” Bruce says. “And because of this half percent lower mass, the ions are over-bent by the magnets.”

The series of over-corrections to the lighter (and thus, more bendable) lead-207 ions could create a separate beam that would oscillate alongside the normal beam.

“By chance, it can pass LHCb and ATLAS without hitting anything,” Bruce says. “We saw very clearly in the simulation that this is possible.”

If the simulations were correct, then Bruce had a pretty good idea of what was creating the contamination: another set of collimators about 10 kilometers upstream of ALICE. According to Bruce, these collimators, at a point on the ring called IR7, have a very special role: They serve as the LHC’s “trash bin.”

“It’s unavoidable that some particles start drifting off course,” Bruce says. “We want to absorb them in a controlled way. Typically, it works well, but it’s not perfect.”

If the collimators were accidentally stripping a single neutron away from some of the lead-208 ions, they might be creating the beam of lead-207.

To test this idea, “we intentionally increased the beam losses at IR7,” Bruce says.

They were right. “We saw immediately that the background went up a lot.”

Back on track

Now that the accelerator scientists understood exactly what was causing the problem, it was time to implement a solution.

First, they considered opening the collimator in front of ALICE to allow the lead-207 ions to pass. But that presented a different problem. “We had to open the collimator so much that it would no longer protect the final focus system,” Bruce says.

Next, they looked into adjusting the settings of the LHC. In addition to dipole magnets (which direct and bend the beam) and quadrupole magnets (which focus the beam) the LHC has a series of smaller magnets to adjust and correct the beam’s position.

“We realized that we could counteract the oscillation by, on purpose, bumping the beam and creating another oscillation which is out of phase,” Bruce says.

The goal was to shift the unwanted lead-207 beam so that it would slide past the ALICE collimator and then collide with other collimators farther down the beampipe, far away from any critical components.

When they tried this, the background signals overwhelming the detector went down by a factor of 100.

For Porteboeuf, saying it was a relief would be an understatement. “I don’t remember exactly how long it took us to solve this problem,” she says, “because for me, it was like a single moment in time.

“That’s the funny thing; when you are inside a problem, it’s all-consuming. But then once the problem is fixed, you’ll realize it was only ten days.”

Even though they missed out on the first 10 days of data-taking, the ALICE collaboration was still able to collect more data in 2023 than all the preceding lead ion runs combined.

“This was an excellent year,” Porteboeuf says. “I never felt like I was alone. Every day, people were coming to me and asking how they could help; even the other experiments were helping us. And this is the power of collaboration.”

Since then, ALICE has more than tripled the amount of data collected during the 2023 ion run and recently wrapped-up their 2025 season of data taking. Porteboeuf says that the new ALICE upgrades are to thank.

“They have opened a new world,” she says. “Now, thanks to the continuous readout we installed as part of the upgrades, we can collect everything at the maximum capability of the machine. This has enabled many new types of analyzes that were not possible before.”

Even though Porteboeuf is no longer run coordinator, she still feels deeply connected to ALICE’s operation.

“We are like a family,” she says. “There is solidarity and we all support each other, no matter the role.”