[Task Proposal #33] Cryo ET reconstruction

[StevenDillmann]

Proposed by @reinhardh

[Task Proposal #33] Cryo ET reconstruction

Scientific Domain

Life Sciences > Biology > Imaging

Scientific Problem

The task addresses the problem of reconstructing a tomogram (volume) from a tilt series (measurements) in cryo-electron tromography. The tomogram is then used to understand aspects of a cellular environment or to reconstruct the structure of proteins. This workflow is something that we run in my lab and that is also run in many other labs that work on cryo-ET.

Workflow Details

You are given a cryo-electron tomography (cryo-ET) tilt series of a 64×64×64 volume with a ±60° tilt range (3° steps) and Gaussian noise (σ = 0.5).

The tilt series consist of a projection stack (41 × 64 × 64, float32) in file `/app/tilt_series.mrc` containing one projection per tilt angle. The 41 tilt angles in degrees are in the file `/app/tilt_angles.txt`, one per line.

Your task is to reconstruct the 3D volume as accurately as possible, overcoming the missing wedge artifact. Use the [DeepDeWedge](https://github.com/MLI-lab/DeepDeWedge) method. Save your result as `/app/reconstruction.mrc` (64×64×64, float32).

The environment is CPU-only (no GPU available).

Your reconstruction is evaluated by comparing to the ground truth using PSNR, but of course the ground truth is not available to you.

Q: Isn't this too easy - to just apply an existing method?
A: It's quite difficult to use this method (and other related ones), since the agent has to make several decisions that require cryo-et reconstruction expertise, and how to do reconstruct. To be able to apply apply DeepDeWedge the agent has to be able to reconstruct a reasonably good volume first, so that DeepDeWedge can refine it.

Q: The instruction should not hint at the solution approach - why specify to use the DeepDeWedge method, and not let the agent use an arbitrary reconstruction method?
A: Because it is a quite common task in cryo-ET to attempt reconstruction with a particular algorithm, and see how well reconstruction works with that method. For some situations some algorithms are more suitable than others. Applying a reconstruction to a given tilt series is however quite non-trivial. Opus 4.6 does for example not succeed as this task.

Dependencies

https://github.com/MLI-lab/DeepDeWedge and a few standard python packages.

Dataset

The environment contains a tilt series that I generated from a crop of a standard tomogram used in the field.

Evaluation Strategy

I check whether reconstruction performance is close to the ground truth from which I generated the tilt series (measurements).

Complexity

Quite difficult, a domain expert not familiar with the method would require an hour to a few hours, to get familiar with the method and to try out a few configurations and evaluate them. Opus 4.6 with Terminus 2 can't solve this task, it's surprisingly challenging for a frontier AI agent.

References & Resources

https://www.nature.com/articles/s41467-024-51438-y
here is the pull request for the task with all the details:
#75

Additional Information

None provided

Author Information

Author: Reinhard Heckel
Email: reinhard.heckel@tum.de
Role: Associate Professor in ECE
Profile: https://scholar.google.com/citations?user=ZWV0I7cAAAAJ&hl=en
GitHub: https://github.com/reinhardh
Discord: reinhardheckel

---

Submitted via Terminal-Bench Science Task Proposal Form

[github-actions[bot]]

📋 Task Proposal Rubric Review

Recommendation: 🟡 Uncertain

Full Review

Scientific Domain

Domain: Life Sciences
Field: Biology / Structural Biology
Subfield: Cryo-Electron Tomography (Cryo-ET) Imaging

Problem Statement

The task asks an AI agent to reconstruct a 3D volume (64×64×64) from a cryo-electron tomography tilt series (41 projections at ±60° with 3° steps, with Gaussian noise). The agent must use the DeepDeWedge method to overcome the missing wedge artifact and produce a reconstruction that achieves high PSNR compared to a hidden ground truth. The agent is given projection data in MRC format and tilt angles in a text file, and must save the reconstruction as an MRC file. The environment is CPU-only.

Verifiable

Positive aspects:

• The output is a concrete numerical array (64×64×64 float32 MRC file) that can be compared against a ground truth volume.
• PSNR is a well-defined, deterministic metric that can be computed programmatically.
• The ground truth is known to the task author (generated synthetically from a known volume), so comparison is straightforward.

Negative aspects:

• The PSNR threshold for "passing" is not explicitly stated in the proposal. There needs to be a clear threshold that separates correct from incorrect solutions. What PSNR constitutes success?
• DeepDeWedge involves neural network training, which on CPU may introduce some non-determinism. However, PSNR thresholds can be set with sufficient margin to handle this.
• A simpler method (e.g., weighted back-projection) might achieve a reasonable PSNR without using DeepDeWedge. The verifier needs to be calibrated so that only DeepDeWedge-quality reconstructions pass, or the task needs to not care about the method used (which contradicts the instructions).

Judgement: Accept — The core verification approach is sound (PSNR against ground truth), though the threshold needs to be carefully calibrated to ensure it's discriminative.

Well-Specified

Positive aspects:

• The inputs are clearly defined: tilt series MRC file, tilt angles text file, and output path.
• The volume dimensions are specified (64×64×64).
• The noise model is stated (Gaussian, σ = 0.5).
• The evaluation metric (PSNR) is stated.

Negative aspects:

• The task specifies "use the DeepDeWedge method" which is a process constraint, not an outcome constraint. This creates a tension: if the agent achieves the same or better PSNR with a different method, should it pass? If the verifier only checks PSNR, then the method constraint is unenforceable (or enforced only via process grading).
• The PSNR threshold is not stated in the proposal. The agent needs to know what "accurate enough" means, or at least the verifier needs a clear cutoff.
• It's unclear whether the agent needs to install DeepDeWedge from source, or if it's pre-installed. The environment setup details matter significantly.

Judgement: Uncertain — The task is mostly well-specified, but the method constraint creates ambiguity about what exactly is being verified. If PSNR alone is the metric, the method constraint is process-verification. If the threshold is set high enough that only DeepDeWedge-class methods can pass, this resolves the issue but needs to be demonstrated.

Solvable

Positive aspects:

• The task author states they run this workflow in their lab, so it is clearly solvable by a domain expert.
• DeepDeWedge is a published method with an open-source implementation.
• The volume is small (64³), which should be manageable on CPU.
• A reference solution presumably exists (the author generated the ground truth).

Negative aspects:

• CPU-only execution of a deep learning method could be very slow. DeepDeWedge involves training a neural network. On CPU, this could take many hours for even a small volume, potentially exceeding agent time limits.
• The solvability within typical agent time constraints (often ~1 hour of wall clock time) is a significant concern. If the neural network training takes hours on CPU, this could be practically unsolvable within the time budget.

Judgement: Uncertain — The method is proven to work, but the CPU-only constraint combined with neural network training raises serious questions about whether this can be completed within typical agent time limits. The proposal should address expected runtime.

Difficult

Positive aspects:

• The task requires domain knowledge: understanding the missing wedge problem, cryo-ET reconstruction, and how to set up DeepDeWedge correctly.
• Multiple decisions must be made: initial reconstruction method (e.g., weighted back-projection), DeepDeWedge configuration, hyperparameters.
• The author reports that Opus 4.6 cannot solve this task.
• A domain expert unfamiliar with the method would need 1-3 hours.

Negative aspects:

• If DeepDeWedge has good default settings and clear documentation, the task might reduce to "install the package and run the example." However, the author argues this is not the case.
• The 64³ volume is quite small, which simplifies things compared to real-world scenarios.
• Much of the difficulty may stem from environment setup (installing dependencies, dealing with CPU-only limitations) rather than scientific reasoning.

Judgement: Accept — The combination of domain knowledge requirements, multiple decision points, and reported LLM failure suggests genuine difficulty. The concern about difficulty being primarily "engineering" rather than "scientific" is partially mitigated by the need for reconstruction expertise.

Scientifically Grounded & Interesting

Positive aspects:

• Cryo-ET reconstruction is a core workflow in structural biology labs worldwide.
• The missing wedge problem is a real and significant challenge in cryo-ET.
• DeepDeWedge is a recent, published method (Nature Communications) that represents the state of the art.
• The task directly mirrors what researchers actually do: apply a reconstruction method to a tilt series and evaluate the results.

Negative aspects:

• The task is somewhat routine application of an existing tool rather than pushing scientific boundaries. However, routine application of complex tools is exactly what many scientists need to do.

Judgement: Strong Accept — This is a genuinely useful scientific workflow that researchers perform regularly. The domain is important and the specific challenge (missing wedge reconstruction) is well-motivated.

Outcome-Verified

Positive aspects:

• The verification is based on PSNR of the output reconstruction vs. ground truth — this is purely outcome-based.
• The output format is simple and well-defined (MRC file).

Negative aspects:

• The task description explicitly says "Use the DeepDeWedge method." This is a process constraint, not an outcome constraint. If the verifier only checks PSNR, then this instruction is either unenforceable or requires a process grader to verify.
• If the PSNR threshold is set so that only DeepDeWedge-class methods can achieve it, then it becomes de facto outcome-verified. But this needs to be demonstrated.
• The author's justification for specifying the method ("it is a quite common task in cryo-ET to attempt reconstruction with a particular algorithm") is reasonable from a scientific perspective, but it creates a tension with the outcome-verified criterion.

Judgement: Uncertain — The method specification is a process constraint that conflicts with outcome-based verification. This can be resolved if the PSNR threshold is set high enough that only sophisticated methods (like DeepDeWedge) can achieve it, making the method constraint implicit rather than explicit. Alternatively, the method constraint should be dropped in favor of a pure outcome-based approach.

Final Analysis

This is a scientifically well-motivated task from a real research workflow. The core idea — reconstructing a 3D volume from a cryo-ET tilt series while overcoming the missing wedge — is genuinely interesting and practically relevant. The verification approach (PSNR against ground truth) is fundamentally sound.

However, there are several concerns that need to be addressed:

1. Process vs. outcome verification: Specifying "use DeepDeWedge" is a process constraint. Either the PSNR threshold should be set high enough to implicitly require advanced methods, or the method constraint should be removed.

2. CPU runtime: DeepDeWedge involves training a neural network. On CPU with a 64³ volume, this could be very slow. The proposal needs to demonstrate that the task is completable within reasonable time limits.

3. PSNR threshold: The specific threshold for passing needs to be stated and justified. It should be shown that naive methods (e.g., simple back-projection) cannot pass while DeepDeWedge can.

4. The task essentially reduces to "install and correctly configure a specific software tool" — which is a legitimate type of difficulty, but the scientific reasoning component could be more prominent.

The task has clear potential but needs refinement, particularly around the process constraint, runtime feasibility on CPU, and threshold calibration.

Decision: Uncertain

🤖 This is an automated recommendation for a human maintainer — not a final decision. Based on TASK_PROPOSAL_RUBRIC.md. Model: claude-opus-4-6

[StevenDillmann]

/approve

[github-actions[bot]]

✅ Proposal Approved

cc @reinhardh @AllenGrahamHart

Hi Reinhard,

Great news — your Terminal-Bench Science task proposal "Cryo ET reconstruction" has been approved by the maintainers! 🎉

You're now invited to build and submit your task as a Pull Request. Here's how to get started:

1. Implement your task — follow the Contributing Guide and the Harbor task format. You can also browse tasks/ for reference implementations.
2. Submit a PR — make sure the automated checks pass as much as possible, then open a PR against main and reference this discussion.
3. Track your task — once your PR is open, follow its review stage, assigned reviewers, and ball-in-court on the 📊 Task Dashboard.

Deadline: Tasks must be submitted by August 17, 2026. Most tasks need a few rounds of review — please start early.

For more on the project, see the Terminal-Bench Science announcement on tbench.ai.

Questions? Drop by #tb-science on Discord or reply in this thread.

Looking forward to your PR,
Steven