Tips for doing research¶

Principles¶

Document/back up everything that may be useful for later.
Keep comparisons and statistical reporting strictly fair. That means equal effort, not just equal intent — a baseline you ran once with default settings isn't a fair comparison, however careful the statistics around it are.
Publishing in a strong venue should be a by-product of having something interesting to say, not the goal itself. Aiming purely at publication is like starting a business to turn a quick profit rather than to build something good.
If you are able to constantly obtain reliable negative results, open and critically analysing the negative results will eventually lead to a positive one.
Premature optimisation is the enemy. Don't tune, scale, or generalise anything until you know it's worth keeping.

7 steps, each with a time budget. The hard part isn't the order — it's sticking to the limits. Most ideas die at step 4, and that's the point.

1–2 weeks

Use Claude to gather related papers. Skim around 20 abstracts and read around 5 properly. Work out the main approaches, the standard benchmarks, and who is active in the area.
Ask people what's actually challenging here, and what has already been tried and failed. Failed attempts rarely get published, so you can't learn this any other way.

1–2 weeks

Build the simplest solution and codebase that runs end to end, starting from a decent existing repository, to get a feel for the project.
The codebase should contain data splits, evaluation metrics, seed control, and clean, structured logging of what's going on during a run.
Have a log file and todo list. Anything unexpected, interesting, or important goes in it. This turns out to be your best source of ideas.

2–4 weeks, hard limit

Pick one direction for handling the challenges in the project.
Pick a strong paper, preferably the strongest method, in that direction with a well-maintained code release. Spend a day checking the code runs before you commit weeks to it.
Reproduce the main result, plus the ablations that are relevant to you.
Analyse the paper's strengths and limitations — failure cases, unsupported assumptions or claims. These become the motivation for your own work.

1–2 weeks

You should have some research questions — ones the existing papers leave unanswered or unexplored, or that are motivated by the limitations of the existing work.
You should have some ideas for exploring the answers to those questions, and some hypotheses — your expected answers to them.
Read the literature again to check whether the answers are already available in the literature. Use Claude to widen the search — but verify anything it names before trusting it. Fifteen minutes can save a month.
Use the strongest existing methods to grow your own codebase, keeping the setups and findings worth keeping, and keeping the whole thing simple, so later comparisons run through the same pipeline and differ in only one place.

repeat until one survives

Small model, small dataset, the fastest signal that could show the idea is wrong. Hours per attempt, not days.
Aim for many quick attempts rather than a few careful ones. Careful comes next.
Most ideas die here. Each one is cheap to lose and usually reshapes the next.

A dead idea usually sends you back to step 3. Expect multiple rounds.

2–4 weeks

Try to break it, not confirm it.
Rigorous comparisons against existing methods, plus ablations, with proper statistical reporting.
Check the claim holds elsewhere: another dataset, another model size, another architecture.
Write the abstract and your main claim before the last round of experiments. It shows you whether you actually know what you're arguing.

2–4 weeks

Full-scale runs. Experiments that show why it works, not just that it does.
Clean code, clean configs, honest limitations — easy to write if you spent step 5 attacking your own result.
Continue the project from the limitations of your current work, probably back to step 2.

Who this is for. It assumes you're new to the area. If you already know it well, steps 0–2 shrink a lot and you start near step 3.