Caveats and intent: Painstakingly extracted via trial and error, ever evolving. Mostly an exercise to think through prior mistakes and avoid making them again. These are my personal rules, they might not work for you but it’s invariably a mistake when I stray from them. Given that I’m not the world’s most successful researcher (I’m not too shabby either though) maybe you’re better off taking advice from someone else. On the other hand, I would contend that these rules are universally useful.
There’s a ton of good blogs and information about how to structure and think about your research vision but comparatively less thinking on what the day to day looks like. I was trying to figure out what rules I would try to insist my students follow or try and figured it might be helpful for other folks. It seems like a lot but I’d say outside of the detailed notebook most of these are are either a few minutes here or there or a suggestion to add structure to your thinking to aid you in thinking more clearly.
This is the mistake I make the most often. ML experiments are so easy to mess up you absolutely need to verify at each step that the minimal experiment is working. The minimal experiment is the absolute bare minimum needed to check that either your code works or your idea makes sense. Unfortunately, I don’t think there’s a coherent definition of minimal, it seems to mostly be an aesthetic thing. I’d call it the thing that has the least moving pieces.
Since nothing is better than concrete details, let me just list all the ways I’ve done this wrong:
I had an RL problem where we had to solve some task where the dynamics were defined by N trajectories. Instead of checking whether I could solve one interesting trajectory perfectly, I kept running the algorithm over all N trajectories.
I had a problem that could be run in single or multi-agent mode. Instead of trying to verify correctness of the codebase by running the single-agent version, I jumped right to the multi-agent version.
I had a multi-agent problem where I could run it with 100s of agents or just a few. I jumped right to 100s. I’ve done this one three times.
This should consist of
It’s so easy for your ideas to seem clear until you write them down. Once you write them down you start noticing that:
Caveat: this doesn’t work for curiosity driven research where you don’t actually know what you’re setting out to find before you get there. However, if you think you know the destination it’s worth checking that you actually want to / can get there!