Just want to let you know that I too have experienced testicular pain from taking Adderall.
In my case, the causation was very clear: The pain first presented in Sept. 2005, about 2 weeks after I started a daily dose of 25mg of Adderall. I tried various things for 2.5 years, and the pain occasionally went away or decreased in intensity, but always came back. The pain didn't completely go away until I stopped the Adderall, at which time, the pain stopped completely.
More details below:
When the testicular pain started, there was a lot of press at the time on bike seats and perineal injury, and I thought my heavy bike-as-transportation habit was partly responsible. I got a new bike seat, I went to several MDs and urologists, I spent a year and a half in frequent therapy with my chiropractor (who also does deep tissue stuff), I took private sessions with a pelvic yoga specialist. The chiro therapy and the yoga both seemed to help somewhat. Over this 2.5 year period I would occasionally have a month or six weeks without pain -- but it always came back. And for the last year, there wasn't a single week where I didn't have at least a little bit of pain.
Finally, in April 2008, I thought "why don't I try getting off the adderall?" For some odd reason, even though the pain started not long after I started Adderall, I had never suspected it was the Adderall (partly because testicular pain is not listed as even a rare side effect and partly because I had other things going on around that same time that confused the issue for me.). My doc switched me to Focalin, and the effect was immediate -- no pain whatsoever since then. (About a month after switching to Focalin, I weaned off of stimulants altogether.)
So anyway, you're not crazy to link Adderall to your testicular pain.