Well, his scientists started it. But it turns out that they weren't as close as some had supposed. They were concentrating on trying to build a heavy water reactor which would run on natural, unenriched uranium in order to produce as a byproduct the enriched uranium needed to build a bomb. As it happens, they were on the right track for that particular system; but they told the Wehrmacht that they were looking at a 10-15 year development cycle at best. So the Wehrmacht decided it didn't have a quick enough military application and relegated the whole program to minimally funded civilian research. The only facility found, after the war, was a small partially completed reactor that hadn't even begun to produce enriched uranium.

Meanwhile, in the US and UK, massive amounts of funding and manpower were spent developing means of enriching uranium without using heavy water reactors. The Manhattan Project was the largest industrial project of the war, at Los Alamos, Hansford and Oakridge. It was a massive effort, while the Germans were working in a tiny little civilian lab. The thought that the Nazis were trying to make a bomb was enough, even though it turns out that their effort was minimal at best.