In the book Structure and Interpretation of Computer Programs, there is a Scheme implementation of Fermat's test for primality. This is known as Fermat's Little Theorem. In a footnote, it is mentioned that there is a rare category of little understood numbers known as "Carmichael numbers." These obscure little beasts will actually generate false positives for Fermat's test.
From a mathematical standpoint, Carmichael numbers demonstrate that Fermat's test is not sufficient for determining whether or not a number is prime. However, when using a "correct" algorithm, the odds of cosmic radiation causing the computer to generate an incorrect result are less than the odds that the number you're testing for primality is a Carmichael number. Thus, from an engineering standpoint one might argue that the "incorrect" algorithm is the better choice because in the real world, it Gets Things Done.
There's an excellent reason why there are more engineers than mathematicians. We need to Get Things Done. I suspect that many computer programmers would prefer to be the mathematician but are forced to be engineers. Heck, some of us are apprenctice carpenters who aren't qualified to pick up a hammer without guidance.
As for myself, I think I'm an engineer who dabbles in mathematics. I want to know why things work the way they do and what the "correct" way to do things is, but I also have to Get Things Done. If using the "wrong" theorem lets me do that, I'll cheerfully smile and be wrong if the "right" theorem causes more problems than it solves.
Incidentally, this is yet another benefit of testing. Yes, someone else can argue that your code is awful (and you may know your code is awful), but having ugly working code versus their beautiful non-working code is a very strong rebuttal. Then the tests let you turn it into beautiful code.