No, no, no. You DO NOT have to worry about voltage. Increasing the voltage can only be done by 1) a hardware mod 2) flashing with a custom/hacked BIOS, neither of which you want to do at this point.
The idle temperature before the benchmark was 56 C, and the maximum temperature during the benchmark was 76 C with a score of 4879. What do y'all think?
The temps aren't bad, but the temps while it is OC'd are more important than at stock. Keep checking temps as you increase the frequency's. IMO, I would keep full load temps under 85C (90C is probably OK, but that starts making me nervous.)
How are you checking load temps BTW? You need to use Rivatuner, which creates a graph. After the test is done, you scroll back through the graph finding the peak temp. If you just check it after the test is done you will not get an accurate load temp.
If it is not artifacting, I would continue to increase frequency's and see what temps and scores do.