If it is a linear power supply (big, heavy, big transforer(s), with one really big (or several good sized) capacitors), then the output voltage (if it goes beyond the specs of the capacitor(s), bridge rectifier(s), etc, could damage the supply... otherwise it'll probably be ok to tweak it up just a bit.
As per the problem you are having, go ahead and put the peltier up to it (if it's a linear supply). If it is a linear supply, then as you near the max output current for the given output voltage (ie, 24V at 7.5 amps), then the voltage will start falling... at some point an equilibrium will be reached (I don't want to work through the equations right now
Also depends somewhat on the transformer/capacitor combination used, etc..) At this equilibrium, the voltage will be below 24 Volts and the supply will be outputing some current above 7.5 Amps... whatever point that coincides with on the peltier's input power curve. (Remember, you can't directly measure the resistance of a peltier with an ohmeter, because the current put into the TEC with the ohm meter will generate a temperature differential in the TEC and, by the Seabach (?) effect, it will create a voltage throwing off the resistance... I've heard you can use an LCR meter, but anyway...) But a lot of TECs (especially from tedist.com) will ship with the resistance on the spec sheet, so you'll have more of an idea what the voltage/current might be.
Bottom line is, if it's a linear power supply, just go ahead and hook it up (if it's a switching power supply, don't try it).
But, since you are 'working' the supply harder, the transformers + etc will get warm(er).. so make sure they get proper air flow for a longer life.. afterall, this is ProCooling.com!
Rich W.