Depending on what AV out you previously used, the PS3 may well not be outputting to HDMI at all, in which case you need to reset to basic output whereby it outputs on which one it detects has a cable in. Then you can up the res etc once it has booted up, I believe you do this by turning it on, and as soon as you have done that hold down the power button until it beeps three times, Google is your friend if this is wrong.
Then you can set it up right. Otherwise, is your monitor HDCP compatible? It must be for this to work, unless you have a decoder built into the cable which you can buy. I use an HDMI>DVI, (no converter), to our HDCP compatible 720p TV and it works fine, so it should be fixable!
Charlie -