OOB effects can be done with only one photo or with several images composed and you can either leave the original background as is or distort it to add a different look. Following example was done using only one photo and the background hasn’t been distorted.
What we are trying to achieve, is taking the photo on the left and transform it into the lion seen on the right, which pops out of the frame and gives that 3D illusion.

Shortcuts used throughout this tutorial:
Brush (B)
Rectangular marquee tool (M)
Layer icon
Layer mask icon
Layer visibility
Adjustment layer icon
- Alt+Backspace to fill the layer with the current foreground color
- Ctrl+Backspace to fill the layer with the current background color
- Ctrl+D to deselect the current selection
- Ctrl+I to invert the mask (invert in general)
- X to swap the foreground with the background color
[1] Open the image you would like to use or
download my original if you would like to work along. Choosing a suitable photo helps a lot and before you start, make all the adjustments like levels, cloning, saturation and such.
To start with, I like to set up the layers, making it is easier to work on afterwards. Double click on the layer thumbnail of your background (see Layers palette) to create a layer and name it something meaningful, in my case “lion”. I assume you start with a single layer JPG image, if not, there is no need to do this.

Create a new layer by clicking on the layer icon at the bottom of your layers palette. Rename it to “white background” and fill it with white. If your foreground is set to black and background set to white, simply hit Ctrl+Backspace.
Create another layer, rename it to “black background” and fill it with black (Alt+Backspace). Move them (click and drag the layers) beneath the lion layer. With CS2, you can Ctrl click (press and hold Ctrl, click on both layers) on the layers to move them at the same time.
Create one more empty layer above the lion and call it “frame”. You should now have something like this:

[2] Now it’s time to add the frame. Make sure the frame layer is selected (click on the frame layer in the layers palette once to highlight it). Use the Rectangular Marquee Tool (M) and drag a frame with the size and shape of your desired frame. Don’t sweat it, we are going to adjust it again later. Hit Ctrl+Backspace to fill the selection with white. See example beneath on the left side.
With the selection still active, go to Select/Modify/Contract… and type in a suitable number. In my case, I chose 15. Hit OK and then delete the area in the middle (press the delete key on your keyboard), leaving only a white frame. Hit Ctrl+D to deselect or go to Select/Deselect. You should now have something like the example beneath on the right side.

[3] To make the frame look 3 dimensional, we are now going to distort it. With the frame layer active, go to Edit/Free Transform or hit Ctrl+T. Right click in the middle of the frame and choose “Perspective”. You can now drag the corners up and down to create the 3 dimensional illusion. Right click and choose “Distort”, if you need to adjust the frame in a different direction. You should end up with something like this:

[4] It’s time to delete or better hide some of that area outside of the frame. Make sure the lion layer is selected and click on the layer mask icon (located at the bottom of your layers palette) to create a mask for the lion. The mask will hide (black) the parts we don’t want and reveal (white) what needs to be visible. Nothing will be deleted, so you can edit this anytime you want.
Select a medium soft brush (B) and make sure that your foreground colour is set to black. If you have done something on a different layer in the meantime, make sure your layer mask is highlighted and not the layer itself. This is easy to forget, so double check.
Paint over the parts you want to hide. Use a hard brush around the frame and use a softer brush for the edge of the lion and the napkin. If you go too far, simply swap the foreground colour to white (X) and paint over the bits you want to reveal. Use the X key on your keyboard to quickly swap between black and white.
In my example, using the brush was quick and easy but if you use a different image, you might have to make a selection with a different tool. Remember those white and black backgrounds we have created at the beginning? Now they come in handy, making it easier to see how well your masking is done. Simply click on the icon looking like an eye on the left side of the black or white background to see the different background colours.
You should now have something like this:

[5] In the next step, we are going to finish the frame. In my case, I wanted to have a bit more room, so I resized the canvas size but that might not be necessary for your image. First, create a drop shadow for the frame. Double click on the layer thumbnail of the frame layer to open the Layer Style dialog. Click on Drop Shadow and dial in the numbers you like. I kept it fairly low in this example but you can adjust it anyway you like.
After you are done, go to Layer/Layer Style/Create Layer. This will put your shadow on a separate layer, so it’s easier to work on and you can mask it separately from the frame.
Now, do the same as you have done for the lion already. Select the frame layer, create a layer mask and use the black brush (B) to hide the parts where the head and the napkin overlaps the frame. Do that once more for the shadow and you should have something like this:

[6] As you can see, it starts to look more like the finished version but something is still missing. The shadows is what makes or breaks the OOB effect. For the lion, I decided that the napkin, the head and the leg need some additional shadow.
Select the lion layer, double click on the layer thumbnail and add a drop shadow. I’ve used a soft shadow and for the position, rather than dialling in the numbers, I used the mouse and moved the shadow on the image itself. Simply click and drag on the photo to move the shadow in position.
After you are done, go to Layer/Layer Style/Create Layer and move the shadow layer to the top of your layers palette. Create a layer mask and invert it (Ctrl+I). You could also just use a black brush to hide the parts you don’t need but I found it easier to invert it and hide everything, then use a white brush to paint the shadow where it’s needed. Adjust the opacity if necessary and move its location if it doesn’t look right.

Create one more layer on top of it and use a black brush to paint the additional shadow beneath the front leg. Use a low opacity brush (around 5%) and slowly paint the shadow. Don’t worry when you paint the leg itself, simply add a layer mask and hide it.
Now it should look like this:

[7] Seeing it now, I wanted to enhance the look, as if it came out of a photo. To do that, I decided to lower the saturation and the contrast of the parts which supposed to be a flat image.
Select the lion layer. Press and hold the Alt key and click on the Adjustment Layer icon (at the bottom of the layers palette), then choose Hue/Saturation. In the New Layer dialog, turn on “Use Previous Layer to Create Clipping Mask” and hit OK. I used -20 for Saturation in the Master channel.
Once more, select the lion layer, press and hold the Alt key but this time choose the Adjustment Layer called Brightness/Contrast. Use the previous layer to create a clipping mask and in my case, I used Brightness +10, Contrast -15.
The clipping masks are used so that the background doesn’t change and only the lion image is affected. Obviously, we don’t need these layers to affect the lion head and the napkin, so use the layer mask (the layer mask has already been created automatically) for each adjustment layer and hide the parts which should be 3D. Use the softest possible black brush and paint over the head, the napkin and the front leg.
Done! Add some sharpening to the 3D parts and send me your creation. Just kidding, but if you like, you can certainly send me a copy :) Below is the finished image with the layers palette I ended up with.
