There are many parts of a stage's landscape that you can interact with. Doing so may reveal Puzzle Pieces. Try pounding the ground near interactive elements, blowing out fires or the leaves on plants. In some cases, collecting all of the Bananas floating in a circle in the stage will reveal a Puzzle Piece. Puzzle Pieces may also be found in secret areas (which are in most stages in the game) in which there will be a challenge to collect Bananas, Banana Coins and Extra Life Balloons and a Puzzle Piece will appear.
The importance of having Puzzle Pieces in a stage is to unlock collectibles in the gallery. Collect all the Puzzle Pieces in a stage and you'll unlock an image that goes in the image gallery. Collect all the Puzzle Pieces in a world and you'll unlock a diorama, which are 3D spinnable models of a world (i.e. the diorama of the Jungle world could be Mugly chasing Donkey Kong up a huge tree). Puzzle Pieces, if you collect every single one in the game, will help you on your way to completing the game completely.