Echocardiography is one of the primary imaging modalities used in the diagnosis of cardiovascular diseases. It is commonly used to extract cardiac functional indices including the left ventricular (LV) volume, mass, and motion. The relevant echocardiography analysis methods, including motion tracking, anatomical segmentation, and registration, conventionally use the intensity values and/or phase images, which are highly sensitive to noise and do not encode contextual information and tissue properties directly. To achieve more accurate assessment, we propose a novel spectral representation for echo images to capture structural information from tissue boundaries. It is computationally very efficient as it relies on manifold learning of image patches, which is approximated using sparse representations of dictionary atoms. The advantage of the proposed representation over intensity and phase images is demonstrated in a multi-atlas LV segmentation framework, where the proposed spectral representation is directly used in deformable registration. The results suggest that the proposed spectral representation can provide more accurate registration between images. This in turn provides a more accurate LV segmentation. Finally, it is the first time that a multi-atlas approach achieves state-of-the-art results in echo image segmentation