Metal nanostructures display several types of resonances. In the visible and near-IR spectral regions, there are localized surface plasmon resonances (LSPRs) that involve the coherent oscillation of the conduction electrons. Extended metal nanostructures, such as nanowires or nanoplates, also exhibit propagating surface plasmon polaritons (PSPPs), which are motions of the electrons at the surface of the structure that have a well-defined momentum. In addition, the vibrational normal modes of metal nanostructures give rise to low frequency resonances in the gigahertz to terahertz range. These different types of motions/resonances suffer energy losses from internal effects and from interactions with the environment. The goal of this perspective is to describe the part of the energy relaxation process due to the environment. Even though the plasmon resonances and acoustic vibrational modes arise from very different physics, it turns out that environmental damping is dominated by radiation of waves. The way the rates for radiation damping depend on the size of the nanostructure and the properties of the environment will be discussed for the different processes. For example, it is well known that for LSPRs, the rate of radiation damping increases with particle size. However, the radiation damping rate decreases with increasing dimensions for PSPPs and for the acoustic vibrational modes.