We discuss some geometric problems related to the definitions of quasilocal mass proposed by Brown and York (Contemporary mathematics, vol 132, American Mathematical Society, Providence, pp 129-142, 1992; Phys Rev D (3) 47(4): 1407-1419, 1993) and Liu and Yau (Phys Rev Lett 90(23): 231102, 2003; J Am Math Soc 19(1): 181-204, 2006). Our discussion consists of three parts. In the first part, we propose a new variational problem on compact manifolds with boundary, which is motivated by the study of Brown-York mass. We prove that critical points of this variation problem are exactly static metrics. In the second part, we derive a derivative formula for the Brown-York mass of a smooth family of closed two dimensional surfaces evolving in an ambient three dimensional manifold. As a by-product, we are able to write the ADM mass (Arnowitt et al. in Phys. Rev. (2), 122: 997-1006, 1961) of an asymptotically flat 3-manifold as the sum of the Brown-York mass of a coordinate sphere S-r and an integral of the scalar curvature plus a geometrically constructed function Phi(x) in the asymptotic region outside Sr. In the third part, we prove that for any closed, spacelike, 2-surface Sigma in the Minkowski space R-3,R-1 for which the Liu-Yau mass is defined, if Sigma bounds a compact spacelike hypersurface in R-3,R-1, then the Liu-Yau mass of Sigma is strictly positive unless Sigma lies on a hyperplane. We also show that the examples given by O Murchadha et al. (Phys Rev Lett 92: 259001, 2004) are special cases of this result.