Scorpius X-1 (Sco X-1) and x-ray transient XTE J1751-305 are low-mass x-ray binaries (LMXBs) that may emit continuous gravitational waves detectable in the band of ground-based interferometric observatories. Neutron stars in LMXBs could reach a torque-balance steady-state equilibrium in which angular momentum addition from infalling matter from the binary companion is balanced by angular momentum loss, conceivably due to gravitational-wave emission. Torque balance predicts a scale for detectable gravitational-wave strain based on observed x-ray flux. This paper describes a search for Sco X-1 and XTE J1751-305 in LIGO science run 6 data using the TwoSpect algorithm, based on searching for orbital modulations in the frequency domain. While no detections are claimed, upper limits on continuous gravitational-wave emission from Sco X-1 are obtained, spanning gravitational-wave frequencies from 40 to 2040 Hz and projected semimajor axes from 0.90 to 1.98 light-seconds. These upper limits are injection validated, equal any previous set in initial LIGO data, and extend over a broader parameter range. At optimal strain sensitivity, achieved at 165 Hz, the 95% confidence level random-polarization upper limit on dimensionless strain h0 is approximately 1.8×10−24. The closest approach to the torque-balance limit, within a factor of 27, is also at 165 Hz. Upper limits are set in particular narrow frequency bands of interest for J1751-305. These are the first upper limits known to date on r-mode emission from this XTE source. The TwoSpect method will be used in upcoming searches of Advanced LIGO and Virgo data.