The reliability of naturalistic observations of social, physical and economic environments of bars

Drinking in bars contributes to numerous public health problems, including violence and motor vehicle crashes. In order to formulate effective preventive interventions, it is essential to identify which specific features of bar environments are related to increased risks. Unobtrusive ethnographic observations are one approach that has been used to characterize these features; however, no studies have assessed reliability in a representative sample of bars. We performed brief scouting assessments in all 165 bars in six purposively selected California cities, followed by unobtrusive observations from a subsequent representative sample of 97 bars which were located in low- and high-bar density areas of the cities. Inter-rater reliability between two independent observers assessed individual item reliability, and principal components analyses assessed the reliability of a series of scales describing the physical, social, and economic characteristics of the bars. For the scouting assessment, items exhibited at least moderate reliability (κ or r ≥ 0.40). For the unobtrusive observations, items assessing physical and economic environments (e.g. pool table present, κ = 0.90; index beer cost, r = 0.82) had moderate to outstanding reliability (κ or r > 0.80). Items describing the social environment generally had poorer reliability, though group aspects (e.g. patron count, r = 0.78; patron circulation, r = 0.64) had better reliability than individual behaviors (e.g. derogatory speech, κ = 0.12). Scales constructed from specific sets of items exhibited modest reliability. The individual metrics and principal components we present will enable future studies seeking to disaggregate relationships between bar characteristics and public health problems.