As early as 1712, the meadows along the South Shore in what was then Speonk were leased to cattle owners from Southampton. Eventually the cattlemen found it easier to build small houses near the meadows to tend the herds. Most of the early residents came west from Southampton and Bridgehampton in the 1740s, built farms and cleared the forests of cordwood. In the 1880s, duck farms thrived in Speonk, but few survived past the century. Supposedly inspired by Indian words meaning "a high place," the name Speonk enchanted some residents and disgusted others. A fraction of the community in 1895 wanted to change the name to Remsenburg, in recognition of Charles Remsen, a prominent resident who donated a new brick Presbyterian Church. The dispute got nasty when Remsenburgers removed the Speonk sign at the railroad station and replaced it with Remsenburg. The Speonk sign was restored and the hard feelings eased with time, but Speonk remained split.