Henry ("Harry") Brown was the natural son of the Third Earl Manvers. He was born in 1844, in Perlethorpe, Nottinghamshire, a village made up of employees of the Earl's Thoresby Estate. Harry joined the 41st Regiment of Foot when he was 17, and served for 21 years, including ten years in India. He married Lucy Street, the daughter of a widowed army schoolteacher, and they had two daughters in India. Back in England, they had eleven more children, nine of whom survived. After the army, Henry was Superintendent of Stewards for Cunard Shipping Lines in Liverpool. He and Lucy spent their final years at Torrington Farm in Chesham, Buckinghamshire.