The house is a 19th -century- flint faced mansion and is one of the largest flint structures in the country. The core of the present house was built over the period 1804-1830 for the Lords Selsey, using James Wyatt as architect.

When William James acquired the estate in 1891, he set about altering and greatly expanding the house under the direction of Sir Ernest George and Harold Peto. Since becoming a college in 1971,under the direction of Edward James, further extensive alterations have been made, great care being taken to ensure that these continue the high standards of building practice and design of previous generations.