The community group says that the public has previously “enjoyed habitual access” to Castletown House via Gay’s Avenue