The Norman Castle was a royal Castle built after the Battle of Hastings, no doubt to pacify the locals. It was built a little further east of here on a large mound of earth now known as Dane John. The mound was roman in origin.
The castle was shifted to this spot and was apparently in place by 1095. The castle was used as a prison for the County of Kent for many centuries but fell into disuse in the 18th Century. It's now rather a battered ruin having been used as a coke store for the gas works. You cannot now enter the building because it's dangerous, but you can walk round the outside.