This is the site of a Roman gate into the city, Worthgate. It survived as a Roman arch until 1791.
Castle Row leads into the centre of the city, and also is the route to find a walk along the city walls that you can see if you turn right.