Can atmospheric pressure act horizontally on a surface?

If atmospheric pressure acts horizontally on a surface, does that not contradict the meaning of atmospheric pressure that says it is the pressure of column of air caused by its weight on a surface? Weight acts vertically downwards so pressure caused by it should be vertically downwards. But if pressure only acts vertically, then the famous collapsing can experiment is invalid. reasoning needed here!