Pakistan, who lead the two-match Test series 1-0, were 111 for four at tea in response to Sri Lanka's first-innings 378 and still trail by 267 runs in Galle.
Fawad Alam, on 18, and Agha Salman, on 13, were batting at the break.
Left-arm spinner Prabath Jayasuriya bowled Azam for 16 on a turning delivery that got the edge of the dangerman's bat to crash onto the stumps.
Imam-ul-Haq attempted to rebuild the innings with Mohammad Rizwan but Dhananjaya de Silva broke through with his off-spin to send back Imam on 32.
De Silva is standing in as captain in place of Dimuth Karunaratne, who did not take the field in the Pakistan innings because of back pain.
Rizwan hit a few boundaries to counterattack but Ramesh Mendis trapped him lbw for 24 and a review did not help the wicketkeeper-batsman, with replays showing the ball would have hit leg stump.
Pakistan had lost opening batsman and first-match hero Abdullah Shafique for a duck before lunch when he inside-edged a delivery from Asitha Fernando onto his stumps.
Sri Lanka had resumed the day on 315-6 in former captain Angelo Mathews' landmark 100th Test, with overnight batsman Niroshan Dickwella making 51 and a useful cameo of 35 by number nine Ramesh.
Fast bowler Naseem Shah and leg-spinner Yasir Shah claimed three wickets each to end Sri Lanka's innings in the first session.