The 46-year-old Paes and Lativia's Ostapenko, the 2017 French Open champion, suffered against the British-American pair 2-6, 5-7 in a second round match that lasted one hour and seven minutes.
The pair had earlier beaten Storm Sanders and Marc Polmans 6-7, 6-3, 10-6 in the opening round.
Australian Open: Bopanna only Indian in the fray now
Meanwhile, Rohan Bopanna is the only Indian left in the fray now.
The 39-year-old and Nadiia Kichenok of Ukraine will take on fifth seed Nikola Mektic and Barbora Krejcikova in the mixes doubles quarter-finals on Wednesday.
Bopanna had earlier lost in the opening round of the Australian Open doubles event.
Bopanna and Yasutaka Uchiyama lost against the Bob and Mike Bryan.
Retiring Paes will hope to sign off with a bang
Paes will hope to sign off with a bang this year and he will be trying his luck in the upcoming French Open next.
The veteran tennis ace also has the Wimbledon and US Open to look forward to.
Paes has had a defining career that has lasted so long.
With 18 Grand Slams (men's doubles and mixed), Paes will eye a final push.
Paes won four career Slam titles at Australian Open
Paes managed to reach four Australian Open doubles finals in his career. He won the title once and was the runner-up on three occasions. Meanwhile, he pocketed three mixed doubles titles at the Australian Open. He was also the runner-up twice here.