And the Rangers are left searching for some answers as to why their dramatic efforts keep coming up short.
"It hurts, because we felt like we could win that game," Rangers manager Ron Washington said.
An RBI double by Rays designated hitter Pat Burrell with two outs in the eighth inning broke a 3-3 tie, and appeared to be the clincher with J. P. Howell pitching the ninth.
But with two outs in the inning, Byrd took a 2-1 pitch and blasted it out to left field, his 15th home run of the season and a clutch shot that seemed to suck the wind quickly out of Tropicana Field.
"J.P.'s done such a wonderful job this year," Rays manager Joe Maddon said. "It's difficult for him in that moment to give up that hit with two outs. The guys absolutely picked him up tonight."
The score would not stay tied for long, though. Rangers reliever Jason Grilli, activated off the 15-day disabled list before the game, hit Evan Longoria with an 0-2 pitch and then walked Ben Zobrist with no outs in the 10th. Pena's line drive up the middle drove in Longoria from second with the winning run.
"I've got to make my pitches and I didn't," Grilli said. "I'm not going to try and let this linger in my head for too long."
Michael Young fell a double short of the cycle -- his third consecutive three-hit night -- but he accounted for almost all of Texas' productivity for most of the game. For the second night in a row, the Rangers couldn't seem to find their way offensively against Tampa Bay's starting pitcher.
Matt Garza gave up six hits and three runs in seven innings with seven strikeouts, outpitching Rangers rookie Tommy Hunter, who lasted only five innings and allowed three runs. It's the sixth straight game a Texas starter has been unable to pitch at least six innings.
"There were a couple at-bats where they worked it," Hunter said. "I need to get them on or out in three or four pitches."
Hunter was done in by two home runs by Pena, his league-leading 33rd and 34th shots of the season. Pena's second homer, a two-run shot in the fourth, gave Tampa Bay a 3-2 lead. Pena said after the game he was motivated by the passing of a close friend from college, who died of cancer on Friday night.
After his home run, Byrd had to deal with rowdy fans in left field who pelted him with peanuts before he called security over to deal with the matter.
"They were throwing peanuts," Byrd said, adding sarcastically, "it was fun."
The Rangers nearly came back from 5-0 down in the eighth inning on Friday, and had the tying run at the plate in the eighth before a costly baserunning error shot down the rally. Texas ended up losing, 5-3.
On Saturday, Washington's happiness with his team's grit and perseverance couldn't drown out the bitterness of falling just short again to a foe in playoff contention.
"Those guys played their hearts out out there," Washington said. "That was two teams battling and fighting to win a ballgame."
Texas has now lost four of its past five, and will try to avoid a sweep at Tropicana Field before heading to New York for another tough series on Tuesday. And directly in the rearview mirror are the surging Rays, reminding the Rangers that they've been through all this before.
"They are a team that went through this last season," Byrd said. "They just kept coming and coming. They were making the pitches. They weren't rattled at all. That's the new Rays for you."