A strong wind made good football difficult, but although both sides had chances it was not until the 78th minute that the Glovers broke the deadlock.
Latvian international Andrejs Stolcers collected a loose ball down Yeovil's left flank and cut inside before curling a superb shot beyond the outstretched fingers of home keeper Willy Gueret.
The Swans tried to hit back but it was Yeovil who sealed a notable victory in front of League Two's biggest crowd of the season.
Leading scorer Phil Jevons was left unmarked as he fired into the roof of the net from close range for his 17th league goal of the season.
As expected Andy Robinson came back into midfield in a Swansea side brimming with confidence after netting 12 goals in their last three matches.
He replaced Marc Goodfellow who had been recalled from his loan spell by Bristol City.
The Glovers kept Swansea penned in their own half in the opening minutes yet it was Lee Trundle who had the first strike on goal when he forced Chris Weale to dive full length before clutching the ball at the second attempt.
Three minutes before the interval, Trundle twisted and turned on the edge of the area before firing in a shot that flew just wide of Weale's far post.
Two minutes after the restart Paul Terry went into the book after bringing down Robinson but the free-kick was lifted high over the visitor's crossbar.
Yeovil hit back and Jevons took advantage of an error in the Swansea defence, but as Gueret came out he toe poked wide.
Swansea had the ball in the net in the 51st minute when Trundle slipped a neat pass into the path of Paul Connor but the former Rochdale striker's effort was disallowed for offside.
In the 59th minute, the visitors almost took the lead when a twice take free-kick by Lee Johnson was headed against his own crossbar by Alan Tate.
Yeovil had another opportunity five minutes later but Gueret brilliantly tipped over a close-range header from Terry Skiverton.
Yeovil's second-half persistence finally paid off with Stolcers' strike and after Lee Trundle had almost equalised with a strong effort that bounced off the frame of the Yeovil goal, Jevons completed the misery with almost the last kick of the match.