He came to acting late, having joined an amateur dramatics group at the age of 27 and gained his part in the Street a year later, after Patricia Phoenix saw him on stage in Bolton and urged the programme's producers to hire him.
Subsequent television credits included Emmerdale Farm, Till Death Us Do Part, Sykes, All Creatures Great and Small, Shoestring, Citizen Smith, Born and Bred, Mansfield Park, Minder, The Gentle Touch, Much Ado About Nothing and Last of the Summer Wine. On film, he has appeared in Jabberwocky, Porridge and Brazil.
In 1977, he first came to the attention of BBC comedy writer and producer David Croft, who cast him in a small part in an episode of It Ain't Half Hot Mum. Impressed by his comic ability, Croft continued to use Kaye in guest roles in several of his sitcoms, including Are You Being Served?, Come Back Mrs. Noah and Oh Happy Band!.
Then in 1982 came the role of Kaye's career when Croft chose him to play the central character of French café owner René Artois in 'Allo 'Allo!, a sitcom based in Nazi-occupied France during World War II. The series was a smash hit around the world, running for 85 episodes until 1992, and Kaye's portrayal of cowardly, sex-mad Artois drew much acclaim and a BAFTA nomination in 1986. A stage version also played to packed houses in the West End of London and internationally for many years.
On 25th January, 1990, Kaye narrowly escaped death when severe gales blew an advertising billboard through the windscreen of his car, and a nail from a plank of wood pierced his forehead. Despite the seriousness of his condition, Kaye defied the odds and pulled through, returning to both stage and screen in 'Allo 'Allo! within months of the accident. Later that year, he made an appearance on Happy Birthday Coronation Street and recalled his time as Bernard Butler.
After 'Allo 'Allo! ended, Kaye concentrated mainly on stage work, appearing in numerous UK tours of comedy plays and farces. In 2004 he was one of many familiar comic actors to appear in Revolver, an anarchic sketch show, which also featured Roy Barraclough and Julie Goodyear. In 2007, he revived the character of René for a BBC2 reunion special The Return of 'Allo 'Allo, as well as an Australian production of the stage version.