In 1878 the architect Richard Norman Shaw undertook extensive building works for Henry Bingham Mildmay, whose ancestors included Sir Walter Mildmay, Chancellor of the Exchequer to Queen Elizabeth I and founder of Emmanuel College Cambridge. Norman Shaw remodelled and extended the house, while retaining the sixteenth/seventeenth century house to the south-west. He incorporated the Mildmay family crest in one of the windows on the main staircase. It was a house built for entertaining distinguished visitors. In 1887 Crown Princess Friedrich Wilhelm visited Flete House as well as the Kaiser's mother Princess Victoria and our own Queen Mary who stayed in 1938. It is said that Goebbels, Hitler's Minister of Propaganda, visited Flete in the 1930s before the war (but we don't talk about that!)
Flete House was used by the City of Plymouth as a maternity hospital during and after the Second World War. At the time the Second World War broke out, the estate was in the hands of Lieutenant-Colonel, the Lord Francis Mildmay of Flete. The first baby to be born at Flete arrived on Monday July 14, 1941, and by the end of 1941 there had been a further 124 births. Lord Mildmay remained in residence at the House until his death in 1947 and was apparently often seen in the wards. Dave Hill of the pop group Slade was one of the babies born at the House. In August 2010 there was a reunion of some of the 'Flete Babies' from the Plymouth area which was filmed by the BBC and hosted by John Sargeant.
The hospital closed on 8 May 1958, less than a month after the last baby was born there. Flete House was returned to the Mildmay family in 1959. For a while the house was operated by the Country House Association, until becoming luxury retirement apartments in 2005.