Gielen was born in Dresden, Germany, to Rose and Josef Gielen, his father, an opera director. Through his mother, Rose, he is the nephew of Eduard Steuermann and Salka Steuermann Viertel. His father was Christian and his mother was Jewish. He began his career as a pianist in Buenos Aires, where he studied with Erwin Leuchter and gave an early performance of Arnold Schoenberg's complete piano works in 1949 (the South-American première). While serving as conductor and répétiteur at the Wiener Staatsoper (1950–60), he conducted much contemporary music outside the opera house. His next operatic appointment was as conductor of Royal Swedish Opera from 1960 to 1965, followed by posts at the Netherlands Opera and the Opern- und Schauspielhaus Frankfurt from 1977. He was principal conductor of the Belgian National Orchestra (1969–73), the Cincinnati Symphony Orchestra (1980–86) and of the Southwest German Radio Symphony Orchestra (1986–99), which he has been closely associated with since.
He has demonstrated a mastery of the most complex contemporary scores, and he has given many premières, including Helmut Lachenmann's Fassade andKlangschatten – mein Saitenspiel, György Ligeti's Requièm, Karlheinz Stockhausen's Carré and Bernd Alois Zimmermann's Die Soldaten. In 1973 he recorded Schoenberg's opera Moses und Aron with Günter Reich, Louis Devos, choir und orchestra of the Österreichischer Rundfunks, used as a soundtrack for the film Moses und Aron. In 1979 he revived Schreker's opera Die Gezeichneten at the Oper Frankfurt, where it had been premiered in 1918. During his time in Frankfurt, later called the Gielen Era, he collaborated with stage directors such as Hans Neuenfels for Verdi's Aida and Ruth Berghaus for Wagner's Der Ring des Nibelungen.
He is also a noted conductor of the symphonies of Ludwig van Beethoven, Anton Bruckner and Gustav Mahler. As a composer, he has elaborated on the tradition of the Second Viennese School and his small oeuvre includes settings of poems by Hans Arp, Paul Claudel, Stefan George, and Pablo Neruda.
In October 2014, Gielen announced his retirement from conducting for health reasons, particularly seriously deteriorated eyesight.
Die Glocken sind auf falscher Spur after Hans Arp, premiere 1970, Joan Carroll, Siegfried Palm, Aloys Kontarsky, Wilhelm Bruck, Christoph Caskel, Michael Gielen, at the Saarländischer Rundfunk festival "Musik im 20.Jahrhundert"
1983 - 1985: string quartet Un vieux souvenir after Baudelaires's Les Fleurs du mal, premiere 1985, LaSalle Quartet, Cincinnati