I don’t know for certain, but if I had to guess it would be that they used to make round drainage pipes for agriculture out of Terracotta so the first gutter systems in Europe likely used them because that was what they had available at the time. Later systems likely kept the shape both to match what was around them and because people were used to it and expected it. For metal gutters manufactured through certain industrial processes it would be simpler to bend a flat sheet of metal into a rectangular shape and fasten it together rather than making a cylinder.

Again, that’s just a semi-educated guess but it seems reasonable to me.