Overextending is the process of a player filling their Monster Zones and/or Spell & Trap Zones with more cards than necessary, leaving the player at risk from nukes such as "Dark Hole" or "Black Rose Dragon". Having a full field may also prevent them from playing new cards.
A modern example of overextending would be the now-infamous Extra Link mechanic. While such a powerful field can win games by itself, without sufficient protection and/or if too many resources are committed to the setup, the loss of the board can lead to a major disadvantage, both in card advantage and spare resources.