Cleaning once a year is recommended for most ponds. If you have a very large pond and the effort, disruption and cost of the work put you off doing this every year, then there is a case for delaying the work.
Ponds that have not been cleaned out in three years will usually look messy and unloved. Five years is definitely too long. If your ponds contain sensitive fish that are difficult to care for outside of the pond and don't always respond well to the disruption of a pond cleanout - such as large koi then again there may be a case for delaying a pond cleanout.