Villagers will only marry and produce children if there is an empty home for them to move into. That is, there can only be one pair of villagers producing children per house. As homes start to become available as your population dies of you will begin to see a growth in your population, but it'll eventually stop growing again since the villagers will just run out of empty homes again. If you want your population to continue to grow you'll need to continue to build new homes for villagers to pair off into and begin families. Be careful, however, since building too many homes can lead to your population growing faster than you can produce food and other resources to support it.
Size - I tend to build as big as possible. My reasoning is that if fields were very small, there would be lost work time between planting and harvesting while waiting for the crops to grow.
Farmers - With only one farmer, there will likely be crops lost to freeze as winter begins. The more farmers you put to work, the faster they will harvest the crops and thus produce more crops.
If your goal is sheer food production, you should max out farmers on every field.
If your goal is maxing out efficiency per worker, you should have one or two farmers per field - they will hit their maximum amount of crops planted and gathered even if some go to waste.
If your goal is maxing out efficiency per space of land, you should max out the farmers on every field to reduce the need for more farms.
Roaming - Farmers are "assigned" to a particular field, split up evenly if you do not micromanage the number of farmers per field. If farmers on one field die or are reassigned to another job, farmers from other fields will come over to even out the numbers to each field.
Fishing dock - If you want to further micromanage your farmers every year, you can re-assign them to other tasks. Without any intervention, they will act as laborers during their downtime. Personally I think it's too much effort to go to that level of micromanagement, but if you really needed extra production in a certain area it wouldn't hurt, so long as you get the farmers back to their fields in time for spring planting.
Regarding your comment on a single farmer able to handle a field by themselves - this could be a number of factors, size of the field, quality of the tool, level of education, the farmer could have just eaten just before it was time to plant, harshness of the weather in early spring or late autumn. Leaving it up to these factors sets you up for large swings in food supply.
Best Answer
As of 7/3/2014 according to the banished wiki:
http://banished-wiki.com/wiki/Farming
"Farming can be considered a sustainable method of obtaining food for your citizens, though farming for many seasons in one place will ruin the soil.[1] Though, the developer has confirmed that soil degradation has not been implemented."
In other words, no, at the moment it is not necessary