As far as the people on it can
5e doesn't offer any rules for how far a ship can "see"; however, the DMG does have some default rules about visibility outdoors. According to these rules, characters can normally see about 2 miles out from ground (or sea) level, and 40 miles out if looking from a vantage point such as a hill or mountain "or are otherwise able to look down on the area around them from a height", though rain or fog can significantly reduce visibility (down to 1 mile in heavy rain, or merely a couple hundred feet in fog).
Applied directly, that would suggest that the crew of a ship in clear weather should be able to see about 2 miles (land, not nautical) to the horizon from the deck or 40 miles to the horizon from the crow's nest atop a mast. These rules don't offer any kind of middle ground between those extremes, and a viewing distance of 40 miles is not realistic for an actual ship, on which the crow's nest will be at best a few tens of meters above sea level - but it is all that the rules have to say about how far characters on a ship might be able to see.
Luckily, the formula for estimating horizon distance based on vantage height is simple and, if you assume the planet you're on has earthlike dimensions, there are online calculators which will do it for you. The world of Oerth from the Greyhawk campaign setting (in which Ghosts of Saltmarsh is set) is almost exactly Earth-sized, according to the 2e sourcebook The Adventure Begins (p.9):
Careful mathematical measurements and magical divinations reveal that the circumference of Oerth is 25,200 miles. Thus, the diameter of Oerth is about 8,021.5 miles...
The actual Earth's circumference is is about 24,900 miles, so the values determined by an Earth-based calculator will be near as makes no difference to Oerth. Popular alternative campaign setting the Forgotten Realms' default planet, (Abeir-)Toril, hasn't been specified with such precision so far as I can find, but the the 2e A Grand Tour of the Realms (p.4) does state it is "Earth-sized", so Earth-based vision measurements should also be valid there.
So, using such a calculator, we can easily figure out some more realistic vision distances for your ships. Using the heights of masts given for sample ships in Ghosts of Saltmarsh Appendix A (and assuming, possibly inaccurately, that the given mast height is measured from sea level rather than the deck), we get the following vision distances:
- Galley. One 120ft mast. 13.4 miles to horizon.
- Keelboat. One 10ft mast. 3.9 miles to horizon.
- Longship. One 20ft mast. 5.5 miles to horizon.
- Sailing ship or warship. Three 80ft masts. 11 miles to horizon.
Larger ships with taller masts have an advantage in being able to spot other ships from further away. Of course, larger ships should also be easier to spot from a distance as their masts will be visible at much greater distance than the body of the ship... but observers from the top of a mast on a large ship would be able to see the body of a smaller vessel while only the tip of their mast is over the horizon, so would almost certainly spot the other ship first and could probably skirt it without being noticed.
In any event, ships that are close enough to meaningfully interact with each other would be able to see each other from their decks - barring unusual weather such as extremely heavy fog, in which case the vision range is up to the DM's determination of the weather.
Best Answer
There are some rules for this in Ghosts of Saltmarsh but they stick to non-gunpowder siege engines. Even in previous D&D versions, cannon were often left to third parties (the presence of cannon being oddly contentious amongst fantasy gamers).
I run a pirate-themed Pathfinder campaign (been going since 2009) where I faced this same issue. There's a lot of earlier ed third party rulesets for ships and naval combat, but many of them either:
I wanted something vaguely real-world realistic, though playable. If you're running an actual full time naval campaign, some detail in ship sizes, is that a 12 pounder or 9 pounder cannon, etc. are interesting and important. I'd say the Razor Coast: Fire As She Bears supplement is the best balance of these concerns if you're looking for a book (it's Pathfinder statted but should be easily convertible).
The cannon in the DMG, at 8d10 damage, would be a large usually land-based cannon, I'd pin that at a 32-pounder. Ship cannon will usually be smaller. Scale damage up and down the 3/6/9/12/24/32/48/64 pound range.
The main problem with firing cannon on a ship isn't the raw tonnage to carry the gun, it's the force from the gun ripping up your ship or tipping you over. So there's some hard limitations on what you can do.
So you can handwave as much as you want, but the semi-realistic answer for cannon on the 5e ships in the DMG is:
Galley: Due to their design galleys can't side-mount cannon (it'd both destroy the oars and tip them over). They can mount one or more quite heavy ones lengthwise however, which in calm Mediterranean-like seas is a hardcore stand-off weapon. The "list" galley could probably do a single 24-pounder. Large mature RL galleys, probably double the stats for the one in the DMG, could do a 32-pounder with a couple 12-pounders flanking it. They'd also line the sides with swivel guns (small like one-pound guns on a swivel. Think 3d6 damage with a ball or 2d6 in a cone with shot).
Keelboat: Swivel guns only, no proper cannon.
Longship: No guns, just not designed for it.
Rowboat: No guns, unless you want to swim home.
Sailing ship: This is usually code for "you know, a small caravel like you done seen on the TV," Nina/Pinta/Santa Maria style. Probably a 60 foot long one, given that 100 tonnage value. Guns up on the top deck. Realistically, about 4 12-pounders per side max, maybe a 9 pound chase gun.
Warship: Usually code for a carrack or galleon. This is where you get real gun decks going on (which allow for heavier guns, since they're farther down on the center of gravity). "As many as you want," based on the size of the ship. The Mary Rose, a 700 ton carrack (more than 100 feet long), had a dozen 42-pounders and a small complement of 32 and 18 pounders. Just scale up per 100 tons from the caravel.
I find this in general hits a sweet spot of dangerous but not too dangerous (cannon get a lot of penalties to hit generally). Of course it's part of a carefully balanced set of naval combat rules and such.
Now for other siege engines - ballistae don't have the same recoil problems, they're just kinda big. I'd say you could get about 1/3 more ballistae than you could cannon on a given ship. But same thing with the size class - a small boat might have a heavy crossbow but not really a ballista; maybe one could legitimately fit on a keelboat. Catapults can't really be used if you have sails and rigging, they went out with the galleys.