Should a washer go on nut side, or the bolt side

Most washers will flex somewhat, so unless it's a really small pipe, or you're installing in really cold temperatures, you shouldn't worry ... but I'd be more concerned with being able to hold the nut still while you're trying to bolt to it in a smaller diameter pipe. (I don't know if the 1.5" is ID or OD, but if it's outer diameter, and it's got any significant wall thickness, it's going to be tight)

If you're really concerned, you should be able to find a rubber gasket to put in there to take up some of the gap; you could also cut down a rubber grommet or bushing to fit in there. You could also use multiple thinner washers which will flex more easily than a single thick one.

Another option would be to take a hole saw that's got a similar inner diameter to the pipe, cut a plug of wood, then slice a chunk off to make the flat-on-one-side-curved-on-the-other piece you're looking for. (and then use a fender washer or similar so the nut doesn't dig into the wood).

After cutting the bolt to length, use a stationary grinder instead of a file to clean up your work. With a grinder, it's easy to square up the end of the bolt and apply the chamfer that you want.

Just be careful, especially if the bolt is shorter than the grinder's table. If the bolt is short, use a pair of vice grips to hold the bolt during grinding. And remember that the end of the bolt is going to be very hot after grinding!