You should consider a few conditions when deciding what is the most efficient (often cheapest) method for heating your home.
The Department of Energy site has some good resources about regional heating fuel costs.
Their heating fuel comparison calculator is a useful example:
http://www.eia.doe.gov/ask/conversionequivalents_faqs.asp#compare_heating_fuels
Consider installation costs as well as operating costs. There are obviously some pretty efficient heating techniques (geothermal and solar) that unfortunately, still have extremely high installation costs. This often makes their installation untenable.
Geothermal Heat Pump
Efficiency
If upfront cost is of no concern, a Geothermal heat pump is the most efficient heating/cooling system. According to the Department of Energy, geothermal heat pumps can be between 300 - 600% efficient.
Financing
There may be government programs available to help fiance the installation of the system, as well as state programs.
Payback
Some sources estimate a payback period as short as 3 years, though actual results may vary. If you switch your main house over as well, use the system for hot water, and take advantage of government programs. The payback period could actually be quite short.
Electric Baseboard Heaters
Efficiency
Since electric heat is 100% efficient, every ounce of electricity used it converted to heat.
Financing
If there is adequate space and power available in the service panel, an electric baseboard heating system is fairly inexpensive to install.
Payback
Depending on the cost of electricity in your area, electric heat may or may not save you money over other heating options.
Natural Gas Room Heater
Efficiency
Since it's a smaller area, a standalone natural gas forced air system may not make sense. However, a smaller natural gas room heater (or a few) might make sense. Natural gas room heaters can be between 65 - 100% efficient (depending on the heater), so they can be quite efficient.
Financing
As with electric room heaters, natural gas room heaters will be fairly cheap to install.
Payback
Again, depending on the cost of fuel. A natural gas room heater may or may not save you money over other heating options.
More Info:
The U.S. Energy Information Administration has a Heating Fuel Comparison Calculator (Xls) that might be useful.
Best Answer
Electrical baseboard heaters and plug-in resistance heaters are already 100% efficient, in that 100% of the electrical energy that goes into them is turned into heat. Whether it's a ceramic heater, oil-filled radiator, or radiant heater, they all are 100% efficient.
However, as you found out, even at 100% efficiency, electrical resistance heating is expensive.
This is why many people that need to use electrical heating will use a heat pump, which can actually be more than 100% efficient because instead of creating heat directly from the electricity, they pump heat from outside the house to inside the house. But this is not a simple retrofit, so you can't simply replace your baseboard heaters with a central heat pump system (though if you live in a relatively mild climate, you can find window unit heatpumps that might be more efficient than a baseboard heater)
One thing that might help would be to turn down the baseboard heaters, and use radiant heaters in places where you want targeted warmth. For example, if when you're in the livingroom you spend most of the time on the couch, you could turn down the thermostat in that room and use a radiant heater in front of the couch (but a safe distance away) to keep you warm even if the rest of the room is not comfortably warm.