Currently, you are able to gain ~125% of the initial value of the given science. Each time you perform science, you reduce the value you will receive in the future. Regardless of how you acquire it, you will always end up maxing out at ~125% the initial value for a particular area and method of doing science. It doesn't matter if you transmit your science or return it to Kerbin. The actual maximum is dependant on the type of science, and varies between 110% and 180% of the initial value.
Also, note that I mentioned the area that science is performed. There's a difference between the surface of Kerbin, the upper atmosphere of Kerbin, and the orbit of the Mun, and each of the places, along with others have their own maximums of science that can be performed. Also note that the science you get from these different areas can be scaled differently. The KSP wiki article on science explains all of these parameters (or in some cases, at least tries to) much better than I could in this limited space.
Now, there is good reason for choosing transmission over return, or vice versa. Early in the career, you don't have access to RTG's, solar panels, or even batteries. This makes transmitting science impractical at best, as you need electricity from your engines in order to transmit again, so you need to return your science. Conversely, if you're sending a robotic mission to Jool, it's impractical return a dozen Science Jr. pods and Mystery Goo containers if you want to gather science from as many of the moons as possible. You're better off sticking a bunch of solar panels on it and transmitting often. As of the 0.23 update, you also have to weigh the effect of the transmission cap on science: you'll only ever get 40% of the maximum science from transmission alone; and without a lab attached, some of the experiments are single use only. Sooner or later, you'll need to return the samples.
Basically, if the cost (in terms of electricity) isn't too high to transmit and you're able to reacquire the same science easily, transmit it, but always hold on to the last bit of science you have if you're returning the craft to Kerbin.
Basically, prefer return over transmission due to transmission cap and single use experiments, but always include the possibility of transmission.
You're right, that setup wont get you to Gilly and back, for a variety of reasons.
Here are four ways you can improve your game and make for yourself a rocket that will perform that mission. I'm assuming, since you're playing KSP in the first place, that you want the achievement of having made and flown the entire thing, not just downloaded someone's .craft file!
- Use Kerbal Engineer. It has no gameplay effects or flight computer, it just does a lot of calculations for you, including dV. Click here to get it from Kerbal Space Port.
- Do research and plan your missions. Real scientists do in-depth analyses to determine the dV needed to get to places, and you can to! Fortunately, the good folks at the Kerbal Space Program Wiki did most of that for you. Now, if only you had some sort of part that you could put on your ship to check that you have enough dV... cough cough.
- Watch tutorials. Everyone can brush up on their fundamentals and their advanced technique. A great place to start would be Scott Manley's tutorial series. Of all of the very best players, he has the most helpful and up-to-date tutorials, and ties in a ton of real-world rocket science that you might not have picked up on your own!
- Lastly, and most importantly, practice! You get better at games by playing them; even tough, physics-based games. Try new things. Maybe you need more fuel. Maybe you need a smaller payload. Maybe you need a different sort of staging or fuel arrangement. Personally, I'm a big fan of Asparagus staging for lifters and Caterpillar staging for transfer vehicles.
*
Caterpillar Staging is when you have engines that aren't jettisoned with the fuel tanks, as pictured below. (The parts with the diagonal lines are meant to represent the 2m Rockomax decouplers.) Just like Asparagus Staging, Caterpillar Staging is meant to drop the dry weight of empty tanks and therefore keep the rest of the craft light and long range.
Best Answer
If you want the science from experiments, you must either transmit it, or return the experiments to the surface of Kerbin. The experimental equipment itself doesn't actually have to make it back to Kerbin though; A kerbonaut can retrieve the data and store it in a command pod that's headed back to Kerbin. There are some restrictions to this, though: you can't store multiple copies of the same experiment/biome/altitude combination, and I haven't seen a way of removing research data from a mobile lab. As a side note, you can store multiple crew reports by performing an EVA, taking the data from the command module, and then returning to the module.
Your experiments also don't have to be attached to the capsule when you return ‒ you can recover experiments returned as debris by disconnecting a science stage from a capsule; however this may be impractical due to limitations of the game where after a certain threshold, debris (or even whole aircraft/spacecraft) will be unloaded when in atmosphere, but it does mean that you don't need to perform an EVA to retrieve the data.
Some science experiments also have a transmission cap where as little as 25% of the data can be transmitted back to Kerbin; eventually, experiments must be returned to max out available science. However, when return is non-trivial, taking the hit in transmission might be worth it (for instance, if the target is the surface of Tylo or Eve). Take a look at the science wiki page for more information.