The NEC wants you to avoid overfilling a conduit so that the conductors can dissipate heat, and so you can remove a conductor without damaging it. See NEC 300.17.
There are only 9 single pole circuits (so 18 conductors?)
Ground conductors count for equipment fill, so you have more than 18 -- maybe 23?
Ironically, when you get close to a fill limit, the NEC actually wants you to increase conductor size, because that reduces heat.
Should I be worried or just put the cable through it anyways?
If this was my house, I would want to add a new conduit, if only because pulling the new conductor will be so much easier.
Do they make "reducing" hubs?
Yes. Here are some examples: http://www.thomasbetts.com/ps/endeca/index.cgi?a=nav&N=3819+598+4294951140&Ntt=
I've always found I needed 6" of conduit to make the transition between the reducer and the male fitting that connects to the box. Luckily my local hardware store will sell PVC conduit by the foot, so I don't have to buy a full stick for this.
As @Tester101 mentions, you can also make the hole bigger. A step bit is a nice tool, but 1 1/4" conduit requires a knockout that's actually ~1 3/4". The one he linked to only goes to 1 3/8", and I haven't seen bigger.
The professional way to make a larger knockout is to use a punch. They're a little expensive. Maybe you can find an electrician who will punch the hole for you in exchange for beer or a favor. Or maybe a tool rental place will have a bunch for you.
You don't have to bring the conduit in through an existing knockout - you could punch a hole on the side in a blank area, if that's more convenient for you. Alternately, if there's a good spot with 2 small knockouts, you can consume that whole area.
In your existing installation, I think the lock washer is upside-down. There are sharp edges that are intended to bit in to the metal, so it won't come loose. Unfortunately fixing this means undoing a lot of wiring.
Also, I think you should use a bushing to protect the conductor from the end of the conduit. Whether your inspector will require it is difficult to guess, but it's seems like cheap insurance. Something like this:
Also, I hope that if you're going through the trouble of pulling a big conductor to a new location, you're installing a subpanel at the other end. Because subpanels are awesome.
For conduit fill calculations you have to treat romex/NM as a ROUND wire of the same size as its largest dimension (ie, the long axis of the oval.) Oddly enough this means that 14-2 is larger than 14-3 for conduit fill. As @SpeedyPetey notes nipple (less than 24") fill is 60%, not 40%, and is a LOT of wire to get into a conduit; but I don't think that's the major issue here if NEC compliance is a desired upgrade. The wire type and location is the main issue.
NM cannot be run in exterior/underground conduit since it is not rated for wet locations. With an outside service box that alone might mean you can't do this and meet code - at which point you'll need a major junction box or a lot of minor junction boxes inside, and some wet-location-rated wire until you are inside.
(And now that I've seen the pictures...)
My understanding of NEC (which does not seem to have been particularly involved in your current service, with exposed NM running outside) is that you would have to run a wet-rated wire (THW, XHHW, etc.) until you get inside the house, where you would connect to the interior type wire you currently have exposed to weather and sunlight. NEC defines (and reality frequently confirms) all exterior conduit as a wet location.
Best Answer
On a seemingly unrelated topic, I like to see houses have LOTS of spare breaker panel spaces. Spaces are cheap, and in the future stuff always comes up, and nobody has ever said "Oh no, I have plenty of breaker spaces available, whatever will I do?"
Hold that thought.
Now, given the thickness of the cable, I do not see way to make that bend. What I advise doing is installing a LARGE junction box on the inside of the garage wall. Now your SE-R cable can enter that box vertically, transition into individual wires, and go through a conduit nipple from this inside box to the outside box. At that point we're only concerned with radius of individual wires, and that's no problem.
Now let's price that. Splices of large wires require very long, tall boxes. We also need splice connectors for 1/0 wire (four of them)... Hold on a minute. What's a LARGE box that comes with big lug connectors, that's cheap?
Why a subpanel, of course!
So if extra breaker spaces wouldn't be useless, the best answer may be to fit a large subpanel on the interior of the garage. "Large" because you want full width, and enough height to handle the large wire bends. A 14" wide psubpane gives you
And that's exactly what I'd do. Select a 200A-bussed, main-lug but convertible subpanel of 20+ actual spaces (not circuits).
I'd arrange the new panel back-to-back with the existing panel, using a "nipple" to connect them that is as large as the knockouts allow.
Downhill from there.
If the SE-R cable's internal conductors have their own markings (as XHHW conductors) then you can enter the new subpanel with a grommet/cable clamp, then pass right through the new sub and into the original panel - at that point you're just using the sub as a giant junction box.