It is acceptable to run Nonmetallic sheathed cable in conduit, in fact code calls for it in some situations.
National Electrical Code 2014
Chapter 3 Wiring Methods and Materials
Article 334 Nonmetallic-Sheathed Cable: Types NM, NMC, and NMS
II. Installation
334.15 Exposed Work.
(B) Protection from Physical Damage. Cable shall be protected from physical damage where necessary by rigid metal conduit, intermediate metal conduit, electrical metallic tubing, Schedule 80 PVC conduit, or other approved means...
Conduit fill calculations become a bit more difficult when dealing with cables, but a few notes in chapter 9 provide some direction.
National Electrical Code 2014
Chapter 9 Tables
Notes to Tables
(5) For conductors not included in Chapter 9, such as multiconductor cables and optical fiber cables, the actual dimensions shall be used.
(9) A multiconductor cable or flexible cord of two or more conductors shall be treated as a single conductor for calculating percentage conduit fill area. For cables that have elliptical cross sections, the cross-sectional area calculation shall be based on using the major diameter of the ellipse as a circle diameter.
Note 5 says that you'll have to use the actual dimensions of the cable, to determine if it will fit in the conduit. While note 9 says that you can think of the cable as a single conductor, which means if it's the only thing in the conduit you can use the 1 wire fill percentage of 53%.
Calculate area of NM cable
You should be able to find a spec sheet from the manufacturer of the cable you choose. I found the spec sheet for Southwire’s Romex® SIMpull ® Type NM-B, which lists the sizes of various cables. It lists 6/3 cable as having a diameter of 650 mils, or 0.650". From which you can calculate the cross-sectional area, using the formula πr².
(0.650 / 2)^2 * pi =
(0.325)^2 * pi =
0.105625 * pi =
0.3318307240354219108126167073589 in.²
Conduit Fill
Once you know the area of the cable, you can use Table 4 from Chapter 9 of the National Electrical Code to determine the size of conduit needed. Remember because of Note 9, you can use the 1 wire column (53%).
Electrical Metallic Tubing (EMT)
1/2" @ 53% fill = 0.161 in.²
3/4" @ 53% fill = 0.283 in.²
1" @ 53% fill = 0.458 in.²
1/2" @ 40% fill = 0.122 in.²
3/4" @ 40% fill = 0.213 in.²
1" @ 40% fill = 0.346 in.²
Intermediate Metal Conduit (IMC)
1/2" @ 53% fill = 0.181 in.²
3/4" @ 53% fill = 0.311 in.²
1" @ 53% fill = 0.508 in.²
1/2" @ 40% fill = 0.137 in.²
3/4" @ 40% fill = 0.235 in.²
1" @ 40% fill = 0.384 in.²
Rigid Metal Conduit (RMC)
1/2" @ 53% fill = 0.166 in.²
3/4" @ 53% fill = 0.291 in.²
1" @ 53% fill = 0.470 in.²
1/2" @ 40% fill = 0.125 in.²
3/4" @ 40% fill = 0.220 in.²
1" @ 40% fill = 0.355 in.²
Rigid PVC Conduit (PVC), Schedule 80
1/2" @ 53% fill = 0.115 in.²
3/4" @ 53% fill = 0.217 in.²
1" @ 53% fill = 0.365 in.²
1/2" @ 40% fill = 0.087 in.²
3/4" @ 40% fill = 0.164 in.²
1" @ 40% fill = 0.275 in.²
In this example, you'd need 1" conduit to fit the 6/3 nonmetallic sheathed cable.
The other option. Is to run 6/3 NM cable in the basement, then switch to three 6 AWG THHN conductors through conduit in the garage. According to Table 5 in Chapter 9 of the NEC, 6 AWG THHN has an area of 0.0507 in.². So the three current carrying conductors would have a total area of just 0.1521 in.².
0.0507 in.² * 3 = 0.1521 in.²
According to Note 8 in Chapter 9, you can use the area listed in Table 8 for bare conductors. So if you install a bare grounding conductor in the conduit, you can use this value instead of the insulated conductor size.
National Electrical Code 2014
Chapter 9 Tables
Notes to Tables
(8) Where bare conductors are permitted by other sections of this Code, the dimensions for bare conductors in Table 8 shall be permitted.
Which means you'll add an additional 0.027 in.², for a grand total of 0.1791 in.².
0.1521 in.² + 0.027 in.² = 0.1791 in.²
Since there will be more than 2 wires in the conduit, you'll have to use the 40% fill column. Even with area reduction, you'll still be able to use 3/4" conduit instead of 1".
Best Answer
To add a few things to George's excellent answer:
I would simply place the sub-panel in the shop
This is a key. Because the garage is attached, you can run each circuit as a home run back to the subpanel-next-to-main-panel. But putting the subpanel in the garage has some real advantages:
If you actually want a subpanel next to the main panel because you need more circuits for the rest of the house, add two subpanels - one in the house and one in the garage. Panels are actually pretty cheap - you can get a 20 space (that's spaces, not "circuits") for $100 or less.
Also keep in mind that the wire size has to match the feed breaker. You could have a 42 space subpanel with a 200A breaker if you get a good deal on one. But if you only need 90A (which happens to fit with some good aluminum wire prices) and you use a 90A breaker in the main panel then you don't need to use expensive 200A-capable feeder.
Most importantly, for any circuit (feeder or branch), pick either cable or conduit with individual wires. Never use cable with conduit - it costs more (you're paying for the cable sheath and for the conduit) and running cable through conduit requires larger conduit than the equivalent individual wires require.