Basement – Do I leave this insulation up

Yes, Rigid foam (XPS and ISO) boards are frequently used in exterior applications, and can be a great option for improving efficiency. There are a couple of issues with your proposed approach, though:

1. Vapor barrier location. The vapor barrier should be on the warm side (probably inside, unless you live in a very hot & humid climate). Ideally, this is immediately behind your interior drywall. You also do not usually need an extra vapor barrier layer adjacent to rigid foam sheets, they are typically highly resistant to moisture.
2. Attachment. You cannot rely on glue to support foam and siding panels. These need to be mechanically fastened.

On a wood frame house, you'd want to use this order:

• interior drywall
• possibly a vapor barrier (e.g. poly sheeting), though the wisdom of this is debatable (see the comment from iLikeDirt below)
• wall studs (optionally with batt or cellulose insulation in between)
• exterior sheathing
• rigid foam
• wooden strapping for attaching siding, screwed to studs through foam. This provides support for the siding and also allows air circulation behind the siding, so any moisture that gets behind the siding can dry out.
• siding panels, nailed to strapping

Here's a visual from Fine Homebuilding, which has a couple of good articles on this:

Note that adding foam on the exterior of your house complicates some things. You will likely need to redo the trim on your windows and doors, which will become more in-set. These inset areas will need to be flashed properly or you will have leaks, which have more potential to do damage when you've got multi-layered, thick walls that could trap moisture. If done well, you can get very good insulation. But make sure to work with someone who knows what they're doing!

I think based on the comments I better understand your scenario now.

Building Science has a lot of great information on basement insulation and the suggested method is to use XPS foam against the walls, optionally bats between the studs and no vapor barrier (since the XPS foam is the vapor barrier).

I would suggest that you use 1 1/2 - 2" XPS foam against the walls, and then space your studs another inch or so away from the wall (to allow your water to drain). If you need higher R values then the XPS provides you can use bats between the studs and I would suggest something like Roxul insulation which is stone wool instead of fiberglass since it is not affected by moisture like fiberglass is.

It would also be advisable to use steel studs so that they don't rot should they come in contact with moisture. You might also opt to put the bottom plate on top of a piece of XPS foam so that there is a less likelihood of the metal studs rusting.