We have setup Google login via OAuth for our Salesforce (Not SAML) and authentication via our custom domain page works fine. But as soon as we try to restrict our users to login only from our domain page (In Setup->Domain Management->My Domain->Login Policy: Prevent login from https://login.salesforce.com), this method for authentication fails when initiated from our custom domain, with the following error:
Problem Logging In
We can’t log you in because of an authentication error. For help, contact your Salesforce administrator.
When I go and visit the login history I notice the following entry associated with this failed login attempt:
Login Type: Third Party SSO
Status: Restricted Domain
As soon as I loosen that restriction, login via Google starts working again.
I've followed this guide to setup SSO: https://help.salesforce.com/articleView?id=sso_provider_google.htm&type=5
I've used this code for my registration handler: https://github.com/trineo/sfdc-rego-handler/blob/master/src/classes/GoogleAppsRegistrationHandler.cls
Best Answer
This looks like to be an issue which was fixed with Summer '17, that may have resurfaced. Refer to this known issue here.
You may like to reach out to Salesforce support to verify and get details on this, if that's the case.
Update: The issue specifically seems to be in the SSO flow as if you login using the my domain url and use your salesforce credentials, it works fine even if "prevent login.." is checked. This could possibly be part of the issue that was fixed with Summer '17 only for my domain urls but not the SSO flow.