The different endpoints are used for different authentication flows, this is all covered in the REST API documentation.

The /authorize endpoint is used for the Web Server OAuth Authentication Flow and User-Agent OAuth Authentication Flow.

The /token endpoint is used for the Username-Password OAuth Authentication Flow and the OAuth Refresh Token Process.

Web Server OAuth Authentication Flow

The Web server authentication flow is used by applications that are hosted on a secure server. A critical aspect of the Web server flow is that the server must be able to protect the consumer secret.

In this flow, the client application requests the authorization server to redirect the user to another web server or resource that authorizes the user and sends the application an authorization code. The application uses the authorization code to request an access token. The following shows the steps for this flow.

User-Agent OAuth Authentication Flow

The user-agent authentication flow is used by client applications (consumers) residing in the user’s device. This could be implemented in a browser using a scripting language such as JavaScript, or from a mobile device or a desktop application. These consumers cannot keep the client secret confidential.

In this flow, the client application requests the authorization server to redirect the user to another Web server or resource which is capable of extracting the access token and passing it back to the application. The following shows the steps for this flow.

Username-Password OAuth Authentication Flow

The username-password authentication flow can be used to authenticate when the consumer already has the user’s credentials.

In this flow, the user’s credentials are used by the application to request an access token as shown in the following steps.

Warning
This OAuth authentication flow involves passing the user’s credentials back and forth. Use this authentication flow only when necessary. No refresh token will be issued.

OAuth Refresh Token Process

The Web server OAuth authentication flow and user-agent flow both provide a refresh token that can be used to obtain a new access token.

Access tokens have a limited lifetime specified by the session timeout in Salesforce. If an application uses an expired access token, a “Session expired or invalid” error is returned. If the application is using the Web server or user-agent OAuth authentication flows, a refresh token may be provided during authorization that can be used to get a new access token.

This was a policy issue related to Permitted Users under the Connected App, once I changed it from 'All users may self-authorize' to 'Admin approved users are pre-authorized' issue was resolved.