Minecraft – /execute a command at players with a certain score

It's because it couldn't find any entity with the direction value between 3 and 4. I tested this command with the executed command changed:

/execute @p[score_direction=4,score_direction_min=3] ~ ~ ~ say FOUND

The executing of the command was succesful after running these 2 commands:

/scoreboard objectives add direction dummy
/scoreboard players set rudygreg direction 3

You cannot use tilde (~) as part of a selector argument, unless you're using Bedrock Edition. Since you didn't state that you were using BE, I'm going to assume you're using the standard Java edition.

Your only real option in 1.12 is to use the actual coordinates that you're trying to point to, such as

execute @p[x=40,y=80,z=120,r=5] ~ ~ ~ spawnpoint @s ~3 ~3 ~-5

Or you can shift the execution point with execute, then you can even simplify the rest of the command:

execute @s ~5 ~2 ~17 spawnpoint @p[r=5] ~3 ~3 ~-5

Also notice that I'm using the @s selector for the first spawnpoint command. This will target the player found in the execute portion.
The same applies to the second execute: You want to execute relative to the player who executes the command, so you can use @s.