So, after a few hours of debugging and fruitless research, I realized that: a: My conjecture in Edit 2 about text format was correct, and b: it can be solved by including the other text json like so (Please pardon any errors in my json format):

execute at @a[nbt={Inventory:[{id:"minecraft:blaze_rod",Count:1b,tag:{display:{Name:"{\"text\":\"Doomstick\",\"color\":\"dark_red\",\"bold\":\"true\",\"italic\":\"true\"}"}}}]}] run say hi

Hopefully this helped anyone out there experiencing a similar issue.

This is the command that ended up working for the asker:

execute as @a at @s positioned ^ ^ ^2 anchored eyes run execute as @e[distance=..1,limit=1] run effect give @s glowing 1 1 true

This command can be shortened to look like this without affecting the outcome:

execute at @a positioned ^ ^ ^2 anchored eyes run effect give @e[distance=..1,limit=1] glowing 1 1 true

This command applies the glowing effect to one entity within a sphere with 1 block radius. The sphere is centered 2 blocks in front of the position of a given player at eye level. The command is executed from the position of every player.

This can also be done with a datapack:

Using a datapack you can put an effect on an entity that you look at, even if they are very far away. This function is called glow and uses the name space look_at:

execute unless entity @e[type=!player,distance=..1] positioned ^ ^ ^0.2 if entity @s[distance=..50] if block ~ ~ ~ air run function look_at:glow
execute unless entity @e[type=!player,distance=..1] positioned ^ ^ ^0.2 if entity @s[distance=..50] if block ~ ~ ~ cave_air run function look_at:glow

effect give @e[type=!player,distance=..1] minecraft:glowing 1 1

The first two commands in this function call this same function again, just 0.2 blocks further ahead, unless a non player entity is close enough to be detected, and only if the player is within 50 blocks from the current position.

The first command executes when you look through air and the second command when you try to look through cave air (cave air is exactly like air, but it's not considered to be the same block by the game). This prevents that the glow effect can be applied to an entity behind a wall, because the function will stop once it hits a wall (it may be possible to look at the edge of a block and still look "through" it, this can be prevented by using smaller steps.

The third command simply gives the glowing effect to any non player entity near the position that's currently being checked.

This function should be called repeatedly with a command like this:

/execute as @a at @s positioned ~ ~1.5 ~ run function look_at:glow

This command calls the function as every player at the position of their eyes (1.5 blocks above their feet).

A simpler to imagine explaination on how this works

Imagine walking in a straight line, you take a small step and check if there is an entity nearby. You take another step and check again if there is an entity nearby. You continue walking like this until you reach any block that isn't air, find an entity, or walked 50 blocks away from where you started.