Both sentences are correct to my (native Canadian English) eyes. I do perceive a subtle difference in meaning between the two, however: using the perfective "have" strengthens the sense in which the author was not able to finish his novel. That is, I see these two meanings:
The author died before he could finish his novel. He was working on his novel, and before he got around to finishing it, he died. Maybe if he worked harder he would have finished it, or maybe it was just impossible.
The author died before he could have finished his novel. He was working hard on his novel, but he died so soon that it was impossible for him to finish the novel. Even if he was putting his full effort in, he died before the possibility itself existed.
That is, adding the perfective puts an emphasis on the impossibility of finishing the novel, where as without it one leaves unspecified whether, given different circumstances, the author might have been able to finish the novel despite dying so soon.
Both of these mean much the same thing. It's really just two ways to ask the same question. The first asks whether a particular action is complete, while the second asks whether a particular condition is true.
A similar example
Have you finished eating?
Are you finished eating?
In much the same way (with slightly different grammar)
Have the children come home from school?
Are the children home from school?
Best Answer
Both are correct.
have finished is present perfect of the verb finish. Present perfect is used when an action completed in the past has an effect that lasts to the present. This version focuses on something that happened in the past, and its effect on the current situation.
In am finished, finished is an adjective: am describes the situation now, without any regard to what happened in the past.
This NGram shows that there is a growing difference between British and American usage.
have finished is the older and possibly more formal form, with am finished only starting to appear in about 1900. In the UK, the new form had a limited life and is now in decline: in the US, however, it has grown steadily in popularity and has now overtaken have finished.
I am British English and could easily imagine my grandmother saying am finished, but I would prefer to use have finished.