The descriptions of the hag eye and soul bag in the MM specifically reference the hags that created the items when talking about who uses the items.
Hag eye:
A hag in the coven can take an action to see what the hag eye sees if the hag eye is on the same plane of existence.
Soul bag:
A soul bag can hold only one evil soul at a time, and only the night hag who crafted the bag can catch a soul with it.
So those items are clearly limited to the hags. But the heartstone, I think, is somewhat open to interpretation.
Heartstone:
This lustrous black gem allows a night hag to become ethereal while it is in her possession. The touch of a heartstone also cures any disease.
From this, it looks like etherealness is only usable by hags, but curing diseases could theoretically work for all.
Jeremy Crawford has commented on this saying it's meant to work only for hags, but DMs can rule otherwise.
If you're looking for a way to use the stone for curing diseases, I'd refer to the 3.5e Wiki, where a heartstone is described as follows:
All night hags carry a periapt known as a heartstone, which instantly cures any disease contracted by the holder. In addition, a heartstone provides a +2 resistance bonus on all saving throws (this bonus is included in the statistics block). A night hag that loses this charm can no longer use etherealness until it can manufacture another (which takes one month). Creatures other than the hag can benefit from the heartstone’s powers, but the periapt shatters after ten uses (any disease cured or saving throw affected counts as a use) and it does not bestow etherealness to a bearer that is not a night hag. If sold, an intact heartstone brings 1,800 gp.
This would be fairly reasonable to use in 5e as well. Naturally, the specifics are down to you as DM. For example, since the saving throws buff is not an official function of the 5e heartstone (according to the MM), you might want to remove that ability from it and adjust the sale price accordingly.
By RAW, no.
The Use Magic Device feature lists the specific attunement requirements that it circumvents, and, as the question states, "spellcaster" isn't one of them. There are other attunement requirements that Use Magic Device doesn't get around — for example, alignment restrictions exist for a Sword of Answering or a Talisman of Pure Good (or Evil).
If the intent was for Use Magic Device to get around all attunement restrictions, then it would say so; we can infer that the intent is that some such restrictions cannot be avoided.
It's possible that the intent of the designers was to allow rogues with Use Magic Device to be able to use the (mostly) wands with the 'spellcaster' restriction, but so far the wording of the rules doesn't support that.
But probably by RAI, yes.
In this Twitter post from October 2016, Jeremy Crawford was asked about how a Wand of Paralysis is used by a rogue, and he doesn't say that it can't be done. But the applicability of Use Magic Device to the spellcaster attunement requirement wasn't the subject of the question, either.
However, this doesn't seem to be a critical game balance issue, and given that Use Magic Device is intended to allow rogues to use spell scrolls, it doesn't seem unreasonable to extend it to the "spellcaster" items as well.
Best Answer
Per Attunement of the 5e DMG (page 136, emphasis mine),
So, by default any character can attune to any item that requires attunement. If the item has a prerequisite, then only those who meet the prerequisite can attune to it.