Everything on your character sheet that you have not explicitly dropped is considered "worn or carried".
We can look to the Arcane Trickster rogue's Mage Hand Legerdemain feature for a little bit of guidance on this:
You can retrieve one object from a container worn or carried by another creature.
Considering the intent of this feature (mage hand gaining the ability to steal items from people), anything a character has should be considered worn or carried. Because otherwise, the regular mage hand would be able to steal from any such containers that were not considered worn or carried.
Under the "Strength" section of chapter 7 in the PHB, we also find this under the Lifting and Carrying rules:
Carrying Capacity. Your carrying capacity is your Strength score multiplied by 15. This is the weight (in pounds) that you can carry, which is high enough that most characters don’t usually have to worry about it.
This claims that everything contributing to a character's encumbrance is "carried" – which explicitly exempts everything that has a listed weight, and implicitly exempts everything without from being ignited by standard fire damage.
This is a simplification in 5e: earlier editions found that subjecting worn/carried objects to burning from spells would either bog down the game with too many rolls or make fire magic too powerful with respect to other damage types. Plus, players hated it.
No, you can not choose a swarm as an animal companion
The simplest argument is that the Ranger must choose a beast, singular, not a swarm of beasts, plural. Rules designer Jeremy Crawford even supports this interpretation in an unofficial tweet from January 2016 (although the tweet is responding to a question to the druid's Wild Shape feature).
In addition to Jeremy Crawford's tweet, there is now an official ruling in the Sage Advice Compendium (as of October 2020) that says a swarm is not an option for the conjure animals spell:
Can conjure animals summon a swarm?
No. Conjure animals summons individual creatures, and swarms are groups of creatures.
The spell conjure animals refers to being able to summon a specified number of beasts, and the Wild Shape feature lets a druid turn into a single beast, just as a Beast Master gains a single beast as an animal companion. As such, the similar logic would suggest that the same restrictions apply.
Natural vs. unnatural
There are also additional arguments to be made against a swarm being a valid option. For example, consider the description of the Beast type, in the introduction to the Monster Manual (p. 6)
Beasts are nonhumanoid creatures that are a natural part of the
fantasy ecology.
And then look at the sidebar titled "The Nature of Swarms" in the section listing various types of swarms (Monster Manual, p. 377; emphasis mine):
The swarms presented here aren't ordinary or benign assemblies of
little creatures. They form as a result of some sinister or
unwholesome influence.
A creature with the beast type is a natural creature. A swarm is something that does not occur naturally.
Buyer beware
Of course, your DM may choose to allow you to choose a swarm. However, be wary of making such a choice. Each swarm, including the Swarm of Ravens, includes the following text in its Swarm trait:
The swarm can't regain hit points or gain temporary hit points.
Not being able to heal your swarm is likely going to cause you issues, especially given the relative squishiness of the animal companion in general.
Best Answer
Just let the swarm turn invisible when the ranger is invisible.
We’re probably getting too rules-focused if we try to answer this based on the swarm being worn or carried. Let’s just make a ruling that doesn’t nerf the ranger’s 10th level feature, that is, just let the swarm turn invisible when the ranger is invisible.
This is supported by the Gathered Swarm feature description:
Ergo,
It’s up to the ranger what their swarm looks like anyway. “You can never truly be invisible even though your class features say they do that” is code for “don’t play this subclass”. Just let invisibility work. Pick whatever narrative reason you want. They’re “intangible spirits”, after all.