Carpet Asbestos – What Is This Yellow Powder Under My Carpet?

Asbestos in building materials is generally only a concern where the material is friable i.e. brittle and crumbly. You haven't said, but if the "panel" was removed as a unit, then almost certainly very little asbestos was released. If, on the other hand, the panel were pulverized and destroyed to the point where a lot of dust was created, that may be cause for more concern.

But, there is little you can do about it now. Also, in my opinion, there is very little to be worried about. I believe much of the fear over asbestos has much more to do with fear over litigation and large damages awards than it does over actual danger. The fact is, people who contract mesothelioma are generally those who had continuous exposure to high concentrations of asbestos fibers over many years. Just as inhaling a bit of coal dust is not likely to give you black lung, the odd asbestos fiber here and there is not likely to kill you.

Any fibers under the carpet (and carpet pad I assume) are effectively encapsulated and unlikely to move through the pad and carpet into the atmosphere. Fiber concentration in the air will decrease over time with air changes and with settling out. After 2 years, I would be surprised if you had any measurable asbestos fibers in the air.

Lastly, you said that the panel "might" contain asbestos. If you don't know and cannot test it or another piece of the same panel, then you will never know if there is asbestos and you should not, in my opinion, live in fear. If you can have the piece tested and it does contain asbestos, bear in mind that in some jurisdictions you will have to disclose that fact on sale of the home.

EDIT: Humorous (and likely accurate) answer from prior similar question: "You are far more likely to die from worrying about asbestos than from asbestos." That answer here:

How can I seal the edges of a linoleum floor that possibly contains asbestos?

Helpful link to a Reader's Digest article also from that answer: http://fumento.com/asbestos/asbest.html

Looks like effloresence to me, which itself is not harmful, but possibly a symptom, mainly of water movement; I would check perimeter walls for further signs of water infiltration, and make sure water movement outside is properly being handled, i.e. gutters, grading.