Electrical – How to connect these two yellow wires

Those wire colors (black, red, green and yellow) are commonly found in the wiring for security systems; smoke detectors that hardwire into security systems typically run on 12VDC or 24VDC, unlike the 120VAC detectors that you generally find at home improvement stores. I'm guessing the wires are stranded instead of solid core (like mains wiring) and of a smaller gauge wire. Another clue is that there's a small hole in the ceiling; if it was a 120V detector, there should be a ceiling box, and the wires would be colored black and white, possibly with a third, red, insulated wire, and probably a bare copper wire for ground.

Since this smoke detector isn't working, the security system may have been disconnected or removed. Do you have other smoke detectors in the house, and do they work? What about other security system components like motion detectors, glassbreaks, the control panel and/or keyboard?

Use a multimeter to check that there's no voltage between any of the wires; the supply for the detector was probably on the red and black, but check all the combinations to be sure. I would seal the hole with some expanding foam to prevent air leaks into the ceiling, then wrap each exposed wire individually with electrical tape and just leave them behind the base plate for the new detector.

Comment converted to answer

A 16/14 cap should take between 1 and 4 #16 wires, but it depends on the cap. Specific information should be available on the packaging, or from the manufacturer.