In a medium bowl, combine coffee, Rum Chata and vanilla until well blended. Refrigerate until ready to serve.
Spoon cookie butter into a small resealable freezer bag. Seal, removing as much air as possible. Cut one tip from a corner of the bag, then pipe about two teaspoons into the bottom of each chocolate cup.
Pour coffee mixture into each cup, filling to 3/4 full.
spray a small baking sheet lined with foil with non-stick spray. Place marshmallow halves cut side down on the baking sheet.
Broil on top rack until tops are a deep brown. Place one half on top of each glass and serve.