Olly Lee gave Craig Levein's side an early lead and fellow new signing Steven MacLean added a second goal in 57 minutes.
Paul McManus pulled one back for the Highland League champions with 15 minutes left but Hearts held on to take all three points.