Sounds like you may have not prep’ed the thumb drive correctly.
First you need to format it using a Mac with a GUID Journaled file system. Next you need the correct OS installer copied to it.
The highest your system can handle is macOS High Sierra (10.13.x) but I strongly recommend sticking with Sierra (10.12.x) as High Sierra alters the file system from HFS+ to APFS which does not run too well on older SATA based drives. Get the installer from here: How to get old versions of macOS older installers have an expired certificate.
Follow this guide to setup the installer onto the thumb drive How to create a bootable macOS Sierra installer drive
Last step if the drive won’t boot up on its own with the thumb drive, time to restart your system and press the Option (⌥) to get to the startup manager so you can select the thumb drive.
Reference: Mac startup key combinations