If you don’t have enough funds in your Checking account to cover the total amount of your check, then the check will not be cashed. However, the vendor is responsible for reversing the transaction.
Ultimately, the transaction won’t be able to be completed, and you won’t be charged any fees by Betterment.