He had taken out loans that were secured by the slaves. But when the slaves were freed, he still had to pay the debts. Into the 1870s, banks in Philadelphia were forcing him to pay back the loans.