Remove hard pandas dependency

Internal calculations are stored in pandas Series and DataFrames. It's probably more future-proof to store them in a more agnostic way. Ideally, users should be able to not have to install pandas if they don't want to.