This has been overkill from the start, and it has been bugging me for a long time. With this change, we're probably a bit slower on most platforms but save huge amounts of space with all in-memory sample datastructures.