Is it mining, refinement, shipping, handling, manufacturing process, middleman, cost of material? or perhaps all of them?
Read this article:
Simply, the lack of a market for high capacity, high output, quick charging, long lasting, multiple cycle, small form-factor batteries is rather small as compared to the norm.
Businesses have become used to using lead acid batteries because they are proven and ubiquitous. Many people have been conditioned to find batteries to be highly disposable since childhood. When it dies, buy more. Until the advent of cellular phones, laptop computers, and tablets, many didn't think much of rechargeable battery technology at all.
In a world where people have gotten used to disposable components of a device, no one wanted to kill the golden goose that laid the golden Duracell®. The market had migrated to the least common denominator, where good enough was good enough, so everything that was demonstrably 'better' was in minimal supply, and thus more expensive in comparison. Mediocre remains the eternal enemy of awesome.
Battery University -- Battery Statistics
Material cost is the largest item. Not lithium but the nickle and cobalt. As these batteries get more common recycling should be feasible bringing costs down. Much of the aluminum and lead used today comes from recycled sources.
It will be a long time before recycled batteries will be enough to make even a little dent in production.