Electric Vehicle (EV) has become a preferable choice in the modern transportation system due to its environmental and energy sustainability. However, in many large cities, the EV drivers often fail to find the proper spots for charging, because of the limited charging infrastructures and the spatiotemporally unbalanced charging demands. Indeed, the recent emergence of deep reinforcement learning provides great potential to improve the charging experience from various aspects over a long-term horizon. In this paper, we propose a framework, named Multi-Agent Spatio-Temporal Reinforcement Learning (MAST), for intelligently recommending public accessible charging stations by jointly considering various long-term spatiotemporal factors. Specifically, by regarding each charging station as an individual agent, we formulate the problem as a multi-objective multi-agent reinforcement learning task. We first develop a multi-agent actor- critic framework with a centralized attentive critic to coordinate the recommendation between geo-distributed agents. Moreover, to quantify the influence of future potential charging competition, we introduce a delayed access strategy to integrate the unpredictable future charging competition. After that, to effectively optimize multiple learning objectives, we propose a multi-critic architecture with a dynamic gradient reweighting strategy to adaptively guide the optimization direction. Finally, extensive experiments on two real-world datasets demonstrate that MAST achieves the best comprehensive performance compared with several baseline approaches.