Pushing or pulling things around wears you out – whether lifting, dragging or accelerating (the force could speed the thing up, slow it down, or change its direction). You are working mechanically: the power is set by the how fast and how much force you exert. That works whatever is exerting the force—it does not have to be you.
Even lifting at a steady speed, you need to exert a force, supporting whatever you're lifting. (If you don't support it, a natural motion results: that's free-fall near planets). The force depends on the thing, but you can also choose to lift in a shorter or longer time. Gaining height in less time is more demanding work—just think of fitness training.
Dragging at a steady speed over a rough surface is also costing you. The force you need to exert depends on the roughness and how much support the thing needs. Again the speed you drag at affects what it's costing you: think resistance training.
Accelerate something by exerting a force on it to change its motion (its velocity, so speed or direction). Again you can choose to change the velocity in a shorter or longer time.