Yes. You can decline any request before sending a quote with no impact. And even after work has started, a one-off cancellation is fine. Things come up, and we understand. What matters is the pattern.
Canceling after work has started
If you cancel mid-project, the user is automatically refunded. Canceled projects show on your profile, so repeated cancellations over time will affect how buyers see you. One here or there won't make a dent. Where possible, talk to the user first and see if you can work something out together before canceling.