You can do this for individual messages or in bulk:
Go to "Messages"
Go to "Undelivered"
Go to "Unseen"
Then click the check boxes next to messages you want to clear
Click "Actions"
Select "Clear Undelivered Label"
How to clean up the undelivered messages sub-category
Written by Anne Heath
Updated over a year ago