And if you want to permanently delete the completed tasks, you can set up a macro, like I've pasted below. There might be an easier way to write this, but this is what I wrote a couple of years ago, and it's been working fine since then.
Customize your toolbar to add an icon for this macro, and it becomes a one-click deal to delete them all.
Dim myTask As taskItem
Dim i As Long
For i = Outlook.ActiveExplorer.CurrentFolder.Items.Count To 1 Step -1
Set myTask = Outlook.ActiveExplorer.CurrentFolder.Items(i)
If myTask.Class = olTask Then
If myTask.Complete Then