Yes - as theshadowxx said - how are you planning to delete it? If you just
remove the recycle bin icon from the desktop , then your recycled material still goes to C:\RECYCLER (or wherever it's configured to go).
If you were able to delete that folder (that would take some work as it is a system folder - highly unlikely it would happen by accident - but then, users will always surprise you with the things they do to computers!) I'm not sure what would happen. I imagine you would get an error message saying "Folder C:\Recycler Doesn't Exist!" and your file would be deleted much the same as it was in the old days (gone from the disc directory, but still in the memory until overwritten).
I'll have to try that out sometime. If anyone else knows, I am now interested myself in what would happen!