Sometimes there are guarenteed tell-tale signs. For instance a BSOD (Blue Screen of Death) check your memory and drivers, 99% of the time it is down to one or the other
If you are hearing a grinding sound, your hard drive is most probably at fault.
If you have artifacts on screen, video card.
Computer restarting or spontaneously shutting down, most probably overheating CPU.
If you are still stumped, for hardware first perform tests. You have utilities for different uses - memtest to test memory, hard drive vendor software for hard drive (only use the manufacturer software if it is available), or stress testing software.
Stress testing software will push the hardware to the limit, proving if the component is stable. If you are getting BSOD's for instance and stress test the CPU and it happens again, the CPU isn't stable, which often happens when overclocking. You can also stress test the video card to try to recreate artifacts or crashes.
Note though that when components are being stress tested they use more power. This means if the power supply is at fault, the symptoms may lead you to believe that it is the components that is being tested which is causing the problems.
The best way to find out which piece of hardware is causing your problem, use a known good. If you can't narrow it down, swapping components out one at a time into a fully working system can help you to find the cause of the problem. If you have the computer working, put a stick of memory in and it then begins to stop working correctly, you can say fairly certainly that the memory is causing your problem.
If it isn't a hardware fault and instead software:
Work out when the issues began. If it happened to be just after you installed a program, updated a driver or started/stopped a service, that is probably the cause. You can use tools in Windows to get back, such as last known good configuration (only works if you aren't able to log in and haven't logged in since the system was last working), rolling back drivers, system restore or safe mode.
If you aren't able to roll back, you can instead manually remove the software which, if you are having problems booting up correctly, may require you to load into safe mode so no additional software or drivers are loaded