Here's some advice to make the mini-game work a bit better for you. Some of it might be very basic, so feel free to roll your eyes and move on to the next...
Most of this advice is applicable to relic/data sites in null sec or in wormholes (where the null sec sites now spawn). If you are doing high sec or low sec sites – they are much easier and you can take more of a whac-a-mole approach and not worry about tactics.
Here are the things you'll run up against (along with their proper names): http://wiki.eveuniversity.org/File:Hacking_Indications.png
I really don’t have a clue about the names when I’m playing - they’re just easily identifiable symbols to me. But for the sake of clarity in these instructions I’ll use their actual authentic officially assigned names instead of “that wrench thing” and “the eagle beak thing” and “the wedge thing” and whatever other image comes to my mind when I’m in the mini-game.
Usually the System Core that you seek is located on the far side of the maze from where you start – head there as quickly as you can. First off, go through the maze and don't mess with any of the defensive subsystems until you need to (except Restoration Nodes and Virus Supressors – see below). Leave the Data Caches (the white dots) alone until all of the regular explorable nodes (the empty circles) are revealed. And even then be careful of the Data Caches - they often contain bad things. Go back to them if you're desperate to find a tool and you're facing defeat.
As you go through the explorable nodes and reveal them – you will also be revealing a number for a short time on the revealed node. That number is how far a Good Thing is from you. It could be the System Core, or a tool, or a Data Cache (which in this case is classified as a “good thing” but probably isn’t). With these numbers you can often eliminate a corner of the maze from the possibility of it having the System Core – if the numbers are 4s or 5s and there are only 3 nodes on the longest path towards the corner, then there’s nothing good in that neighborhood. Head to the other corner of the diagram.
If you run into a Restoration Node - deal with it immediately. Usually without using any of your tools (tools can are better used on other stuff - this might not be good advice, but it's the way I do things) - it's two or three clicks to get rid of the Restoration Node. But if you don't - it'll make all the other bad things even more badder every time you take an action...
If you run into a Virus Suppressor - you are in for some very bad times unless you have a Secondary Vector handy. Never use the Secondary Vector on anything other than the Virus Suppressor unless you are desperate. The Virus Suppressor cuts your hacking strength in half - so don't try to do anything other than explore new nodes if you don't have a Secondary Vector handy. If you do have a Secondary Vector - have it attack the Virus Suppressor as you explore more nodes - it does the work independently and simultaneously, you don't need to keep clicking on it.
When you get a Utility Subsystem - use it immediately. There is no reason to not click on it and get those points and free up a spot for useful tools.
Eventually you’ll have no more routes to try without running into a Firewall or an Antivirus. Attack an obstacle with low strength and in the direction of where you think the System Core might be and continue on.
You have two main tools to help against the obstacles. The Kernel Rot cuts the strength of any Defensive Subsystem in half. The Polymorphic Shield protects you for a couple rounds from getting any damage. Try to figure out which would be most appropriate against whatever you are up against. I don’t really make much distinction between them and use them randomly (I do pay more attention when my coherence is almost gone - then I tend to use the Polymorphic Shield for protection).
And remember that if you're getting close to the end of your coherence (your “hit points”) and finally stumble on the System Core - you can use these Utility Subsystems to help against it and maybe save enough to survive and get the loot.
Hope that helps!