Guide · free to read
IPsec VPN phase 1 and phase 2: what each negotiation actually establishes.
Every IPsec tunnel negotiation goes through two distinct phases, and a huge share of “the tunnel won’t come up” troubleshooting time gets wasted because it’s not obvious which phase is actually failing. The two phases secure different things, use different parameters, and fail for different reasons — knowing which is which turns a guessing exercise into a specific, checkable diagnosis.
Phase 1: negotiating a channel to negotiate over
Phase 1’s whole job is to establish one secure, authenticated, bidirectional channel — the IKE security association (SA) — between the two ends of the tunnel. Nothing you actually care about (your real traffic) travels over this channel; its only purpose is to protect the next negotiation, phase 2.
IKE version and mode. FortiGate supports both IKEv1 and IKEv2. If you’re on IKEv1, you also pick a mode:
- Main mode exchanges the phase 1 parameters across multiple rounds, with the authentication information itself encrypted. This is the default choice whenever it’s available.
- Aggressive mode exchanges everything in a single message, with authentication information sent unencrypted. It exists mainly because it’s required in specific scenarios — most notably when the remote peer has a dynamic IP address and there are multiple dialup configurations sharing the same local interface, a case main mode can’t handle. Aggressive mode isn’t a relevant choice on IKEv2, which negotiates differently from the start.
Authentication. Two methods: a pre-shared key — a shared secret configured identically on both ends — or certificates/signatures, where each side authenticates using a digital certificate instead of a shared secret. Certificate authentication avoids the operational problem every shared secret eventually has (it has to be distributed and rotated out-of-band to every peer that uses it).
What actually gets proposed and matched. Phase 1 negotiates a proposal: an encryption-algorithm/authentication-digest pair, plus a Diffie-Hellman group, and the two peers have to agree on at least one full combination. FortiGate supports DES, 3DES, and AES in 128/192/256-bit variants for encryption, plus AES-GCM modes and CHACHA20POLY1305 that are IKEv2-only; for authentication digests, MD5, SHA1, SHA256, SHA384, and SHA512. DH groups run from the older, weaker groups 1, 2, and 5 through the modern group range 14–32 — at least one configured group has to match on both sides or negotiation fails outright, and which groups are offered is one of the first things worth checking when a tunnel won’t establish at all.
Identifying the peer. Each side has a Local ID identifying itself and configurable peer options that control how the remote side’s identity gets validated — by a specific peer ID, by certificate, or against a dialup group for tunnels that accept connections from unpredictable remote addresses.
Phase 2: negotiating the tunnel your actual traffic uses
Once phase 1 succeeds, phase 2 negotiates over the now-secure IKE channel — and this negotiation is the one that actually produces the IPsec SA(s) that carry real traffic.
Proposals, again, but not the same parameter set. Phase 2 has its own encryption and authentication proposal list — NULL, DES, 3DES, AES variants, and IKEv2-specific modes like AES128GCM, plus the same family of authentication digests (NULL, MD5, SHA1, SHA256, SHA384, SHA512). These are negotiated independently of the phase 1 proposal; a tunnel can legitimately use different algorithms for each phase, and the two lists are not required to match each other.
Perfect Forward Secrecy (PFS). PFS forces a fresh Diffie-Hellman exchange every time the phase 2 key lifetime expires, rather than deriving new keys from material that’s already been in use. The practical effect: even if a past key were somehow compromised, PFS means that compromise doesn’t help decrypt data protected by a subsequent key, because each key’s derivation is independent. The DH group range for PFS is the same 1, 2, 5, and 14–32 list as phase 1, configured separately, and again has to have at least one matching group between peers.
Phase 2 (quick mode) selectors. These define which traffic — which source and destination subnets — is actually allowed to use the tunnel.
This is a common point of confusion worth stating plainly: phase 2 selectors decide what traffic can use the tunnel; they are not the same mechanism as the firewall policy that decides what traffic is allowed to reach the tunnel in the first place. A tunnel can be technically up with correct selectors on both sides and still pass no traffic because no firewall policy permits it — and the reverse is also possible, a correct policy pointed at a tunnel whose selectors don’t actually cover the subnet in question.
Replay detection. FortiGate checks inbound IPsec packets against ones it’s already seen, using 64-bit extended sequence numbers, and discards duplicates or out-of-order-beyond-window packets. This is a standard IPsec anti-replay mechanism, not something specific to FortiGate.
Key lifetime. Phase 2 keys expire on a lifetime you configure — by time (seconds), by volume (kilobytes processed), or both, whichever limit is hit first triggers rekeying.
Keeping the tunnel actually working. Auto-negotiate controls whether phase 2 SA negotiation starts automatically; autokey keep-alive keeps the tunnel active during periods with no real traffic, which matters because some tunnel designs would otherwise let the SA lapse during a quiet period and then have to renegotiate from scratch the moment traffic resumes — a delay worth avoiding on a tunnel carrying anything latency-sensitive.
Reading a failure by phase
The practical payoff of knowing this split: a tunnel that never comes up at all is almost always a phase 1 problem — mismatched pre-shared key, no overlapping proposal, wrong peer ID, or a mode mismatch (one side expecting main mode, the other configured for aggressive). A tunnel that comes up but won’t pass traffic, or that comes up and drops shortly after, is more often a phase 2 problem — selectors that don’t actually overlap between the two sides, or a PFS/DH group mismatch specific to phase 2 even though phase 1 negotiated fine. Checking phase 1 status and phase 2 status separately — rather than just “is the tunnel up” — is what turns that distinction from theory into a fast diagnosis.
Sources
Both phases above were checked against the IPsec VPN chapters of the FortiOS 7.6.0 Administration Guide. These two pages are the whole source list for this guide, they are free to open with no account, and checking them is the point of listing them:
- Phase 1 configuration — what phase 1 negotiates: IKEv1 main versus aggressive mode, authentication methods, proposal parameters, Diffie-Hellman groups, and local and peer ID.
- Phase 2 configuration — what phase 2 negotiates: proposal parameters, Perfect Forward Secrecy, phase 2 selectors, replay detection, key lifetime, and autokey keep-alive.
Every practice question we publish carries the Fortinet page it was written from, the same way this guide does. You can see fifteen of them, scored by blueprint domain, in the free readiness assessment — it is drawn from the same NSE 4 bank this guide’s subject belongs to, no account, nothing recorded. How they are written is set out on the methodology page, and the catalog says which exams are on sale today.