cone is used to define the constraints norm(x) < y without invoking the overhead of using the norm operator.
Constraining the Euclidean norm of a vector to be less than 1 is done with
F = cone(x,1);
Of course, arbitrary complicated constructs are possible, such as constraining the norm of the diagonal to be less than the sum of the off-diagonal terms in a matrix.
F = cone(diag(x),sum(sum(x-diag(diag(x)))))
An alternative is to use the norm operator instead (see the examples on nonlinear operators for details)
F = norm(diag(x)) <= sum(sum(x-diag(diag(x))))
The benefit of using the cone command is that the overhead in YALMIP is reduced (no convexity analysis is required).