I have a mathematical function that changes the rotation of an object. Unfortunately, this function sometimes produces indefinite numbers: the console outputs "-1.#IND". How can you check for an indefinite number? Also I have been trying to use a ternary operator according to Lua's wiki website, yet I can't get it to work.
It is a simple as:
object.rotation = ( angle > 0 ) ? angle : object.rotation
Yet I get the error "unexpected symbol near '?'".
So how do I check for -1.#IND and use it in a ternary operator?

There is no ternary operator in Lua. There is an ugly hack with the use of "and" and "or" but it is complex and there are some issues so my advice: stay away from it.

As for your other problem, just check that your angle is between the limits:

1
2
3

if angle >= 0 and angle < 360 then
   object.rotation = angle
end