Tokens

The basic fundamental unit of program that cannot be further breakdown is known as
Tokens.
Tokens are divided into following categories
1) Identifiers
2) Keywords
3) Operators
4) Literals
5) Punctuators

1) Identifiers in Python: Identifier is a user-defined name given to a

variable, function, class, module, etc. The identifier is a combination of character
digits and an underscore. They are case-sensitive i.e., ‘num’ and ‘Num’ and ‘NUM’
are three different identifiers in python. It is a good programming practice to give
meaningful names to identifiers to make the code understandable.

We can also use the Python string isidentifier() method to check whether a string is
a valid identifier or not.

Rules for Naming Python Identifiers

 It cannot be a reserved python keyword
 It can be a combination of A-Z, a-z, 0-9, or underscore.
 It should start with an alphabet character or an underscore ( _ ).

2) Keywords in Python: Python Keywords are some predefined and

reserved words in python that have special meanings. Keywords are used to define
the syntax of the coding. The keyword cannot be used as an identifier, function, or
variable name. All the keywords in python are written in lowercase except True
and False. There are 35 keywords in Python 3.11.

In python, there is an inbuilt keyword module that provides an iskeyword()

function that can be used to check whether a given string is a valid keyword or not.
Keywords Description
This is a logical operator which returns true if both the operands are true else
and
returns false.
This is also a logical operator which returns true if anyone operand is true
or
else returns false.
This is again a logical operator it returns True if the operand is false else
not
returns false.
if This is used to make a conditional statement.
Elif is a condition statement used with an if statement. The elif statement is
elif
executed if the previous conditions were not true.
Else is used with if and elif conditional statements. The else block is executed
else
if the given condition is not true.
for This is used to create a loop.
while This keyword is used to create a while loop.
break This is used to terminate the loop.
as This is used to create an alternative.
def It helps us to define functions.
lambda It is used to define the anonymous function.
pass This is a null statement which means it will do nothing.
return It will return a value and exit the function.
TRUE This is a boolean value.
FALSE This is also a boolean value.
try It makes a try-except statement.
with The with keyword is used to simplify exception handling.
This function is used for debugging purposes. Usually used to check the
assert
correctness of code
class It helps us to define a class.
continue It continues to the next iteration of a loop
del It deletes a reference to an object.
except Used with exceptions, what to do when an exception occurs
Finally is used with exceptions, a block of code that will be executed no
finally
matter if there is an exception or not.
from It is used to import specific parts of any module.
global This declares a global variable.
import This is used to import a module.
in It’s used to check whether a value is present in a list, range, tuple, etc.
is This is used to check if the two variables are equal or not.
This is a special constant used to denote a null value or avoid. It’s important
none
to remember, 0, any empty container(e.g empty list) do not compute to None
nonlocal It’s declared a non-local variable.
 raise This raises an exception.
yield It ends a function and returns a generator.
async It is used to create asynchronous coroutine.
await It releases the flow of control back to the event loop.

You can also list out all keywords of any version of python. For that you just have to
import keyword library and call kwlist function of such library in following manner.

import keyword

print(keyword.kwlist)

Output
['False', 'None', 'True', 'and', 'as', 'assert', 'async', 'await', 'break', 'class', 'continue', 'def', 'del',
'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal',
'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']

3) Operators: are symbols used to perform operations over the operands.

Operators are classified into two categories on basis of number of operands
a) Unary Operators
b) Binary Operators

Unary Operators: are those operators that requires exactly one operand
to perform its working.
E.g. Logical NOT, Unary +, Unary -, Bitwise Not ~ etc.

Binary Operators: are those operators that requires exactly two operands
to perform its working. It is further classified into following categories.

 Arithmetic operators
 Assignment operators
 Comparison operators
 Logical operators
 Identity operators
 Membership operators
 Bitwise operators
Python Arithmetic Operators
Arithmetic operators are used with numeric values to perform common mathematical operations:

Operator Name Example

+ Addition x+y
- Subtraction x–y
* Multiplication x*y
/ Division x/y
% Modulus x%y
** Exponentiation x ** y
// Floor division x // y

Python Assignment Operators

Assignment operators are used to assign values to variables:

Operator Example Same As

= x=5 x=5
+= x += 3 x=x+3
-= x -= 3 x=x-3
*= x *= 3 x=x*3
/= x /= 3 x=x/3
%= x %= 3 x=x%3
//= x //= 3 x = x // 3

Python Comparison/Relational Operators

Comparison operators are used to compare two values:

Operator Name Example

== Equal x == y
!= Not equal x != y
> Greater than x>y
< Less than x<y
>= Greater than or equal x >= y
to
<= Less than or equal to x <= y
Python Logical Operators
Logical operators are used to combine conditional statements:

Operator Description Example

and Returns True if both x < 5 and x
statements are true < 10

or Returns True if one of x < 5 or x <

the statements is true 4

not Reverse the result, not(x < 5

returns False if the and x < 10)
result is true

Python Identity Operators

Identity operators are used to compare the objects, not if they are equal, but if they are actually the
same object, with the same memory location:

Operator Description Example

is Returns True if both x is y
variables are the
same object

is not Returns True if both x is not y

variables are not the
same object
Python Membership Operators
Membership operators are used to test if a sequence is presented in an object:

Operator Description Example

in Returns True if a x in y
sequence with the
specified value is
present in the object

not in Returns True if a x not in y

sequence with the
specified value is not
present in the object

Python Bitwise Operators

Bitwise operators are used to compare (binary) numbers:

Operator Name Description Example

& AND Sets each bit to x & y

1 if both bits are
1

| OR Sets each bit to x | y

1 if one of two
bits is 1

^ XOR Sets each bit to x ^ y

1 if only one of
two bits is 1

~ NOT Inverts all the ~x

bits
 << Zero fill left shift Shift left by x << 2
pushing zeros
in from the right
and let the
leftmost bits
fall off

>> Signed right Shift right by x >> 2

shift pushing copies
of the leftmost
bit in from the
left, and let the
rightmost bits
fall off

Operator Precedence
Operator precedence describes the order in which operations are performed.

The precedence order is described in the table below, starting with the highest precedence at the
top:

Operator Description
() Parentheses
** Exponentiation
+x -x ~x Unary plus, unary
minus, and bitwise
NOT
* / // % Multiplication,
division, floor
division, and
modulus
+ - Addition and
subtraction
<< >> Bitwise left and
right shifts
& Bitwise AND
^ Bitwise XOR
 | Bitwise OR
== != > >= < <= is is Comparisons,
not in not in identity, and
membership
operators
not Logical NOT
and AND
or OR

4) Literals in Python

A literal in Python is a syntax that is used to completely express a fixed value of a
specific data type. Literals are constants that are self-explanatory and don’t need to be
computed or evaluated. They are used to provide variable values or to directly
utilize them in expressions. Generally, literals are a notation for representing a fixed
value in source code.
They can also be defined as raw values or data given in variables or constants. In this
article, we will explore the different types of literals in Python along with examples to
demonstrate their usage.

Types of Literals in Python

Python supports various types of literals, such as numeric literals, string literals,
Boolean literals, and more. Let’s explore different types of literals in Python with
examples:
1. String literals
2. Character literal
3. Numeric literals
4. Boolean literals
5. Literal Collections
6. Special literals

5) Punctuators: Punctuators are nothing but symbols used in programming

languages to organize sentence structures, and indicate the rhythm and emphasis of
expressions, statements and program structure.

Examples of punctuators in python: ‘, “, #, \, /, (, ), [, ], {, }, @, ;, :, . Etc.