Lecture 18

E21 Computer Engineering Fundamentals

Emad Masroor

November 5, 2026

Divergence and Convergence

[ 4.  -0.4 -2. ]
[-6.2 -6.4  2.2]
[-15.8  11.6  20.2]
[ 74.2   34.16 -33.8 ]
[  33.88 -131.44 -101.48]
[-598.28  -94.    395.32]

[-0.5     0.3333 -0.25  ]
[0.2917 0.7083 0.0833]
[ 0.0313  0.6736 -0.1042]
[ 0.1337  0.724  -0.0868]
[ 0.1124  0.7323 -0.112 ]
[ 0.1229  0.7414 -0.1162]

Tolerance, Residual, Norm

  • An iterative numerical algorithm for \(Ax=b\) can be terminated when the sequence \[x^{(0)} \rightarrow x^{(1)} \rightarrow x^{(2)} \rightarrow ... \rightarrow x^{(k)}\] sufficiently converges

  • Under certain conditions, this is guaranteed to happen.

  • Typically, we use while loops to implement this:
    Pseudo-code:

    while norm(residual) > tolerance:
      carry out 1 iteration of numerical method
      calculate residual

Convergence Guarantees

Jacobi and Gauss-Seidel methods are guaranteed to converge to a solution of \(Ax=b\) $under either of the following conditions:

  1. The matrix \(A\) is diagonally dominant, or
  2. The matrix \(A\) is symmetric (\(A_{ij} = A_{ji}\) for all \(i\) and \(j\)) and positive definite (all eigenvalues are positive)

But does a solution to \(Ax=b\) even exist?

  • Convergence guarantees are of no use to us if the system of equations \(Ax=b\) has no solution.
  • When does \(Ax=b\) have no unique solution? Worksheet

Four equivalent statements about solvability of \(Ax=b\)

The following four statements are equivalent, and any one of them implies all the others.

  1. There is a unique solution \(x\) for the equation \(Ax=b\)
    Statement 1 does not hold if:
    • If more than one different vectors \(x\) satisfy the equation \(Ax=b\), then statement 1 does not hold.
    • If no vector \(x\) satisfies the equation \(Ax=b\), then statement 1 does not hold
  2. The matrix \(A\) is not singular
    • A singular matrix is bad — it represents equations that cannot be solved.
  3. The determinant of \(A\) is not equal to zero
    • For 2x2 matrices, the determinant of \(A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}\) is \(\det A = |ad-bc|\)
  4. \(A\) is invertible, i.e., the matrix \(A^{-1}\) exists such that \(A A^{-1} = I\)
  5. The rows of \(A\) are linearly independent
    • i.e., the indvidual scalar equations in \(Ax=b\) are not multiples of each other
    • and no row is a linear combination of the other rows

Pivoting (in direct methods)

  • Python’s numpy package contains numpy.linalg.solve(A,b) which pretty much always works if a uniques solution to \(Ax=b\) exists
  • Direct methods with pivoting!

The need for pivoting

  • Consider the two matrices \[ A = \begin{bmatrix} 0 & -1 & 1 \\ -1 & 2 & -1 \\ 2 & -1 & 0 \end{bmatrix}, \quad B = \begin{bmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1 & 1 \end{bmatrix} \]

Pivoting Procedure

  1. Calculate a scale factor for each row: \(\displaystyle s_i \equiv \max_{j} |A_{ij}|\)
  2. Define the relative size of an element \(A_{ij}\) as \(r_{ij} \equiv \frac{|A_{ij}|}{s_i}.\) This measures how big element \(A_{ij}\) is relative to the other elements in its row.
  3. In the Gaussian elimination procedure, before you start eliminating the entries of column \(k\), check the relative size of all elments in column \(k\) below the diagonal, i.e., determine \(\displaystyle r_{pk} = \max_j (r_{jk}), \quad j \ge k\)
  4. Swap rows \(k\) and \(p\)

Well-conditioned and ill-conditioned matrices

The condition number of a matrix is defined as the product of the norm of the matrix and the norm of its inverse:

\[\text{cond } A = || A || \cdot || A^{-1}||\]

Note: When computing \(|| A ||\) for a matrix we will use the Frobenius norm: \[|| A || \equiv \sqrt{\sum_{i=1}^{N} \sum_{j=1}^N A_{ij}^2}.\]

>>> a = np.array([[4,2],[2,-1]])
>>> np.linalg.norm(a)*np.linalg.norm(np.linalg.inv(a))
np.float64(3.125)
>>> np.linalg.cond(a)
np.float64(2.763085794518659)
>>> np.linalg.cond(a, 'fro')
np.float64(3.125)

Ill-conditioned matrices lead to error accumulation

Identifying Ill-conditioned matrices

Description Criterion
Well conditioned \(\text{cond}(A) \approx 1\)
Ill conditioned \(\text{cond}(A) \gg 1\)
Singular \(\text{cond}(A) \rightarrow \infty\)

Heuristic:

If a matrix is ill-conditioned, its determinant (or the determinant of its inverse) is very small.

But what does ‘small’ mean?

A matrix’s determinant is very small if \[\det A \ll ||A||\]