Lecture 16

E21 Computer Engineering Fundamentals

Emad Masroor

October 29, 2026

Algorithm for Gaussian Elimination

Algorithm for Gaussian Elimination

Algorithm for backward substitution

\[ \left[\begin{array}{ccc|c} A^*_{11} & A^*_{12} & A^*_{13} & b^*_1 \\ 0 & A^*_{22} & A^*_{23} & b^*_2 \\ 0 & 0 & A^*_{33} & b^*_3 \end{array}\right] \]

\[x_3 = \frac{b_3^*}{A_{33}^*}\] \[x_2 = \frac{b_2^* - A_{23}^* x_3}{A_{22}^*}\] \[x_1 = \frac{b_1^* - A_{13}^* x_3 - A_{12}^* x_2}{A_{11}^*}\]

\[\boxed{x_k = \frac{1}{A^*_{kk}} \left( b^*_k - \sum^n_{j = k +1} \left[ A^*_{kj} x_j \right] \right)}\]

  • The symbol \(\sum^n_{j = k +1}\) means sum over the term inside square brackets using index \(j\) from \(j = k+1\) to \(j = n\) (inclusive)

  • The \(*\) reminds us that backward substitution is performed on the row-reduced augmented matrix \(A^* | b^*\), not the originally given problem \(A | b\).

Gaussian Elimination by hand

Simplify the given problem to row-reduced form

Worksheet

Solution

Solve \(Ax = b\) where \[ A =\begin{bmatrix} 2 & 1 & -1 & 0\\ 1 & 1 & 2 & 0\\ -1 & 2 & 1 & 1\\ 6 & 1 & 1 & -2 \end{bmatrix}, \quad b = \begin{bmatrix} 1 \\ -1 \\ 0 \\ 2 \end{bmatrix}. \]

By first obtaining a row-echelon form \[ \left[ \begin{array}{cccc|c} A_{11} & A_{12} & A_{13} & A_{14} & b_1\\ 0 & A_{22} & A_{23} & A_{24} & b_2 \\ 0 & 0 & A_{33} & A_{34} & b_3 \\ 0 & 0 & 0 & A_{44} & b_4 \end{array} \right] \]

The solution is \[\displaystyle \left[ \begin{array}{cccc|c} {2} & {1} & {-1} & {0} & {1}\\ {0} & {0.5} & {2.5} & {0} & {-1.5} \\ {0} & {0} & {-12} & {1} & {8} \\ {0} & {0} & {0} & {-5/6}& {7/3} \end{array} \right]\]

Gaussian Elimination in Python

Copy into your Python workspace

A = np.array([[2,1,-1,0],
               [1,1,2,0],
               [-1,2,1,1],
               [6,1,1,-2]],
              dtype=np.float64)
b1 = np.array([[1],[-1],[0],[2]],
              dtype=np.float64)

Write one line of code for each step of Gaussian elimination

# Line-by-line Gaussian elimination

A[1,:] = A[1,:] - (A[1,0]/A[0,0]) * A[0,:]
A[2,:] = A[2,:] - (A[2,0]/A[0,0]) * A[0,:]
A[3,:] = A[3,:] - (A[3,0]/A[0,0]) * A[0,:]

A[2,:] = A[2,:] - (A[2,1]/A[1,1]) * A[1,:]
A[3,:] = A[3,:] - (A[3,1]/A[1,1]) * A[1,:]

A[3,:] = A[3,:] - (A[3,2]/A[2,2]) * A[2,:]

Further look inside the Gaussian Elimination algorithm

# Line-by-line Gaussian elimination

A[1,:] = A[1,:] - (A[1,0]/A[0,0]) * A[0,:]
A[2,:] = A[2,:] - (A[2,0]/A[0,0]) * A[0,:]
A[3,:] = A[3,:] - (A[3,0]/A[0,0]) * A[0,:]

A[2,:] = A[2,:] - (A[2,1]/A[1,1]) * A[1,:]
A[3,:] = A[3,:] - (A[3,1]/A[1,1]) * A[1,:]

A[3,:] = A[3,:] - (A[3,2]/A[2,2]) * A[2,:]

Original Matrix

\[ \begin{bmatrix} 2 & 1 & -1 & 0 \\ 1 & 1 & 2 & 0 \\ -1 & 2 & 1 & 1 \\ 6 & 1 & 1 & -2 \end{bmatrix} \]

Step 1

\[ \begin{bmatrix} 2 & 1 & -1 & 0 \\ 0 & 1/2 & 5/2 & 0 \\ -1 & 2 & 1 & 1 \\ 6 & 1 & 1 & -2 \end{bmatrix} \]

Step 2

\[ \begin{bmatrix} 2 & 1 & -1 & 0 \\ 0 & 1/2 & 5/2 & 0 \\ 0 & 5/2 & 1/2 & 1 \\ 6 & 1 & 1 & -2 \end{bmatrix} \]

Step 3

\[ \begin{bmatrix} 2 & 1 & -1 & 0 \\ 0 & 1/2 & 5/2 & 0 \\ 0 & 5/2 & 1/2 & 1 \\ 0 & -2 & 4 & -2 \end{bmatrix} \]

Step 4

\[ \begin{bmatrix} 2 & 1 & -1 & 0 \\ 0 & 1/2 & 5/2 & 0 \\ 0 & 0 & -12 & 1 \\ 0 & -2 & 4 & -2 \end{bmatrix} \]

Step 5

\[ \begin{bmatrix} 2 & 1 & -1 & 0 \\ 0 & 1/2 & 5/2 & 0 \\ 0 & 0 & -12 & 1 \\ 0 & 0 & 14 & -2 \end{bmatrix} \]

Step 6

\[ \begin{bmatrix} 2 & 1 & -1 & 0 \\ 0 & \frac{1}{2} & \frac{5}{2} & 0 \\ 0 & 0 & -12 & 1 \\ 0 & 0 & 0 & -\frac{5}{6} \end{bmatrix} \]

The LU Decomposition Method

Any square matrix \(A\) can be written as the product of an upper triangular matrix and a lower traingular matrix, which we call \(U\) and \(L\) respectively.

\(A = L U\)

\(Ax = b \implies L U x = b\)

Let \(Ux = y\). Then, \(L y = b\)

Why do this?

  • The two equations \(Ly = b\) and \(Ux = y\) are easy to solve!

\[L = \begin{bmatrix} L_{11} & 0 & 0 \\ L_{21} & L_{22} & 0 \\ L_{31} & L_{32} & L_{33} \end{bmatrix}, U = \begin{bmatrix} U_{11} & U_{12} & U_{13} \\ 0 & U_{22} & U_{23} \\ 0 & 0 & U_{33} \end{bmatrix}\]

Doolittle’s Decomposition

  • The \(L\) and \(U\) for a given matrix \(A\) are not unique.
  • There are different approaches to finding \(L\) and \(U\) uniquely
  • One approach is Doolittle’s Decomposition:
    • Let \(L_{11}\), \(L_{22}\), \(L_{33}\) etc. all be equal to 1.
    • Now, \(L\) and \(U\) are unique for any \(A\)
    • For any given square \(A\), there exist unique \[L = \begin{bmatrix} 1 & 0 & 0 \\ L_{21} & 1 & 0 \\ L_{31} & L_{32} & 1 \end{bmatrix}, U = \begin{bmatrix} U_{11} & U_{12} & U_{13} \\ 0 & U_{22} & U_{23} \\ 0 & 0 & U_{33} \end{bmatrix}\] such that \(A = LU\)
  • How do we find \(L\) and \(U\) ?

Doolittle’s Decomposition & Gaussian Elimination

\[L = \begin{bmatrix} 1 & 0 & 0 \\ L_{21} & 1 & 0 \\ L_{31} & L_{32} & 1 \end{bmatrix}, U = \begin{bmatrix} U_{11} & U_{12} & U_{13} \\ 0 & U_{22} & U_{23} \\ 0 & 0 & U_{33} \end{bmatrix}\]

  • \(U\) is the row-echelon form of \(A\), found by Gaussian elimination
  • \(L_{ij}\) is the ‘multiplier’ used to eliminate \(A_{ij}\) in Gaussian elimination
  • Gaussian Elimination of the square matrix \(A\) is equivalent to finding Doolittle’s \(LU\) Decomposition of the square matrix \(A\)
    • Why? Perform Gaussian Elimination on the product \(LU\): \[LU = \left[\begin{array}{lll} U_{11} & U_{12} & U_{13} \\ U_{11} L_{21} & U_{12} L_{21} + U_{22} & U_{13} L_{21} + U_{23} \\ U_{11} L_{31} & U_{12} L_{31} + U_{22} L_{32} & U_{13} L_{31} + U_{23} L_{32} + U_{33} \end{array}\right] \]

If you have \(L\) and \(U\), solve \(Ax=b\) …

  1. Given the linear system of equations \(Ax=b\),
  2. Write \(A\) as the product of \(L\) and \(U\), i.e. \(LUx = b\).
  3. Let \(y = U x\).
  4. Solve \(L y = b\) for \(y\) by forward substitution. \[\begin{bmatrix} 1 & 0 & 0 \\ L_{21} & 1 & 0 \\ L_{31} & L_{32} & 1 \end{bmatrix}\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix} = \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix}\]
  5. Solve \(Ux=y\) for \(x\) by backward substitution. \[\begin{bmatrix} U_{11} & U_{12} & U_{13} \\ 0 & U_{22} & U_{23} \\ 0 & 0 & U_{33} \end{bmatrix}\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = \begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix}\]

Banded Matrices

In engineering, we often encounter banded matrices

\[\begin{bmatrix} A_{11} & A_{12} & 0 & 0 & 0 & 0 \\ A_{21} & A_{22} & A_{23} & 0 & 0 & 0 \\ 0 & A_{32} & A_{33} & A_{34} & 0 & 0 \\ 0 & 0 & A_{43} & A_{44} & A_{45} & 0 \\ 0 & 0 & 0 & A_{54} & A_{55} & A_{56} \\ 0 & 0 & 0 & 0 & A_{65} & A_{66} \end{bmatrix}\]

  • A tridiagonal matrix has 3 bands surrounding the diagonal.

  • Specialized solvers can be used for these; don’t waste operations on the zeros!

  • The \(LU\) Decomposition of a banded matrix preserves the banded structure. e.g.,

    \[L = \begin{bmatrix} L_{11} & 0 & 0 & 0 & 0 & 0 \\ L_{21} & L_{22} & 0 & 0 & 0 & 0 \\ 0 & L_{32} & L_{33} & 0 & 0 & 0 \\ 0 & 0 & L_{43} & L_{44} & 0 & 0 \\ 0 & 0 & 0 & L_{54} & L_{55} & 0 \\ 0 & 0 & 0 & 0 & L_{65} & L_{66} \end{bmatrix}, U = \begin{bmatrix} U_{11} & U_{12} & 0 & 0 & 0 & 0 \\ 0 & U_{22} & U_{23} & 0 & 0 & 0 \\ 0 & 0 & U_{33} & U_{34} & 0 & 0 \\ 0 & 0 & 0 & U_{44} & U_{45} & 0 \\ 0 & 0 & 0 & 0 & U_{55} & U_{56} \\ 0 & 0 & 0 & 0 & 0 & U_{66} \end{bmatrix}\]

LU Decomposition of Tridiagonal matrices

Solve \(Ax=b\) using LU Decomposition when

\[A = \begin{bmatrix} d_1 & e_1 & 0 & 0 & \cdots & 0 \\ c_1 & d_2 & e_2 & 0 & \cdots & 0 \\ 0 & c_2 & d_3 & e_3 & \cdots & 0 \\ 0 & 0 & c_3 & d_4 & \cdots & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 0 & c_{n-1} & d_n \end{bmatrix}\, c = \begin{bmatrix} c_1 \\ c_2 \\ \vdots \\ c_{n-1} \end{bmatrix}\, d = \begin{bmatrix} d_1 \\ d_2 \\ \vdots \\ d_n \end{bmatrix}\, e = \begin{bmatrix} e_1 \\ e_2 \\ \vdots \\ e_{n-1} \end{bmatrix}\]

Gaussian Elimination of second row of \(A\):

\[\boxed{\text{Row k} = \text{Row k} - \frac{A_{21}}{A_{11}} \times \text{Row }(k-1)}\]

Write down a formula for \(c_1\), \(d_2\) and \(e_2\) under one step of Gaussian elimination, in terms of all other \(c\)’s, \(d\)’s and \(e\)’s.

\[ \begin{align} \text{New } c_1 &= c_1 - \frac{c_1}{d_1} \times d_1 \implies c_1 = 0 \\ \text{New } d_2 &= d_2 - \frac{c_1}{d_1} \times e_1 \implies d_2 = d_2 - \frac{c_1}{d_1} e_1 \\ \text{New } e_2 &= e_2 - \frac{c_1}{d_1} \times 0 \implies e_2 = e_2 \end{align} \]

  • \(c_1\) is eliminated, \(d_2\) changes, and \(e_2\) doesn’t change.
  • The multiplier for Row 2 was \(\left(c_1/d_1\right)\)

Continued

The \(LU\) Decomposition of \(A\) is therefore

\[L = \begin{bmatrix} 1 & 0 & 0 & 0 & \cdots & 0 \\ \lambda_1 & 1 & 0 & 0 & \cdots & 0 \\ 0 & \lambda_2 & 1 & 0 & \cdots & 0 \\ 0 & 0 & \lambda_3 & 1 & \cdots & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 0 & \lambda_{n-1} & 1 \end{bmatrix}, \quad U = \begin{bmatrix} d_1^* & e_1 & 0 & 0 & \cdots & 0 \\ 0 & d_2^* & e_2 & 0 & \cdots & 0 \\ 0 & 0 & d_3^* & e_3 & \cdots & 0 \\ 0 & 0 & 0 & d_4^* & \cdots & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 0 & 0 & d_n^* \end{bmatrix}\]

  • where \(\lambda_k\) is the multiplier used to eliminate \(c_k\),
  • the \(*\) indicates that the terms have been (possibly) updated from their original values

Recall that \(LUx = b\) can be solved in two steps:

  1. Forward substitution of \(Ly = b\)

    \(\left[\begin{array}{cccccc|c} 1 & 0 & 0 & 0 & \cdots & 0 & b_1 \\ \lambda_1 & 1 & 0 & 0 & \cdots & 0 & b_2 \\ 0 & \lambda_2 & 1 & 0 & \cdots & 0 & b_3 \\ 0 & 0 & \lambda_3 & 1 & \cdots & 0 & b_4 \\ \vdots & \vdots & \vdots & \vdots & \cdots & \vdots & \vdots \\ 0 & 0 & \cdots & 0 & \lambda_{n-1} & 1 & b_n \end{array}\right]\)

  1. Backward substitution of \(Ux = y\)

    \(\left[\begin{array}{cccccc|c} d_1^* & e_1 & 0 & \cdots & 0 & 0 & y_1 \\ 0 & d_2^* & e_2 & \cdots & 0 & 0 & y_2 \\ 0 & 0 & d_3^* & \cdots & 0 & 0 & y_3 \\ \vdots & \vdots & \vdots & & \vdots & \vdots & \vdots \\ 0 & 0 & 0 & \cdots & d_{n-1}^* & e_{n-1} & y_{n-1} \\ 0 & 0 & 0 & \cdots & 0 & d_n^* & y_n \end{array}\right]\)