Numerical Methods
Floating point operation
refers to a mathematical operation performed on two or more floating point numbers. Examples of floating point operations include addition, subtraction, multiplication, division, and exponentiation. The result of a floating point operation is also a floating point number,
FLOP count for matrix-vector
n(2n-1) = 2n^2-n
Implement a bisection solver in MATLAB
% Define the function to find the root of f = @(x) x^3 - 3*x + 1; % Define the interval to search for the root in a = -2; b = 2; % Define the maximum number of iterations and tolerance max_iter = 100; tolerance = 1e-6; % Initialize iteration counter and interval size iter = 0; interval_size = abs(b - a); % Perform bisection method while (iter < max_iter) && (interval_size > tolerance) % Calculate midpoint and function values at endpoints and midpoint c = (a + b) / 2; fa = f(a); fb = f(b); fc = f(c); % Determine which subinterval to search next if fa * fc < 0 b = c; fb = fc; else a = c; fa = fc; end % Update interval size and iteration counter interval_size = abs(b - a); iter = iter + 1; end % Display the final result root = (a + b) / 2; fprintf('The root is approximately %f, found in %d iterations.\n', root, iter);
Given a time stepping scheme, implement an ODE solver using that scheme in MATLAB, and use analytical solutions to validate the numerical solver
% Define the time step and time interval dt = 0.1; tspan = 0:dt:1; % Define the initial condition y0 = 1; % Initialize the solution vector y = zeros(size(tspan)); y(1) = y0; % Implement the forward Euler method for i = 2:length(tspan) y(i) = y(i-1) + dt * (-y(i-1)); end % Plot the numerical solution plot(tspan, y, 'b-', 'LineWidth', 2); % Plot the analytical solution for comparison hold on; y_exact = exp(-tspan); plot(tspan, y_exact, 'r--', 'LineWidth', 2); % Add labels and legend xlabel('Time'); ylabel('Solution'); legend('Numerical solution (forward Euler)', 'Analytical solution');
Know the forward Euler, midpoint, and backward Euler schemes, their orders of local and global accuracy, implicit/explicit classification, and stability properties
Forward Euler Scheme: The forward Euler scheme is an explicit method that approximates the solution of an ODE at the next time step by using the derivative at the current time step. The formula is given by: y_{n+1} = y_n + h f(t_n, y_n) where y_n is the numerical approximation of the solution at time t_n, f(t_n, y_n) is the derivative of the solution at time t_n, and h is the time step. The local truncation error of the forward Euler scheme is O(h^2), and the global error is O(h). The forward Euler scheme is conditionally stable for linear systems and unstable for nonlinear systems. Midpoint Scheme: The midpoint scheme is a second-order method that uses the midpoint of the interval between two time steps to approximate the solution of an ODE at the next time step. The formula is given by: y_{n+1} = y_n + h f(t_n + h/2, y_n + (h/2) f(t_n, y_n)) where y_n is the numerical approximation of the solution at time t_n, f(t_n, y_n) is the derivative of the solution at time t_n, and h is the time step. The local truncation error of the midpoint scheme is O(h^3), and the global error is O(h^2). The midpoint scheme is conditionally stable for linear systems and unstable for nonlinear systems. Backward Euler Scheme: The backward Euler scheme is an implicit method that approximates the solution of an ODE at the next time step by using the derivative at the next time step. The formula is given by: y_{n+1} = y_n + h f(t_{n+1}, y_{n+1}) where y_n is the numerical approximation of the solution at time t_n, f(t_{n+1}, y_{n+1}) is the derivative of the solution at time t_{n+1}, and h is the time step. The backward Euler scheme is unconditionally stable for linear and nonlinear systems. However, it requires solving a nonlinear equation at each time step, which can be computationally expensive. In the case of the backward Euler scheme, the local truncation error is O(h^2), meaning that the error in each step of the method is proportional to the square of the time step. The global error is also O(h^2), which means that the accuracy of the method is proportional to the square of the time step over the entire time interval of interest. The backward Euler scheme is an implicit method because it involves solving an equation at each time step,
Implement a Newton solver in MATLAB
unction [x, niter] = newton_solver(f, df, x0, tol, maxiter) % f: function handle for the function to find the root of % df: function handle for the derivative of the function % x0: initial guess for the root % tol: tolerance for the solution % maxiter: maximum number of iterations to perform % Initialize variables x = x0; niter = 0; % Perform iterations until convergence or maxiter is reached while niter < maxiter fx = f(x); dfx = df(x); if abs(fx) < tol return end x = x - fx / dfx; niter = niter + 1; end
Define the terms: well-conditioned problem, ill-conditioned problem
well-conditioned problem is one where small changes in the input data result in small changes in the output, while an ill-conditioned problem is one where small changes in the input data result in large changes in the output.
FLOP count for dot product
2n - 1
Machine precision
(or floating point precision) refers to the smallest possible difference between two floating point numbers that can be represented by a computer.
Describe the iterative approach to root finding, including the key ingredients for an iterative strategy
1. Initial guess 2. stopping criteria 3. guess update rule
State the necessary conditions for Newton's method to converge to a root
1. The function f(x) must be continuous and differentiable in an interval [a, b] containing the root x*. 2. The derivative f'(x) must be continuous and non-zero in [a, b]. 3. The initial guess x0 must be sufficiently close to the root x*.
Explain the necessary conditions for the bisection method to be guaranteed to converge to a root
1. The function must be continuous on the interval [a, b]: This means that the function must not have any abrupt changes or discontinuities within the interval. If there are any such points, the bisection method may not converge or may converge to a different root. 2. The function must have opposite signs at the endpoints of the interval [a, b]: This means that the function must evaluate to a positive value at one endpoint and a negative value at the other endpoint. If the function has the same sign at both endpoints, the bisection method will not converge or may converge to a different root.
Describe the cost-vs-accuracy trade-off for iterative root-finding strategies including the impact of solver parameters on cost and accuracy.
1. Tolerance: The tolerance specifies the desired level of accuracy in the solution. A smaller tolerance leads to higher accuracy but requires more iterations and therefore higher cost. In general, the convergence rate of the method determines the number of iterations required to reach a given tolerance, so it is important to choose a tolerance that is appropriate for the convergence rate of the method. 2. Maximum number of iterations: The maximum number of iterations limits the total number of iterations that the solver can perform. Setting a higher maximum number of iterations may improve the accuracy of the solution, but it also increases the cost of the solver. It is important to choose a maximum number of iterations that is large enough to ensure convergence but not too large to waste computational resources. 3. Initial guess: The choice of initial guess can greatly impact the cost and accuracy of the solver. A good initial guess can lead to faster convergence and higher accuracy, while a poor initial guess can lead to slow convergence or failure to converge. It is important to choose an initial guess that is close enough to the true solution to ensure convergence but not too close to avoid numerical instability.
Calculate the condition number of a matrix using built-in functions in numerical software
>> A = [1 2 3; 4 5 6; 7 8 9]; >> c = cond(A) c = 4.97e+15
Know how to solve a linear system Ax = b with full rank A ∈Rn×n using built-in functions in numerical Software
Ax = b x = A \ b;
Explain what big-O notation means for how the cost of a numerical linear algebra algorithm scales with the size of its input
Big-O notation is a mathematical notation used to describe the asymptotic behavior of a function as its input size grows. In the context of numerical linear algebra algorithms, the input size typically refers to the size of the matrix or system of linear equations being solved. Big-O notation is used to describe how the computational cost of the algorithm scales with the size of its input.
Define consistency and determine if a given multi-step scheme is consistent
Consistency is a property of a numerical method for solving a differential equation, which indicates how closely the method approximates the true solution of the differential equation. A method is said to be consistent if the local truncation error of the method approaches zero as the step size approaches zero. In other words, the method is consistent if the error introduced by the method at each time step vanishes as the step size becomes smaller and smaller.
Define eigenvalue stability describe how to find the stability regime for a given multi-step scheme
Eigenvalue stability is another type of stability analysis for numerical methods for solving differential equations, which involves analyzing the behavior of the method for specific test problems. Specifically, it focuses on the stability of the numerical solution for linear differential equations of the form: y'(t) = A y(t)
Describe the relative merits of implicit/explicit schemes
Explicit schemes are easy to implement and computationally efficient because they only require the solution at the current time step to be calculated using information from the previous time step. They are therefore particularly useful when solving problems that have a simple structure and when the time step size is relatively large. Explicit schemes are also easy to parallelize and are well-suited for solving problems with large amounts of data. However, the disadvantage of explicit schemes is that they can be unstable for certain types of problems, particularly for stiff problems where the solution varies rapidly over short time intervals. This can lead to numerical errors and inaccurate results. Implicit schemes: Implicit schemes, on the other hand, are more accurate and stable for stiff problems because they use information from both the current and previous time steps to calculate the solution. This means that they are less likely to produce numerical errors and can handle larger time step sizes. However, implicit schemes can be computationally more expensive and require more computational resources to solve the problem. They are also more difficult to implement and may require the use of iterative methods to solve the equations at each time step.
Define convergence and global order of accuracy
In numerical analysis, convergence refers to the property of a numerical method that the solution produced by the method approaches the true solution of the problem as the step size or time step approaches zero. A method is said to be convergent if the numerical solution approaches the exact solution as the discretization parameter (step size, time step) goes to zero. The global order of accuracy of a numerical method is the rate at which the global error decreases as the step size or time step goes to zero. It is determined by the order of the local truncation error and the consistency of the method. The order of a method is usually denoted by p, and a method is said to be of order p if the global error is proportional to the step size raised to the power of p. For example, a second-order method has a global error that decreases quadratically as the step size goes to zero
Describe the relative advantages and disadvantages of the bisection method and Newton's method.
In summary, the bisection method is simple, robust, and guaranteed to converge, but it may be slow and require many iterations to converge. Newton's method is faster and can converge in fewer iterations, but it requires knowledge of the derivative and may be sensitive to the initial guess and the behavior of the function near the root. Therefore, the choice of method depends on the specific problem and the trade-off between speed, robustness, and accuracy.
Use Taylor series to derive the most accurate timestepping scheme of a given form
In summary, using Taylor series to derive the most accurate timestepping scheme of a given form involves approximating the solution of the differential equation at each time step using a Taylor series expansion, rearranging the expansion to solve for y'(t_n), substituting the resulting expression into the timestepping scheme, and simplifying the resulting expression to obtain the most accurate timestepping scheme of the given form.
Classify schemes as implicit/explicit
In the context of numerical methods for solving ODEs, a scheme is classified as implicit if the value of the solution at the next time step depends on an equation that involves the value of the solution at that time step. Conversely, a scheme is classified as explicit if the value of the solution at the next time step can be computed directly from the values of the solution at the current time step. For example, the forward Euler scheme is an explicit scheme, while the backward Euler scheme is an implicit scheme. The midpoint scheme is also explicit, but the trapezoidal rule is implicit. Higher-order Runge-Kutta methods are typically explicit, but some implicit variants exist as well.
Condition number K
K(x) <= ||A||||A^-1||
Compare and contrast multi-step vs. multi-stage schemes
Multi-step schemes: Multi-step schemes use information from previous time steps to calculate the solution at the current time step. These methods typically require the solution to be computed at several intermediate points to get to the final solution. For example, in the popular Adams-Bashforth method, the solution at the current time step is computed using a weighted average of the solution at previous time steps. The main advantage of multi-step schemes is that they are usually computationally efficient and accurate. Multi-stage schemes: Multi-stage schemes are a type of explicit Runge-Kutta methods that use multiple function evaluations at each time step to approximate the solution. These methods are often used when the differential equation being solved is stiff, which means that the solution varies rapidly over short time intervals. Multi-stage schemes are usually more accurate than multi-step schemes but can be computationally more expensive.
Describe the cost-vs-accuracy trade-off for time-stepping schemes including the influence of number of steps/stages in multi-step/-stage schemes, whether the scheme is implicit/explicit, and structure of the ODE
Multi-step schemes: Multi-step schemes, such as the Adams-Bashforth method, are explicit and use information from previous time steps to calculate the solution at the current time step. The accuracy of multi-step schemes depends on the order of the method, which refers to the number of previous time steps used to calculate the solution. Higher-order methods provide better accuracy but require more computational resources, which increases the cost. Multi-stage schemes: Multi-stage schemes, such as the Runge-Kutta methods, are explicit or implicit and use multiple function evaluations at each time step to approximate the solution. The accuracy of multi-stage schemes depends on the number of stages used to evaluate the function. Higher-order methods with more stages provide better accuracy but also require more computational resources, which increases the cost. Implicit vs. Explicit schemes: Implicit schemes are generally more accurate and stable for stiff problems, but they are also more computationally expensive. Explicit schemes, on the other hand, are computationally efficient and easier to implement, but they can be unstable for certain types of problems. Structure of the ODE: The structure of the ODE being solved also plays a role in the cost-vs-accuracy trade-off. Stiff problems, where the solution varies rapidly over short time intervals, require more accuracy and stability, which often results in higher computational costs. Non-stiff problems, where the solution varies more slowly, can be solved with less accuracy and stability, resulting in lower computational costs.
Describe the focus and importance of numerical linear algebra as a subject
Numerical linear algebra is an essential subject that provides the mathematical tools and techniques for solving a wide range of problems in various fields. It is a vital area of study for scientists, engineers, and researchers who work on computational problems that involve linear algebra.
Use the Dahlquist Equivalence Theorem to assess whether a scheme is convergent
The Dahlquist Equivalence Theorem is a criterion for the convergence of numerical methods for solving initial value problems of ordinary differential equations. According to this theorem, a numerical method is convergent if and only if it is consistent and zero-stable. Consistency means that the local truncation error of the numerical method converges to zero at a rate at least as fast as the step size goes to zero. Zero-stability means that the errors introduced in each step of the numerical method do not grow unboundedly as the number of steps increases.
Explain how to calculate the computational cost of a numerical linear algebra algorithm in terms of FLOPs
The computational cost of a numerical linear algebra algorithm can be measured in terms of the number of floating-point operations (FLOPs) required to perform the algorithm. FLOPs are a standard measure of computational complexity and represent the number of arithmetic operations involving floating-point numbers (such as addition, subtraction, multiplication, and division) that are required to perform a given algorithm.
State the convergence rate of Newton's method and explain what this means for the distance between the current Newton iterate and the true solution as iterations progress
The convergence rate of Newton's method is quadratic, |en+1| <= C|en|^2 This means that as iterations progress, the distance between the current Newton iterate and the true solution decreases rapidly, and the number of correct digits roughly doubles at each iteration
State the convergence rate of the bisection method and explain what this means for the maximum possible error of the bisection method as iterations progress
The convergence rate of the bisection method is linear Specifically, the error reduces by a factor of 1/2 with each iteration. This means that the maximum possible error of the bisection method after n iterations is given by: |error| ≤ (b-a)/2^n
Define the initial value problem for numerical solution of an ODE
The initial value problem for numerical solution of an ordinary differential equation (ODE) involves finding a numerical approximation of the solution of the ODE, given an initial value or set of initial values. Specifically, the initial value problem consists of the following: An ODE of the form y' = f(t, y), where y is the unknown function to be solved, t is the independent variable, and f is a given function. An initial value y(t0) = y0, where t0 is the initial time and y0 is the initial value of y at t0. The goal of the numerical solution of the initial value problem is to find a function y(t) that satisfies the ODE and the initial condition. The solution is typically obtained by discretizing the independent variable t and using a numerical method to approximate the value of y at a set of discrete time points.
Define local order of accuracy for a timestepping scheme
The local order of accuracy for a timestepping scheme is the order of the polynomial approximation used to represent the exact solution of the differential equation over a single time step. In other words, it measures how well a timestepping scheme approximates the solution of the differential equation over a single time step. Specifically, the local order of accuracy is determined by comparing the error of the timestepping scheme at one time step to the error of a more accurate method with one fewer step, assuming both methods use the same step size.
Define the root-finding problem and list a few engineering applications in which it arises
The root-finding problem is a fundamental problem in mathematics and engineering, which involves finding the roots or zeros of a function. f(x) = 0, Control Systems, Optimization, Fluid dynamics
Use Taylor series to analyze the local truncation error of a timestepping scheme
Write the differential equation in the form y' = f(t,y), where y is the solution and f is some function. Assume that the exact solution y(t_{n+1}) can be approximated by a Taylor series expansion about the point t_n:y(t_{n+1}) = y(t_n) + h y'(t_n) + \frac{h^2}{2} y''(t_n) + O(h^3)where h is the step size. Substitute the timestepping scheme into the Taylor series expansion, using the approximation y_{n+1} \approx y(t_{n+1}) and f_n \approx f(t_n, y_n):y_{n+1} = y_n + h f_n + O(h^2) Subtract the timestepping scheme from the exact solution and simplify the expression:y_{n+1} - (y_n + h f_n) = \frac{h^2}{2} y''(t_n) + O(h^3) The left-hand side of the equation is the local truncation error of the timestepping scheme. By analyzing the terms on the right-hand side, the local order of accuracy of the scheme can be determined.
Define zero stability and determine if a given multi-step scheme is zero stable
Zero stability is a property of a numerical method for solving a differential equation, which indicates how well the method can handle small perturbations in the initial data. A method is said to be zero stable if small perturbations in the initial data result in small perturbations in the numerical solution over time. In other words, the method is stable if errors in the numerical solution do not grow exponentially with time.
Floating point representation
is a way of representing real numbers in a computer's memory. In this representation, a real number is stored as a binary fraction with a fixed number of digits (or bits) allocated for the mantissa and the exponent. The mantissa represents the significant digits of the number, while the exponent determines the scale of the number.
For the numerical solution of a linear system Ax = b, quantify how perturbations in b affect the calculated solution x in terms of the condition number of A
||Δx|| / ||x|| ≤ κ(A) ||Δb|| / ||b|| where ||·|| denotes the Euclidean norm, and κ(A) is the condition number of A.
