8 Single Run Benchmark

In this section, we will measure the performance differences among r, python and stata, specifically the single_run function of each software, which is the atomic function of whole power simulation program. But, due to the differences of parameters selection, the measurement is not technically subjective, the benchmark results here are for reference only.

8.1 R

In implementation with R, we used all parameters (\(\beta_0\), \(\beta_1\), \(\omega_0\), \(\tau_0\), \(\tau_1\), \(\rho\), \(\sigma\)) to generate simulated data and used the function lmer() from the package {lmerTest} to perform linear mixed effects analyze.

Here are the definitions of functions sim_data and single_run.

We’ll use benchmark function from package {rbenchmark} to measure the running time of function single_run.

library(rbenchmark)

result <- within(
  benchmark(
    single_run = single_run(),
    replications = 100,
    columns = c("test", "elapsed", "replications")
  ),
  {
    average <- elapsed / replications
  }
)

r_average <- result[1, "average"]

sprintf("Average running time is %f s", r_average)

## [1] "Average running time is 0.128400 s"

8.2 Python

In implementation with Python, we used all parameters (\(\beta_0\), \(\beta_1\), \(\omega_0\), \(\tau_0\), \(\tau_1\), \(\rho\), \(\sigma\)) to generate simulated data and used the function mixedlm() from package {statsmodels} to perform linear mixed effects analyze.

Here are the definitions of functions sim_data and single_run.

We’ll use timeit function from the package {timeit} to measure the running time of function single_run.

import timeit

result = timeit.timeit(single_run, number=100)
python_average = result/100

print(f"Average running time is {python_average} s")

## Average running time is 0.11294644300010986 s

8.3 Stata

In implementation with Stata, we only used 4 parameters (\(\beta_0\), \(\beta_1\), \(\tau_0\), \(\sigma\)) to generate simulated data and used the function mixed to perform linear mixed effects analyze.

Here are the definitions of functions sim_data and single_run.

In stata, we have to implement the benchmark code by our own.

capture program drop tstart
program tstart, rclass
  timer clear 1
  timer on 1
end

capture program drop tend
program tend, rclass
  timer off 1
  qui timer list 1
  scalar r = r(t1)
end

tstart
quietly {
    forval i = 1/100 {
        single_run 25 15 15 60 5 7 8
    }
}
tend
local result = scalar(r)
local stata_average = r / 100
di "Average running time is " `stata_average' " s"

quietly {
    file open result using "./data/stata_average.txt", write replace
    set more off

    file write result "`stata_average'"
    file close result
    set more on
}

Average running time is .09861 s

8.4 Benchmark Results

Finally, we get all the results of r, python and stata, let’s show them with a table.

Table 8.1: Benchmark results.

Software	Parameters	Average Time
R	\(\beta_0\), \(\beta_1\), \(\omega_0\), \(\tau_0\), \(\tau_1\), \(\rho\), \(\sigma\)	0.13
Python	\(\beta_0\), \(\beta_1\), \(\omega_0\), \(\tau_0\), \(\tau_1\), \(\rho\), \(\sigma\)	0.11
Stata	\(\beta_0\), \(\beta_1\), \(\tau_0\), \(\sigma\)	0.10