Suppose that subjects are observed over time and a response, either continuous or categorical, is recorded at multiple time points on each subject. An intervention is applied at a specific time dividing the responses from each subject into measurements pre- and post-intervention. Assessing the effect of the intervention is of primary interest. The manner of assessment depends on the details of the study design and research goals.
In the simplest case, a single measurement is made pre-intervention and again post-intervention, and the difference in these measures assesses the intervention effect. For studies in which measurements are taken at multiple times before and after the intervention, the analysis is often referred to as interrupted time series analysis (ITS) or intervention analysis. One aspect of interest in the ITS analysis is a comparison of the slopes of lines or shapes of curves fitted to the measurements pre- versus post-intervention. When the study includes an independent group of subjects who do not experience the intervention but is selected to be similar to the group receiving the intervention, then it is also of interest to compare the above assessment measures between the treated and untreated groups. For the case of two groups with single measurements before and after intervention, this comparison is the so-called difference in difference (DID), which estimates the difference in the before to after changes in the two groups. The difference in difference analysis is discussed and illustrated in SAS Note 61830.
The following examples illustrate ITS analysis in which subjects are observed longitudinally and measured at multiple time points before and after intervention. Studies with a single group as well as with treated and untreated groups are considered. Cases with a continuous, normally distributed response or a binary response are also discussed. The analyses shown for the binary response cases illustrate the general approach that can be used when the response model is a generalized linear model involving a link function other than the identity link.
When the periods before and after intervention are lengthy with many measurements collected, a time series modeling approach might be of more interest to account for any seasonal or other cyclical patterns. See the discussion of intervention and ITS models in the documentation of the ARIMA procedure in SAS/ETS®.
Related to ITS analysis is the general question of assessing the effect of a continuous predictor involved in interaction or spline effects in the model. That is addressed in SAS Note 67024.
The following statements simulate data from ten subjects measured over six months. An intervention, such as a drug or treatment, is administered between months 3 and 4. A continuous, normal response is measured on each subject at each month. Variable IV is created to indicate either the pre-intervention period (IV=0) or the post-intervention period (IV=1). PROC SGPLOT is then used to plot the data:
data a; sd=3; keep id month iv y; do month=1 to 6; iv=(month>3); input mean @; do ID=1 to 10; Y=rannor(23425)*sd+mean; output; end; end; datalines; 25 35 40 45 70 85 ; proc sgplot; vbox y / category=month group=iv connect=mean intboxwidth=40; refline 3.5 / axis=x; xaxis type=linear; run;
It is assumed that the trends of the response in both periods can be adequately approximated by straight lines. It is of interest to determine whether the intervention affects the period slopes, and also to assess the change in the response in months immediately before and after the intervention. A suitable model is fitted to estimate the intercepts and slopes of those lines and to compare the slopes.
To account for the correlation that is expected among the multiple measurements on a given subject, a Generalized Estimating Equations (GEE) model is used with the subject identifier, ID, in the SUBJECT= option of the REPEATED statement. As discussed in SAS Note 24177, the pre- and post-intervention slopes can be compared by including a common slope term, MONTH, as well as the IV*MONTH interaction, which estimates the difference between the period slopes. The test of this difference parameter is a test of whether the slopes are equal. The results (not shown) indicate that the period slopes differ significantly (p<0.0001):
proc gee data=a; class id iv; model y = iv month iv*month / noint; repeated subject=id; run;
The following statements fit a model with separate intercepts and separate slopes for the two periods. The IV parameters are the estimated intercepts for the periods and the IV*MONTH parameters are the estimated slopes. The EFFECTPLOT statement plots the fitted model. The first two ESTIMATE statements compute the predicted response means at month 3 and 4. Note that the ESTIMATE statement simply defines a vector of coefficients, L, to multiply the vector of parameter estimates, β. It then computes Lβ and provides a test that Lβ=0. With the model as specified with two intercept parameters and two slope parameters, the predicted mean when MONTH=3 is L3=(1 0 3 0), and when MONTH=4 is L4=(0 1 0 4). The change in the response means from month 3 to month 4 is then (L4-L3)β = (-1 1 -3 4)β as specified in the last ESTIMATE statement:
proc gee; class id iv; model y = iv iv*month / noint; repeated subject=id; effectplot slicefit(x=month sliceby=iv) / extend=data clm; estimate 'month 3' iv 1 0 iv*month 3 0 / cl; estimate 'month 4' iv 0 1 iv*month 0 4 / cl; estimate 'month 4 - month 3 diff' iv -1 1 iv*month -3 4 / cl; run;
The following results show that slope before intervention is estimated to be 7.4, and after intervention is 20.7. The previous analysis found that these slopes differ significantly. The change in response mean from month 3 to month 4 is estimated to be 45.3-40.5 = 4.8 and differs significantly from zero (p=0.0039) indicating a significant increase in the mean:
Similar to the continuous response example above, the following statements simulate and plot binary response data, but this time from 300 subjects over the six month period. The plot shows the observed probability of the response (Y=1) in each month. Unlike in the previous example, note that the months are numbered identically within the periods as MONTH=1, 2, or 3. The reason for this is discussed below:
data a; keep id month iv y; do iv=0,1; do month=1 to 3; input mean @; do ID=1 to 300; Y=ranbin(23425,1,mean); output; end; end; end; datalines; .25 .35 .40 .45 .70 .85 ; proc summary data=a nway; class iv month; var y; output out=mns mean=; run; proc sgpanel; panelby iv; series y=y x=month / markers; colaxis type=discrete; run;
As above, a GEE model can be fit with tests for the equality of slopes and estimation of the response change in the months just before and after the intervention. However, since the model for the binary response is a logistic model that uses the logit link, all estimates and tests are on the link (log odds) scale, not on the mean (event probability) scale. In the first analysis below, the test of the IV*MONTH parameter is a test for equal slopes on the log odds.
Recall that in this example the months are numbered identically in each period. So, the coefficient for the first month in the post-intervention period is 1 instead of 4 as in the previous example, and this affects the ESTIMATE coefficients for that period. The results from the second analysis (not shown) indicate that the period slopes on the log odds scale are 0.3 and 1. These differ significantly as noted from the first analysis. The odds ratio from month 3 to 4 is 1.08, which does not differ significantly from 1. This is illustrated by the plot on the log odds (link or linear predictor) scale from the second EFFECTPLOT statement, shown below. The STORE statement is used to save the fitted model for later use:
proc gee data=a; class id iv; model y(event="1") = iv month iv*month / dist=bin noint; repeated subject=id; run; proc gee data=a; class id iv; model y(event="1") = iv iv*month / dist=bin noint; repeated subject=id; effectplot fit(x=month plotby(pack)=iv) / clm; effectplot fit(x=month plotby(pack)=iv) / link clm; estimate 'm3 log odds' iv 1 0 iv*month 3 0 / ilink; estimate 'm4 log odds' iv 0 1 iv*month 0 1 / ilink; estimate 'm3 to m4 odds ratio' iv -1 1 iv*month -3 1 / exp cl; store gee; run;
The following plot from the first EFFECTPLOT statement above shows the fitted model on the probability scale. Note that the fitted model is not linear in each period like it is on the log odds scale:
Estimation and testing on the event probability scale can be done using the Margins macro (SAS Note 63038). The macro both fits the model using PROC GENMOD and estimates requested quantities on the probability scale. To use the macro for this analysis, it is important that the levels of the continuous variable referencing the repeated measurements, MONTH in this example, have the same set of values in both the pre- and the post-intervention periods. As noted above, MONTH has values 1, 2, and 3 in each period.
The following statements call the Margins macro to fit the same logistic GEE model as above. However, the macro uses PROC GENMOD rather than PROC GEE. effect=month, margins=iv requests estimation of the average marginal effect in each period and their difference. To estimate the after - before change in period slopes, diff=all, options=reverse is also specified:
%margins(data=a, class=id iv, response=y, roptions=event="1", model=iv iv*month, dist=binomial, modelopts=noint, geesubject=id, effect=month, margins=iv, diff=all, options=reverse)
The first two tables give the overall average event probability in each period and their difference, though this might not be of as much interest. The average marginal effect in each period is obtained by computing the slope of the probability curve at each of the three months in the period (that is, the slope of a line drawn tangent to the probability curve at each month) and averaging them. The final two tables show the average marginal effect of MONTH in the two periods and their difference. These are the estimated period slopes. The difference, 0.1325, shows that the post-intervention slope significantly exceeds the pre-intervention slope (p<0.0001):
An additional call of the macro can be used to estimate and test the change in the event probability from month 3 to 4. The event probability at each month is requested by margins=iv month, and diff=3 requests all comparisons with month 3. The desired difference is included with others that might not be of interest. options=reverse reverses the direction of the difference so that the probability in month 3 is subtracted from each of the others:
%margins(data=a, class=id iv, response=y, roptions=event="1", model=iv iv*month, dist=binomial, modelopts=noint, geesubject=id, margins=iv month, diff=3, options=reverse)
The Predictive Margins table shows the predicted event probabilities in each month in each period. The Differences table provides the requested differences with the probability in month 3. In particular, the difference in the months immediately before and after intervention is 0.0194 and does not differ significantly from zero (p=0.5953):
If the model is not supported by PROC GENMOD, then the Margins macro cannot be used. Examples are models with random effects or with spline effects. In many cases the analysis can instead be done using the ESTIMATE statement in the modeling procedure followed by the NLMeans macro (SAS Note 62362).
To illustrate, the above analysis is repeated using NLMeans. The saved GEE model is recovered by PROC PLM and the ESTIMATE statement with the ILINK option estimates the event probabilities in each month (not shown). The E option produces a table that contains the coefficients used in the ESTIMATE statement and the ODS OUTPUT statement saves the table of coefficients in a data set:
proc plm restore=gee; estimate 'm1' iv 1 0 iv*month 1 0, 'm2' iv 1 0 iv*month 2 0, 'm3' iv 1 0 iv*month 3 0, 'm4' iv 0 1 iv*month 0 1, 'm5' iv 0 1 iv*month 0 2, 'm6' iv 0 1 iv*month 0 3 / e ilink; ods output coef=coef; run;
The NLMeans macro can then read the saved model and coefficients and estimate the period slopes and their difference. This is done by creating a data set in which each observation defines the desired contrast of the six event probabilities. See the NLMeans macro documentation for details:
data c; length label $32767; infile datalines dsd; input label k1-k6; set=1; datalines; pre slope, -.5, 0, .5, 0, 0, 0 post slope, 0, 0, 0, -.5, 0, .5 post-pre slope diff, .5, 0, -.5, -.5, 0, .5 1 month change m4-m3, 0, 0, -1, 1, 0, 0 ; %nlmeans(instore=gee, coef=coef, link=logit, contrasts=c, title=Period slopes and differences)
The results from the NLMeans macro are substantially similar to those from the Margins macro:
By including a control group, in which subjects are observed longitudinally but are never exposed to the intervention, any unrelated trend can be estimated and used to adjust the trend seen in the treated group. The control group should be similar to the treated group. Of interest is estimating and comparing slopes, for both groups, in the periods before and after the intervention time. In addition to comparing the before and after slopes, the group slopes within each period can be compared. Similar to a difference in difference analysis, the differences of the slope differences can be estimated and tested. Finally, similar to above, the probability change in the two months around the intervention time can be estimated in each group and compared.
These statements simulate data from treated and control groups, each with ten subjects, observed over six months with the treated group receiving the intervention between months 3 and 4 as in the previous examples. The response is normally distributed. IV indicates pre- (0) and post-intervention (1) periods. EXPOSED indicates the treated (1) and control (0) groups. EXPIV just combines the group and period variables into a single variable with four levels. PROC SGPANEL plots the data:
data a; sd=3; keep id exposed month iv expiv y; do exposed=1,0; do month=1 to 6; iv=(month>3); expiv=cats(exposed,'_',iv); input mean @; do ID=1 to 10; Y=rannor(23425)*sd+mean; output; end; end; end; datalines; 26 30 32 45 52 63 27 31 33 29 32 36 ; proc sgplot; vbox y / category=month group=exposed groupdisplay=overlay connect=mean intboxwidth=40; refline 3.5 / axis=x; xaxis type=linear; run;
The statements below fit a GEE model similar to the single group example above, but now using EXPIV in the CLASS statement so that the model estimates four intercepts and four slopes—one pair for each of the group-period combinations. If it is desired to test that all four slopes are equal, MONTH could be included in the model as a common slope. If the TYPE3 option is also added in the MODEL statement, then the type 3 test of the interaction is a test that all four slopes are equal.
The EFFECTPLOT statement plots the fitted model and the multi-row ESTIMATE statement estimates each of the differences discussed above:
proc gee data=a; class id expiv; model y = expiv expiv*month / noint; repeated subject=id; effectplot slicefit(x=month sliceby=expiv) / extend=data clm; estimate 'compare group slopes, pre' expiv*month -1 0 1 0, 'compare group slopes, post' expiv*month 0 -1 0 1, 'compare period slopes, unexposed' expiv*month -1 1 0 0, 'compare period slopes, exposed' expiv*month 0 0 -1 1, 'compare post-pre slope changes' expiv*month 1 -1 -1 1, '1 month change unexposed' expiv -1 1 0 0 expiv*month -3 4 0 0, '1 month change exposed' expiv 0 0 -1 1 expiv*month 0 0 -3 4, '1 month change diff' expiv 1 -1 -1 1 expiv*month 3 -4 -3 4; run;
The first table in the results, the Parameter Estimates table, shows the four intercepts (EXPIV parameters) and the four slopes (interaction parameters). The labels next to the Estimate column indicate the group-period combination. For example, 0_1 indicates the unexposed, control group (0) in the post-intervention period (1). The plot of the fitted model shows the four group-period lines.
In the Estimates table following the plot, note that the control and treated group slopes do not differ before intervention (p=0.4404) but do differ after intervention (p<0.0001). Within the control group, the slopes pre- and post-intervention do not differ (p=0.9369) but do differ in the treated group (p<0.0001). The difference in the slope differences (whether regarded as differences of group slope differences or of period slope differences) is significant (p<0.0001). That is, the change in slopes in the treated group is 6.74, but when adjusted for the change seen in the control group (0.07), it slightly diminishes to 6.67:
The above analysis could also be done using the Margins macro in the same way as shown in the following example with a binary response. For this example with a normal response, dist=normal would be specified in the macro calls. Similarly, if the model on the response is a generalized linear model with link function other than the identity link (such as a model for a gamma distributed response using log link), the analysis on the mean scale would be performed similar to the following with appropriate specification of the response distribution and link function in the Margins or modeling procedure and NLMeans macro call:
Similar to the single group example above with a binary response, the following simulates data for 300 subjects in each of two groups, treated and control, observed over six months with the treated group receiving the intervention between months 3 and 4. EXPOSED and EXPIV variables are created in the same way as for the two group, continuous response example preceding:
data a; keep id month exposed iv expiv y; do exposed=1,0; do iv=0,1; do month=1 to 3; expiv=cats(exposed,'_',iv); input mean @; do ID=1 to 300; Y=ranbin(23425,1,mean); output; end; end; end; end; datalines; .26 .30 .32 .45 .52 .63 .27 .31 .33 .29 .32 .36 ; proc summary data=a nway; class iv exposed month; var y; output out=mns mean=; run; proc sgpanel; panelby iv; series y=y x=month / group=exposed markers; colaxis type=discrete; run;
As for the single group case above, an analysis on the probability (mean) scale can be performed using the Margins macro. The following fits a GEE logistic model of the same form as in the previous example with separate intercepts and slopes for the four combinations of group and period (not shown). The average marginal effect of MONTH in each group and period is requested by effect=month and margins=expiv providing estimates of the slopes in each group and period. Comparisons of period slopes and group slopes are requested by diff=all, options=reverse cl, which also requests confidence intervals and sets the order of the differencing.
An adjusted estimate of the intervention effect on the slope can be obtained by reducing the slope change between the two periods in the treated (exposed) group by the change in the control group. It is the difference in difference of group slopes and can be estimated using a contrast on the four slopes. The contrast is defined in data set C with the coefficients using the order of the marginal effects as displayed by the macro. Estimation of the contrast is requested by adding contrasts=c in the Margins macro call:
data c; length label f $32767; infile datalines delimiter='|'; input label f; datalines; diff post-pre slope change | 1 -1 -1 1 ; %margins(data=a, class=id expiv, response=y, roptions=event="1", model=expiv expiv*month, dist=binomial, geesubject=id, effect=month, margins=expiv, contrasts=c, diff=all, options=reverse cl)
The first table shows the slopes in each group and period. The post-intervention slope for the treated group (EXPIV=1_1) is 0.094 and is significantly different from zero (p<0.0001), while in the pre-intervention period the slope is 0.023 and does not differ from zero (p=0.1958). The difference of these two slopes is labeled 4 - 3 (using the Index values in the first table) in the Differences table and is estimated as 0.071. The small p-value (p=0.0074) indicates that these two slopes are significantly different. Finally, in the Contrasts table, 0.039 is the estimated adjusted difference in the effect of the intervention on the slope in the treated group and does not differ significantly from zero (p=0.2928):
In the following call, margins=month expiv requests the predictive margins (predicted event probabilities) for each group in each month. To avoid redisplaying the fitted model, nomodel is specified in options=:
%margins(data=a, class=id expiv, response=y, roptions=event="1", model=expiv expiv*month, dist=binomial, geesubject=id, margins=month expiv, options=cl nomodel)
To estimate the immediate change in probability in each group from month 3 to 4 and compare them, contrasts of these probabilities can be created. For example, for the unexposed (control) group, this is the change from month 3 in the pre-intervention period (MONTH=3 EXPIV=0_0) to month 4 in the post-intervention period (MONTH=1 EXPIV=0_1). Recall that months 4, 5, and 6 are numbered 1, 2, and 3 with IV=1 indicating the post-intervention period. So, the desired estimate is the difference as defined by the coefficients in the first observation of data set C below. Similarly, the second row defines the change in the exposed group. The third row is simply the exposed-unexposed difference of the two previous contrasts. In the Margins macro, contrasts=c is specified to estimate and test the contrasts:
data c; length label f $32767; infile datalines delimiter='|'; input label f; datalines; 1 month change unexposed | 0 1 0 0 0 0 0 0 -1 0 0 0 1 month change exposed | 0 0 0 1 0 0 0 0 0 0 -1 0 1 month change diff | 0 -1 0 1 0 0 0 0 1 0 -1 0 ; %margins(data=a, class=id expiv, response=y, roptions=event="1", model=expiv expiv*month, dist=binomial, geesubject=id, margins=month expiv, contrasts=c, options=cl nomodel)
The event probability declines from month 3 to month 4 by 0.11 in the unexposed control group but increases by 0.09 in the treated group. Both changes are found to differ significantly from zero. The difference in these differences, which is the change in the treated group adjusted for the change in the control group, is 0.2 and differs significantly from zero (p<0.0001):
As for the single group analysis above, if the model is not supported by PROC GENMOD as used by the Margins macro, the analysis can usually be performed using the NLMeans macro. The following uses PROC GLIMMIX to fit a GEE type model similar to the one fit by PROC GENMOD via the Margins macro.
In the following, the ESTIMATE statement with the ILINK option is used to obtain the event probability estimates at the beginning and end of each period (months 1 and 3) in each group. The fitted model is saved by the STORE statement and the ESTIMATE statement coefficients are saved with an ODS OUTPUT statement. These can then be used in the macro.
A set of contrasts that defines the same slopes, slope differences, one month pre- to post-intervention probability changes and difference as above is specified in data set C. The NLMeans macro can then provide an estimate and test of each contrast. The results (not shown) are very similar to those above from the Margins macro:
proc glimmix data=a empirical; class id expiv; model y(event="1") = expiv expiv*month / dist=binary noint s ddfm=none; random _residual_ / subject=id; estimate 'unexp m1' expiv 1 0 0 0 expiv*month 1 0 0 0, 'unexp m3' expiv 1 0 0 0 expiv*month 3 0 0 0, 'unexp m4' expiv 0 1 0 0 expiv*month 0 1 0 0, 'unexp m6' expiv 0 1 0 0 expiv*month 0 3 0 0, 'exp m1' expiv 0 0 1 0 expiv*month 0 0 1 0, 'exp m3' expiv 0 0 1 0 expiv*month 0 0 3 0, 'exp m4' expiv 0 0 0 1 expiv*month 0 0 0 1, 'exp m6' expiv 0 0 0 1 expiv*month 0 0 0 3 / e ilink; ods output coef=coef; store gmx; run; data c; length label $32767; infile datalines dsd; input label k1-k8; set=1; datalines; unexp pre slope, -.5, .5, 0, 0, 0, 0, 0, 0 unexp post slope, 0, 0, -.5, .5, 0, 0, 0, 0 unexp slope change, .5, -.5, -.5, .5, 0, 0, 0, 0 exp pre slope, 0, 0, 0, 0, -.5, .5, 0, 0 exp post slope, 0, 0, 0, 0, 0, 0, -.5, .5 exp slope change, 0, 0, 0, 0, .5, -.5, -.5, .5 unexp Post-Pre change, 0, -1, 1, 0, 0, 0, 0, 0 exp Post-Pre change, 0, 0, 0, 0, 0, -1, 1, 0 adjusted exp change, 0, 1, -1, 0, 0, -1, 1, 0 ; %nlmeans(instore=gmx, coef=coef, link=logit, contrasts=c, title=ITS analysis: slopes and comparisons)
Product Family | Product | System | SAS Release | |
Reported | Fixed* | |||
SAS System | SAS/STAT | z/OS | ||
z/OS 64-bit | ||||
OpenVMS VAX | ||||
Microsoft® Windows® for 64-Bit Itanium-based Systems | ||||
Microsoft Windows Server 2003 Datacenter 64-bit Edition | ||||
Microsoft Windows Server 2003 Enterprise 64-bit Edition | ||||
Microsoft Windows XP 64-bit Edition | ||||
Microsoft® Windows® for x64 | ||||
OS/2 | ||||
Microsoft Windows 8 Enterprise 32-bit | ||||
Microsoft Windows 8 Enterprise x64 | ||||
Microsoft Windows 8 Pro 32-bit | ||||
Microsoft Windows 8 Pro x64 | ||||
Microsoft Windows 8.1 Enterprise 32-bit | ||||
Microsoft Windows 8.1 Enterprise x64 | ||||
Microsoft Windows 8.1 Pro 32-bit | ||||
Microsoft Windows 8.1 Pro x64 | ||||
Microsoft Windows 10 | ||||
Microsoft Windows 11 | ||||
Microsoft Windows 95/98 | ||||
Microsoft Windows 2000 Advanced Server | ||||
Microsoft Windows 2000 Datacenter Server | ||||
Microsoft Windows 2000 Server | ||||
Microsoft Windows 2000 Professional | ||||
Microsoft Windows NT Workstation | ||||
Microsoft Windows Server 2003 Datacenter Edition | ||||
Microsoft Windows Server 2003 Enterprise Edition | ||||
Microsoft Windows Server 2003 Standard Edition | ||||
Microsoft Windows Server 2003 for x64 | ||||
Microsoft Windows Server 2008 | ||||
Microsoft Windows Server 2008 R2 | ||||
Microsoft Windows Server 2008 for x64 | ||||
Microsoft Windows Server 2012 Datacenter | ||||
Microsoft Windows Server 2012 R2 Datacenter | ||||
Microsoft Windows Server 2012 R2 Std | ||||
Microsoft Windows Server 2012 Std | ||||
Microsoft Windows Server 2016 | ||||
Microsoft Windows Server 2019 | ||||
Microsoft Windows Server 2022 | ||||
Microsoft Windows XP Professional | ||||
Windows 7 Enterprise 32 bit | ||||
Windows 7 Enterprise x64 | ||||
Windows 7 Home Premium 32 bit | ||||
Windows 7 Home Premium x64 | ||||
Windows 7 Professional 32 bit | ||||
Windows 7 Professional x64 | ||||
Windows 7 Ultimate 32 bit | ||||
Windows 7 Ultimate x64 | ||||
Windows Millennium Edition (Me) | ||||
Windows Vista | ||||
Windows Vista for x64 | ||||
64-bit Enabled AIX | ||||
64-bit Enabled HP-UX | ||||
64-bit Enabled Solaris | ||||
ABI+ for Intel Architecture | ||||
AIX | ||||
HP-UX | ||||
HP-UX IPF | ||||
IRIX | ||||
Linux | ||||
Linux for x64 | ||||
Linux on Itanium | ||||
OpenVMS Alpha | ||||
OpenVMS on HP Integrity | ||||
Solaris | ||||
Solaris for x64 | ||||
Tru64 UNIX |
Type: | Usage Note |
Priority: | |
Topic: | Analytics ==> Categorical Data Analysis Analytics ==> Regression SAS Reference ==> Macro SAS Reference ==> Procedures ==> GEE SAS Reference ==> Procedures ==> GENMOD SAS Reference ==> Procedures ==> GLIMMIX |
Date Modified: | 2023-11-17 09:13:30 |
Date Created: | 2023-11-08 13:15:56 |