xref: /petsc/doc/developers/callbacks.md (revision 0baf8eba40dbc839082666f9f7396a225d6f663c) 
1# How the Solvers Handle User Provided Callbacks
2
3The solver objects in PETSc, `KSP` (optionally), `SNES`, and `TS`
4require user provided callback functions (and contexts for the
5functions) that define the problem to be solved. These functions are
6supplied by the user with calls such as `SNESSetFunction(SNES,...)`
7and `TSSetRHSFunction(TS,...)`. One would naturally think that the
8functions provided would be attached to the appropriate solver object,
9that is, that the SNES callbacks would be attached to the `SNES`
10object and `TS` callbacks to the `TS` object. This is not the case.
11Or possibly one might think the callbacks would be attached to the
12`DM` object associated with the solver object. This is also not the
13case. Rather, the callback functions are attached to an inner nonpublic
14`DMXXX` object (`XXX` is `KSP`, `SNES`, or `TS`) that is
15attached to the `DM` that is attached to the `XXX` solver object.
16This convoluted design is to support multilevel and multidomain solvers
17where different levels and different domains may (or may not) share the
18same callback function or callback context. You can control exactly what
19`XXX`/`DM` objects share a common `DMXXX` object.
20
21:::{figure} /images/developers/callbacks1.svg
22:name: fig_callbacks1
23
24Three levels of KSP/DM share the same DMKSP
25:::
26
27In the preceding figure, we depict how three levels of `KSP`
28objects share a common `DMKSP` object. The code to access the inner
29`DMKSP` object is
30
31```
32DM    dm_2;
33DMKSP dmksp;
34KSPGetDM(ksp_2,&dm_2);
35DMGetDMKSP(dm_2,&dmksp);
36```
37
38To obtain a new DMKSP object for which you can change the callback
39functions (or their contexts) without affecting the original DMKSP, call
40
41```
42DM    dm_2;
43DMKSP dmksp;
44KSPGetDM(ksp_2,&dm_2);
45DMGetDMKSPWrite(dm_2,&dmksp_2);
46```
47
48This results in the object organization as indicated in the following figure
49
50:::{figure} /images/developers/callbacks2.svg
51:name: fig_callbacks2
52
53Two levels of KSP/DM share the same DMKSP; one has its own private copy
54:::
55
56The `DMKSP` object is essentially the list of callback functions and
57their contexts, for example,
58
59```
60typedef struct _p_DMKSP *DMKSP;
61typedef struct _DMKSPOps *DMKSPOps;
62struct _DMKSPOps {
63  PetscErrorCode (*computeoperators)(KSP,Mat,Mat,void*);
64  PetscErrorCode (*computerhs)(KSP,Vec,void*);
65  PetscErrorCode (*computeinitialguess)(KSP,Vec,void*);
66  PetscErrorCode (*destroy)(DMKSP*);
67  PetscErrorCode (*duplicate)(DMKSP,DMKSP);
68};
69
70struct _p_DMKSP {
71  PETSCHEADER(struct _DMKSPOps);
72  void *operatorsctx;
73  void *rhsctx;
74  void *initialguessctx;
75  void *data;
76  DM originaldm;
77
78  void (*fortran_func_pointers[3])(void); /* Store our own function pointers so they are associated with the DMKSP instead of the DM */
79};
80```
81
82We now explore in more detail exactly how the solver calls set by the
83user are passed down to the inner `DMKSP` object. For each user level
84solver routine for setting a callback a similar routine exists at the
85`DM` level. Thus, `XXXSetY(XXX,...)` has a routine
86`DMXXXSetY(DM,...)`.
87
88```
89PetscErrorCode KSPSetComputeOperators(KSP ksp,PetscErrorCode (*func)(KSP,Mat,Mat,void*),void *ctx)
90{
91  DM dm;
92
93  PetscFunctionBegin;
94  PetscValidHeaderSpecific(ksp,KSP_CLASSID,1);
95  PetscCall(KSPGetDM(ksp,&dm));
96  PetscCall(DMKSPSetComputeOperators(dm,func,ctx));
97  if (ksp->setupstage == KSP_SETUP_NEWRHS) ksp->setupstage = KSP_SETUP_NEWMATRIX;
98  PetscFunctionReturn(PETSC_SUCCESS);
99}
100```
101
102The implementation of `DMXXXSetY(DM,...)` gets a “writable” version of
103the `DMXXX` object via `DMGetDMXXXWrite(DM,DMXXX*)` and sets the
104function callback and its context into the `DMXXX` object.
105
106```
107PetscErrorCode DMKSPSetComputeOperators(DM dm,PetscErrorCode (*func)(KSP,Mat,Mat,void*),void *ctx)
108{
109  DMKSP kdm;
110
111  PetscFunctionBegin;
112  PetscValidHeaderSpecific(dm,DM_CLASSID,1);
113  PetscCall(DMGetDMKSPWrite(dm,&kdm));
114  if (func) kdm->ops->computeoperators = func;
115  if (ctx) kdm->operatorsctx = ctx;
116  PetscFunctionReturn(PETSC_SUCCESS);
117}
118```
119
120The routine for `DMGetDMXXXWrite(DM,DMXXX*)` entails a duplication of
121the object unless the `DM` associated with the `DMXXX` object is the
122original `DM` that the `DMXXX` object was created with. This can be
123seen in the following code.
124
125```
126PetscErrorCode DMGetDMKSPWrite(DM dm,DMKSP *kspdm)
127{
128  DMKSP kdm;
129
130  PetscFunctionBegin;
131  PetscValidHeaderSpecific(dm,DM_CLASSID,1);
132  PetscCall(DMGetDMKSP(dm,&kdm));
133  if (!kdm->originaldm) kdm->originaldm = dm;
134  if (kdm->originaldm != dm) {  /* Copy on write */
135    DMKSP oldkdm = kdm;
136    PetscCall(PetscInfo(dm,"Copying DMKSP due to write\n"));
137    PetscCall(DMKSPCreate(PetscObjectComm((PetscObject)dm),&kdm));
138    PetscCall(DMKSPCopy(oldkdm,kdm));
139    PetscCall(DMKSPDestroy((DMKSP*)&dm->dmksp));
140    dm->dmksp = (PetscObject)kdm;
141    kdm->originaldm = dm;
142  }
143  *kspdm = kdm;
144  PetscFunctionReturn(PETSC_SUCCESS);
145}
146```
147
148The routine `DMGetDMXXX(DM,DMXXX*)` has the following form.
149
150```
151PetscErrorCode DMGetDMKSP(DM dm,DMKSP *kspdm)
152{
153  PetscFunctionBegin;
154  PetscValidHeaderSpecific(dm,DM_CLASSID,1);
155  *kspdm = (DMKSP) dm->dmksp;
156  if (!*kspdm) {
157    PetscCall(PetscInfo(dm,"Creating new DMKSP\n"));
158    PetscCall(DMKSPCreate(PetscObjectComm((PetscObject)dm),kspdm));
159    dm->dmksp = (PetscObject) *kspdm;
160    (*kspdm)->originaldm = dm;
161    PetscCall(DMCoarsenHookAdd(dm,DMCoarsenHook_DMKSP,NULL,NULL));
162    PetscCall(DMRefineHookAdd(dm,DMRefineHook_DMKSP,NULL,NULL));
163  }
164  PetscFunctionReturn(PETSC_SUCCESS);
165}
166```
167
168This routine uses `DMCoarsenHookAdd()` and `DMRefineHookAdd()` to
169attach to the `DM` object two functions that are automatically called
170when the object is coarsened or refined. The hooks
171`DMCoarsenHook_DMXXX()` and `DMRefineHook_DMXXX()` have the same form:
172
173```
174static PetscErrorCode DMCoarsenHook_DMKSP(DM dm,DM dmc,void *ctx)
175{
176  PetscFunctionBegin;
177  PetscCall(DMCopyDMKSP(dm,dmc));
178  PetscFunctionReturn(PETSC_SUCCESS);
179}
180```
181
182where
183
184```
185PetscErrorCode DMCopyDMKSP(DM dmsrc,DM dmdest)
186{
187  PetscFunctionBegin;
188  PetscValidHeaderSpecific(dmsrc,DM_CLASSID,1);
189  PetscValidHeaderSpecific(dmdest,DM_CLASSID,2);
190  PetscCall(DMKSPDestroy((DMKSP*)&dmdest->dmksp));
191  dmdest->dmksp = dmsrc->dmksp;
192  PetscCall(PetscObjectReference(dmdest->dmksp));
193  PetscCall(DMCoarsenHookAdd(dmdest,DMCoarsenHook_DMKSP,NULL,NULL));
194  PetscCall(DMRefineHookAdd(dmdest,DMRefineHook_DMKSP,NULL,NULL));
195  PetscFunctionReturn(PETSC_SUCCESS);
196}
197```
198
199ensures that the new `DM` shares the same `DMXXX` as the parent
200`DM` and also inherits the hooks if it is refined or coarsened.
201
202If you provide callbacks to a solver *after* the `DM` associated with
203a solver has been refined or coarsened, those child `DM`s will not
204share a common `DMXXX`.
205
206The `TS` object manages its callback functions in a way similar to
207`KSP` and `SNES`, although there are no multilevel `TS`
208implementations so in theory the `DMTS` object is currently unneeded.
209