FINERACT-2668: Stop bulk-import loan-repayment template loading the entire tenant#6071
Open
oluexpert99 wants to merge 1 commit into
Open
FINERACT-2668: Stop bulk-import loan-repayment template loading the entire tenant#6071oluexpert99 wants to merge 1 commit into
oluexpert99 wants to merge 1 commit into
Conversation
…ntire tenant
- Downloading the loan-repayment template (GET /v1/loans/repayments/downloadtemplate) eagerly fetched every
client (ClientReadPlatformService.retrieveAll(null) applies no LIMIT) and every active loan, embedding them
as a tenant-wide Clients sheet, a per-loan lookup table and a 3000-row x 5 VLOOKUP block. The cost is
O(tenant size) -- multi-MB .xls, slow Excel open, and server heap pressure / OOM risk under concurrent
downloads -- for an operation that is conceptually O(rows entered). The import handler never reads the
Clients/Offices sheets; it resolves the loan from the typed account number
(LoanRepaymentImportHandler -> retrieveLoanIdByAccountNumber), so the embedded dataset is discarded at import.
- Add an optional lean template, selected by a new LookupMode {AUTO, INCLUDE, EXCLUDE}.
BulkImportWorkbookPopulatorService gains a 5-arg getTemplate(..., LookupMode); the existing 4-arg signature
becomes a default delegating with AUTO, so all other callers and templates are unchanged.
- LoansApiResource#getLoanRepaymentTemplate exposes ?includeLookups={true|false}, mapped to a LookupMode
(null -> AUTO, true -> INCLUDE, false -> EXCLUDE). INCLUDE keeps the previous unbounded behaviour (the caller
accepts the full cost). EXCLUDE always produces a lean template. AUTO bounds the client/loan fetch with a SQL
LIMIT and degrades to lean when either set exceeds fineract.bulk-import.template.max-lookup-rows
(default 10000), surfaced through FineractProperties.
- LoanRepaymentWorkbookPopulator gains a lean mode (static lean()/full() factories) that omits the
client/office sheets, the per-loan lookup table and the 3000-row VLOOKUP block, while keeping the data-entry
column layout intact (the import handler parses by fixed column index) and retaining the payment-type
dropdown sourced from the small Extras sheet (extracted to a shared addPaymentTypeValidation()).
- Add unit tests: LookupMode mapping; the max-lookup-rows cap boundary; and lean-mode output asserting the
Clients/Offices sheets and per-loan lookup rows are omitted while the data-column headers are preserved.
Signed-off-by: oluexpert99 <farooq@techservicehub.io>
adamsaghy
reviewed
Jul 1, 2026
|
|
||
| // FINERACT-2668: max tenant-wide lookup rows (clients/loans) embedded in a download template before it | ||
| // auto-degrades to a lean form (typed keys only). Honoured by templates that opt in (loan repayment). | ||
| private int maxLookupRows = 10000; |
Contributor
There was a problem hiding this comment.
No need for hardcoded value
Contributor
|
I am having some concerns... limiting the number of fetched data, limiting the data you got... What if you need all 20k clients or loan? Maybe i dont fully understand this functionality... can you please help me to understand what these |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
client (ClientReadPlatformService.retrieveAll(null) applies no LIMIT) and every active loan, embedding them
as a tenant-wide Clients sheet, a per-loan lookup table and a 3000-row x 5 VLOOKUP block. The cost is
O(tenant size) -- multi-MB .xls, slow Excel open, and server heap pressure / OOM risk under concurrent
downloads -- for an operation that is conceptually O(rows entered). The import handler never reads the
Clients/Offices sheets; it resolves the loan from the typed account number
(LoanRepaymentImportHandler -> retrieveLoanIdByAccountNumber), so the embedded dataset is discarded at import.
- Add an optional lean template, selected by a new LookupMode {AUTO, INCLUDE, EXCLUDE}.
BulkImportWorkbookPopulatorService gains a 5-arg getTemplate(..., LookupMode); the existing 4-arg signature
becomes a default delegating with AUTO, so all other callers and templates are unchanged.
- LoansApiResource#getLoanRepaymentTemplate exposes ?includeLookups={true|false}, mapped to a LookupMode
(null -> AUTO, true -> INCLUDE, false -> EXCLUDE). INCLUDE keeps the previous unbounded behaviour (the caller
accepts the full cost). EXCLUDE always produces a lean template. AUTO bounds the client/loan fetch with a SQL
LIMIT and degrades to lean when either set exceeds fineract.bulk-import.template.max-lookup-rows
(default 10000), surfaced through FineractProperties.
- LoanRepaymentWorkbookPopulator gains a lean mode (static lean()/full() factories) that omits the
client/office sheets, the per-loan lookup table and the 3000-row VLOOKUP block, while keeping the data-entry
column layout intact (the import handler parses by fixed column index) and retaining the payment-type
dropdown sourced from the small Extras sheet (extracted to a shared addPaymentTypeValidation()).
- Add unit tests: LookupMode mapping; the max-lookup-rows cap boundary; and lean-mode output asserting the
Clients/Offices sheets and per-loan lookup rows are omitted while the data-column headers are preserved.
Description
Describe the changes made and why they were made. (Ignore if these details are present on the associated Apache Fineract JIRA ticket.)
Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!
Your assigned reviewer(s) will follow our guidelines for code reviews.