Rules-engine-suggestion-for-hard-coded-if-then-else

https://softwareengineering.stackexchange.com/questions/378813/rules-engine-suggestion-for-hard-
coded-if-then-else

Reposting question as Previous post on same Topic was not clear.

Currently our financial application receives multiple feeds in csv format from
client, there are usually 100k to 5M rows of data. When this information is
processed we need to enrich the data and store them, various look ups are
performed and based on data we need to categorize or assign more values.

An example mapping looks like below,

Firm Category Sub-Category Code Acct = InternalCode( Enriched )

A a1 sc1 c1 acc1 = ACCT1
B acc2 = ACCT2
B b1 = ACCT3
B b1 sc3 c3 acc4 = ACCT4
Here (from data feed), if firm =B and category =b1, then InternalCode is ACCT3,
if firm=B and Acct =acc2, then InternalCode is ACCT2.

Currently all of these are hard-coded in various programs with duplication and
hard to maintain/technical debt. We want remove hard coding to configurable
way.

There are many such lookups happening and each lookup has hundreds of such
business rules.

What is the best way to remove hard coding ? I am thinking about using rules
engine but not sure which criteria to apply to pick one for such use case. Also,
are there any other methods to move this kind of logic out of code ?

Thanks.

business-logic   business-rules   rules-and-constraints   rules-engine

shareimprove this questionfollow

edited Sep 22 '18 at 7:08

Doc Brown
155k2727 gold badges291291 silver badges447447 bronze badges
asked Sep 22 '18 at 3:44
 akumar
1911 bronze badge
 I took the freedom to make a slight change to your question to remove the red herring of a "3rd
party resource request", for which you already got a close vote by an anonymous voter (someone who is
obviously more interested in close voting for picky reasons than answering your question, which is IMHO
a good one). – Doc Brown Sep 22 '18 at 7:15 
 3
For future reference: When a question gets closed/put on-hold, you have the option to edit the question to
improve it. If the edit is good enough to make the question answerable, it will get reopened. That
mechanism is generally preferred by the community over asking a new question. – Bart van Ingen
Schenau Sep 22 '18 at 8:45
add a comment
4 Answers

Active OldestVotes

1
I think it is important to realize your goals first - it is typically not
just "better maintainability". A common motivation behind removing
hard-coded rules from code to configuration is often to shift
responsibility for maintaining the rules from the developer team
back to the business teams (and I bet this is your case here as well).

That means, configuring and maintaining the rules in some

configuration files should be less complex than implementing them
in code, and it should be doable by some people from the business
team. Otherwise a configurable solution won't bring you any benefit.

You
Unfortunately, there is seldom a "one size fits all" solution to this .
have probably lots of different use cases, each one requiring a
different level of complexity. I would recommend to try
out different approaches, start with a simple one and look how
well it works. Here are some ideas
1. For some cases, using a decision table like the one shown
in your example may be fully sufficient as a configuration.
Put the table into a spreadsheet or a database (pick what
your business team prefers), and write a program which
evaluates the table and processes the data according to
those rules.
2. For some cases, it may be sensible to implement your own
Domain Specific Language. This can be very simple, or it can
be as complex like other full-scale programming languages.
 (For example, I did this a few times using a tabular form,
with a column for preconditions, one for actions, and one for
parameters, which is a very simple approach, easy to
implement but still powerful enough for lots of cases.) Then
one can implement an interpreter for the DSL. Note the DSL
needs to stay simple enough your business people (or at
least some "power users" amongst them) can handle it.
To give you another example: I once had some users who
were no devs, but had some experience with SQL. They
required a validation tool for a database with >100
validation rules, and they wanted to manage those by
themselves. The data model was quite cryptic, but the
business team had a good understanding of its semantics,
better that every one else in our dev team. We made them
an Excel sheet where they could store the relevant parts for
a SELECT statement by themselves, and defined something
like a "DSL" for processing the result sets in various ways.
The program we gave them then read the SQLs, run them
against the DB and interpreted the results according to the
DSL. The solution is still in production, over years, with zero
maintenance requirements for the devs.
3. Using a rule engine. This is some form of a
"predefined, general-purpose DSL" for maintaining rules.
It may be a good solution for your case, but it also bears
the risk that you end up needing specialists for writing
and maintaining the rules, ending with a solution which
can be harder to maintain and debug than your current
system (just by different people). I recommend you go
through all the answers to this older SO question "When
should you NOT use a Rules Engine?", and check which
of the pro and con arguments apply to your specific
situation.
shareimprove this answerfollow
edited Sep 23 '18 at 14:39
answered Sep 22 '18 at 7:56
Doc Brown
155k2727 gold badges291291 silver badges447447 bronze badges
add a comment
0
The wrong way
It would be tempting to tell you to put each mapping in a
database table and query by providing all the relevant fields.
This would however miss the fact that rules may be based on
partial matching, and that when several matching rules are
found, the more specific should apply:
Example:

Input data: Output Explanations on rules

Firm Category Sub-Category Code

B b1 sc2 c3 ACCT3 (R2,R3 match but R3 is the most
specific)
B b1 sc3 c3 ACCT4 (R2,R3,R4 matches but R4 is the
most specific)
B b2 sc3 c3 ACCT2 (only R2 matches, for B)
Solution 1: sophisticated matching algorithm for
each mapping
Of course you could elaborate on your mapping table by
adding a rule precedence value and generate a more
complex query that would find all candidates and take
the one with the highest precedence.
This is error prone (comlex query generation, risk of
unexpeted rules being selected, need for defining
precedence manueally). In addition, it might lead to a
lot of expensive queries.
Solution 2: matching algorithm using successive
mappings
Another approach could be to determine a set of mapping
table, and let your rule engine iterate through successive
conditions, stoping whenever a matching rule is found:
Mapping 1:
Firm Category Sub-Category Code Acct =
InternalCode( Enriched )
A a1 sc1 c1 acc1 = ACCT1
B b1 sc3 c3 acc4 = ACCT4

Mapping 2:
Firm Category =
InternalCode( Enriched )
B b1 = ACCT3

Mapping 3
Firm =
InternalCode( Enriched )
B acc2 = ACCT2
This kind of approach is used in high performance pricing
engines (and even resulted in patent litigations that
concluded that it was a well known approach: disclaimer:
I'm not lawyer, and that's my personal understanding and
not legal advice).
This approach can be used either in a record by record way
(going through each set of mapping for each record). But it
can also be used in a more efficient way if you've uploaded
your CSV in a database, using some successive update
statements that don't update account values already filled
by previous steps.

Other solutions
Another approach could be to use some more
sophisticated rule engine, and translate all your if-then-else
into business rules. The advantage is that you don't have to
determine the mapping tables as you have done. You don't
have to think about mapping dependencies (e.g. if on ogf
 the mapping field is in fact determiend by a previous
mapping). It's also easy to add new rules.
The inconvenience is that the rule engine is invoked for
each CSV record. So it might be more heavy than the table
based approach (see solution 2). Also, it's difficult for the
rule writer to understand the interaction between different
kind of rules.
If you don't want to use an existing rule based engine and
develop your own, you could be interested in this SE
question.
shareimprove this answerfollow
answered Sep 23 '18 at 18:17

Christophe
39.9k55 gold badges5959 silver badges9999 bronze badges
add a comment
-1
You can make the mappings centralised by keeping them in
database and exposing get/update/delete/edit functionality
of a single/some/all mappings via RESTful APIs. Mapping
could be fetched before a processing of each batch.
shareimprove this answerfollow
answered Sep 22 '18 at 7:26

Kashan Danish
111 bronze badge

 The idea looks appealing but misses the problem: if the mappings are
all centralized, they may nonetheless be defined in a generic manner by
not specifying some of the fields. For example how would you determine
the ACCT for a CSV record (B, b1,sc2,c3) for which several rules would
match (taking into account the null fields in the rule, which mean any
value). – Christophe Sep 23 '18 at 18:24
add a comment
-1
 I think the simple and fast solution is to convert the conditions and
corresponding look ups in Redis database like this :

In this scenario you should consider all possible conditions for once
and set theme in Redis and for the lookup you can use Redis clients
directly or write a web service or etc.

The advantage of this method is time complexity for Set and

Get operations in Redis which is O(1) and you can retrieve
values corresponding to keys very fast.
shareimprove this answerfollow
answered Sep 22 '18 at 7:42

Mojtaba Tajik
10944 bronze badges

 The idea looks good but misses the problem: the records may have all the
fields all the time but the rules do not require all the fields. So the keys generated
for records will not necessary find the right rule. – Christophe Sep 23 '18 at
18:20

When should you not use a rules-engine

I have a pretty decent list of the advantages of using a Rules Engine, as well as some
reasons to use them, what I need is a list of the reasons why you should NOT use a Rules
Engine

The best I have so far is this:

Rules engines are not really intended to handle workflow or process
executions nor are workflow engines or process management tools
designed to do rules.
Any other big reasons why you should not use them?

I get very nervous when I see people using very large rule sets (e.g., on the order of
thousands of rules in a single rule set). This often happens when the rules engine is a
singleton sitting in the center of the enterprise in the hope that keeping rules DRY will make
them accessible to many apps that require them. I would defy anyone to tell me that a Rete
rules engine with that many rules is well-understood. I'm not aware of any tools that can
check to ensure that conflicts don't exist.
I think partitioning rules sets to keep them small is a better option. Aspects can be a way to
share a common rule set among many objects.

I prefer a simpler, more data driven approach wherever possible.

I will give 2 examples from personal experience where using a Rules Engine was a bad
idea, maybe that will help:-

1. On a past project, I noticed that the rules files (the project used Drools)
contained a lot of java code, including loops, functions etc. They were
essentially java files masquerading as rules file. When I asked the architect
on his reasoning for the design I was told that the "Rules were never
intended to be maintained by business users".
Lesson: They are called "Business Rules" for a reason, do not use rules when you cannot
design a system that can be easily maintained/understood by Business users.
2. Another case; The project used rules because requirements were poorly
defined/understood and changed often. The development team's solution
was to use rules extensively to avoid frequent code deploys.
Lesson: Requirements tend to change a lot during initial release changes and do not
warrant usage of rules. Use rules when your business changes often (not requirements).
Eg:- A software that does your taxes will change every year as taxation laws change and
usage of rules is an excellent idea. Release 1.0 of an web app will change often as users
identify new requirements but will stabilize over time. Do not use rules as an alternative to
code deploy.

I think a good way to rephrase your lesson #2 is "don't implement DSL when you know the
abstraction contained by the DSL is likely to change." Designing and implementing DSL is a
resource-heavy process that you aim to reap awards in the future. If you have to re-create DSL
every cycle, then the rules engine won't be a good fit yet until some stabilization happens. 
I'm a big fan of Business Rules Engines, since it can help you make your life much
easier as a programmer. One of the first experiences I've had while working on a
Data Warehouse project was to find Stored Procedures containing complicated
CASE structures stretching over entire pages. It was a nightmare to debug, since it
was very difficult to understand the logic applied in such long CASE structures, and
to determine if you have an overlapping between a rule at page 1 of the code and
another from page 5. Overall, we had more than 300 such rules embedded in the
code.

When we've received a new development requirement, for something called

Accounting Destination, which was involving treating more than 3000 rules, i knew
something had to change. Back then I've been working on a prototype which later on
become the parent of what now is a Custom Business Rule engine, capable of
handling all SQL standard operators. Initially we've been using Excel as an
authoring tool and , later on, we've created an ASP.net application which will allow
the Business Users to define their own business rules, without the need of writing
code. Now the system works fine, with very few bugs, and contains over 7000 rules
for calculating this Accounting Destination. I don't think such scenario would have
been possible by just hard-coding. And the users are pretty happy that they can
define their own rules without IT becoming their bottleneck.

Still, there are limits to such approach:

 You need to have capable business users which have an

excellent understanding of the company business.
 There is a significant workload on searching the entire system
(in our case a Data Warehouse), in order to determine all hard-
coded conditions which make sense to translate into rules to be
handled by a Business Rule Engine. We've also had to take good
care that these initial templates to be fully understandable by
Business Users.
 You need to have an application used for rules authoring, in
which algorithms for detection of overlapping business rules is
implemented. Otherwise you'll end up with a big mess, where no
one understands anymore the results they get. When you have a
bug in a generic component like a Custom Business Rule Engine,
it can be very difficult to debug and involve extensive tests to
make sure that things that worked before also work now.
More details on this topic can be found on a post I've
written: http://dwhbp.com/post/2011/10/30/Implementing-a-Business-Rule-
Engine.aspx
Overall, the biggest advantage of using a Business Rule Engines is that it allows the
users to take back control over the Business Rule definitions and authoring, without
the need of going to the IT department each time they need to modify something. It
also the reduces the workload over IT development teams, which can now focus on
building stuff with more added value.

Cheers,

Nicolae

The one poit I've noticed to be "the double edged sword" is:

placing the logic in hands of non technical staff

I've seen this work great, when you have one or two multidisciplinary geniuses
on the non technical side, but I've also seen the lack of technicity leading to
bloat, more bugs, and in general 4x the development/maintenance cost.

Thus you need to consider your user-base seriously.

shareimprove this answerfollow

answered Apr 22 '09 at 0:00
community wiki

Robert Gould

GREAT article on when not to use a rules Engine...(as well as when to

use one)....
http://www.jessrules.com/guidelines.shtml
Another option is if you have a linear set of rules that only apply once in
any order to get an outcome is to create a groovy interface and have
developers write and deploy these new rules. The advantage is that it is
wickedly fast because normally you would pass the hibernate session
OR jdbc session as well as any parameters so you have access to all
your apps data but in an efficient manner. With a fact list, there can be
alot of looping/matching that really can slow the system down.....It's
another way to avoid a rules engine and be able to be deployed
dynamically(yes, our groovy rules were deployed in a database and we
had no recursion....it either met the rule or it didn't). It is just another
option.....oh and one more benefit is not learning rules syntax for
incoming developers. They have to learn some groovy but that is very
close to java so the learning curve is much better.
It really depends on your context. Rules engine have their place and the
above is just another option if you have rules on a project that you may
want to deploy dynamically for very simplified situations that don't require
a rules engine.
BASICALLY do NOT use a rules engine if you have a simple ruleset and
can have a groovy interface instead.....just as dynamically deployable
and new developers joining your team can learn it faster than the drools
language.(but that's my opinion)

In my experience, rules engines work best when the following are true:

1. Well-defined doctrine for your problem domain

2. High quality (preferably automated) data to help drive most of your inputs
3. Access to subject matter experts
4. Software developers with experience creating expert systems
If any of these four traits are missing, you still might find a rules engine works
for you, but every time I've tried it with even 1 missing, I've run into trouble.

shareimprove this answerfollow

answered Sep 30 '09 at 15:37
community wiki

DaveParillo

his>_Software developers with experience creating expert systems_< If you read the drools docs carefully
you will realize that business rules should be implemented by software developers that have a strong
understanding of the Rete Algorithm. Access to the parameterization of rules can be provided to business
users via spreadsheets or CSV. Rules are nevertheless described by business users, but implemented
by, as you say, software developers with expertise in expert systems. The drools docs describe
this. – Matt Friedman Feb 24 '18 at 0:29
That's certainly a good start. The other thing with rules
engines is that some things are well-understood,
deterministic, and straight-forward. Payroll withholding is
(or use to be) like that. You could express it as rules that
would be resolved by a rules engine, but you could express
the same rules as a fairly simple table of values.
So, workflow engines are good when you're expressing a
longer-term process that will have persistent data. Rules
engines can do a similar thing, but you have to do a lot of
added complexity.
Rules engines are good when you have complicated
knowledge bases and need search. Rules engines can
resolve complicated issues, and can be adapted quickly to
changing situations, but impose a lot of complexity on the
base implementation.
Many decision algorithms are simple enough to express as
a simple table-driven program without the complexity
implied by a real rules engine.
shareimprove this answerfollow
answered Apr 21 '09 at 23:57
community wiki

Charlie Martin

I would strongly recommend business rules engines like Drools as

open source or Commercial Rules Engine such as LiveRules.
 When you have a lot of business policies which are volatile in
nature, it is very hard to maintain that part of the core
technology code.
 The rules engine provides a great flexibility of the framework
and easy to change and deploy.
 Rules engines are not to be used everywhere but need to used
when you have lot of policies where changes are inevitable on a
regular basis.
shareimprove this answerfollow
edited Apr 30 '13 at 13:55
community wiki

2 revs
muruga

https://jessrules.com/guidelines.shtml

Some Guidelines For Deciding Whether To Use A

Rules Engine
by George Rudolph
Assistant Professor of Computer Science, The Citadel

1. Is your algorithm primarily compute-intensive or is there significant decision-making

capability involved?
o If your basic algorithm is compute-intensive or a table-lookup, without much
conditional branching or decision-making involved, then don't use a rules engine.
o If however, the algorithm involves significant conditional branching or decision-
making, then consider using a rules engine.
2. Once you've determined that your algorithm involves significant decision-making
capability, you ought to be able to write some rules specifying the decisions that need to
be made. Are the decisions that need to be made relatively simple, or potentially
complex?
o If you find that you can't write the decision rules, for whatever reason, then stop
here until you can, or use some other tool instead that will help you discover the
rules you need. Put another way, if you can't state some rules, don't use a rules
engine.
o If you have 2, or fewer, conditions in your rules (or, for example, a block with 2
nested if-statements or less), don't use a rules engine-it's probably overkill.
o If you have 3 or more conditions in your rules (or, for example, a block with 3 or
more nested if-statements in pseudo-code), then consider using a rules engine.
3. Once you've determined that the decisions are complex enough, is your algorithm
basically static, or are the rules likely to change reasonably often over time?
o If the rules/logic are well-defined and static, then don't use a rules engine-you
probably don't need the overhead or flexibility.
o If the rules are likely to change over time due to the nature of the application,
then consider using a rules engine-the flexibility is worth the overhead.
4. Once you've determined that rules may need to be flexible, are you going to maintain the
code or finished product over time, or is this effort a one-shot effort?
o If the code is not going to be maintained over time, then don't use a rules engine-
you probably won't gain any significant advantage from it.
o If the code is going to be maintained over time, consider using a rules engine-the
ROI will be worth it (see question #6).
 5. Rules engines continue to get faster and faster, but some applications simply require
every bit of speed and performance optimization you can reasonably give. Does your
customer require a custom solution or do you need to hard-wire the algorithm for
absolute high-end performance, or can the solution accommodate a rules engine?
o If you need to optimize for speed and memory, or your customer requires a
custom solution, then don't use a rules engine.
o If the performance requirements will accommodate a rules engine solution, then
consider using one.
6. If you answered all the other questions appropriately, can the project/product line afford
the overall cost of using a rules engine over the project/product lifecycle?
o There are a number of costs typically associated with using rules engine tools:
 Licensing fees for development and deployment of the engine
 Training developers and (if necessary) end-users (time and money)
 ROI (return on investment)-financial analysts have shown that you don't
begin to break even monetarily on the typical investment in rules engine
technology until at least 1 year after deployment to your customer.
However, at that point, the flexibility and ease-of-maintenance begin to
show significant returns.
o If the project schedule or product lifecycle can't accommodate the cost of a rules
engine, in terms of time and money, then don't use one.
o If you can't afford to wait at least a year to break even and begin to see a
significant ROI, don't use a rules engine.
o If you can't afford to train your developers and end-users, and you can't afford to
hire a consultant, don't use a rules engine.
o If, however, your project/product lifecycle costs can accommodate the use of a
rules engine, it would be well worth the investment, so use one.

Additional Note

A rules engine tool can be very helpful during software development, regardless of other
considerations:

 For simulation and prototyping.

 In cases where you find you may not really know or understand the rules you are trying
to encode in your algorithm, a rules engine can provide a flexible way to encode and
modify the rules over time as they are discovered.
 A rules engine architecture also provides a convenient structure for separating "business
logic" from the rest of the system, aiding in the effort to clearly "separate concerns".

Last modified: Thu Jul 10 14:12:27 EDT 2008