Push your configuration to the limit - spice it up with statistics

So - you built a service with an advanced algorithm that uses some kind of classification in order to detect certain events - How would you know your configuration data is optimal? How would you measure the impact on accuracy each time the algorithm is changed?

The following post will show you how to use a mixture of concepts from Machine Learning, Statistics &  Operations Research in order to maximize configuration set of an algorithm.
This method is highly efficient in a client-side classification mechanism that requires accuracy but lacking learning time or processing power to do it.

The suggested method is divided into 5 phases:

• Test data preparation
• Running the test
• Result measurement
• Analysis
• Determine the winner

Test data preparation

Prepare a CSV file with data that have an indication of its correctness. e.g. if the algorithm should detect cars, we would have a data such as:
Toyota,1
mouse,0
Nissan,1
keyboard,0
Ford,1
button,0

Where 0 is an indication that the data is not a car and 1 is an indication that the data is a car.
The amount of data set will be discussed later on and might deserve a separate post, however, for simplification, assume we have 10 tests with 5 correct data and 5 incorrect data.

Running the test

Building a test to send data to your algorithm
The overall concept is clear:

1. Read the test data
2. Send it to your algorithm
3. Record the result (i.e. whether the algorithm identified the data correctly)

However, since we want to find out the optimal configuration we shall add additional step, before sending the data to the algorithm:
Set the algorithm with a certain configuration, meaning that the parameters your algorithm use would be able to be configured, for each run.
So the high level run steps should be:

1. Read the test data
2. Prepare algorithm configuration permutations
3. Send the data to your algorithm for each configuration permutation
4. Record the result for each run

Few things to note:

• Your algorithm should always return a result whether it identified the data or not, so you could record true negative and false negative.
• Externalize your configuration, enable to set it from outer class, as it required to do with each configuration set.
• Shuffle the test data between each run & execute multiple runs per specific configuration set. Shuffling the data will  enable to reduce correlation to the data order, executing multiple runs with the same configuration will help reduce any random standard deviation.

Result measurement

When your test gets the data back from the algorithm compare the algorithm predicted condition (positive/negative) versus the true condition, based on the indication in the test data.
Make sure to count the following:

• TruePositive (TP) = correctly identified, a hit!
  Classified correctly as positive
• True negative (TN) = correctly rejected, equivalent to correct rejection
  Classified correctly as negative
• False positive (FP) = incorrectly identified, equivalent to false alarm (type I error)
  Classified wrongly as positive
• False negative (FN) = incorrectly rejected, a miss (type II error)
  Classified wrongly as negative

The algorithm accuracy would be measured as follow:

Accuracy (ACC) = (TP + TN)/(TP + FP + FN + TN)

I like also to measure the following:

• True Positive Rate (TPR), also known as sensitivity:
  TPR = TP/(TP + FP), If all your positive data were identified correctly, the value would be 1, so as closer the result is to 1 - the better.
• True Negative Rate, also known as specificity:
  NPV = TN/(TN+FN), If all the negative data were identified correctly, the value would be 1, so as closer the result is to 1 - the better.
• False Positive Rate (FPR), also known as fall-outs:
  FPR = FP/(FP + TN), The closer the value is to 0, means fewer false alarm.
• False Negative Rate (FNR):
  FNR = FN/(TP+FN), The closer the value is to 0, means fewer missed out.

Cost function:

Highly important.

This is where you quantify the statistical analysis and the results into a single score that will help to determine how good the overall run/configuration is and will also make it simpler to compare between runs.

The cost function formula would look as follows:

Where CTP Is the cost of True Positive results, CTN is the cost of True Negative results and so on.

In a simplified manner, you need to sum each occurrence based on a certain weight.

You need to decide how much weight to give each data result, i.e. the value of TP, TN, FP & FN.

This may vary for each customer need, for example, some customers would like your algorithm to provide more results at the expense of quality, so you would give less penalty/weight on FP / FN result. Others would be very sensitive to faulty results, so your cost function would give higher value/weight to FP.

Note: I would recommend setting the cost value of TP and TN as such that if all the results are correct the expected sum value would be round number. For example, if we have 10 tests, 5 positive and 5 negatives, we would give each TP & TN the cost value of 10, so the overall score of a fully successful run would be 100.

Save in a file the configuration permutation and its score.

 Analysis

Put all the run result in excel. 

It is very easy to find the maximum cost function score and its configuration set, however, to find the most robust configuration, find the configuration value which had the best average across other permutations. To do this create a pivot table where all the configuration values are in the "Rows" section and the drag the cost function score field into the "Sum Value" section. Duplicate the score field in the "Sum Value" section and set one as an average function and the other as Max function.

Since all the configuration set is in the Rows section in the pivot table,  it is possible to aggregate it so only the first value is shown.

The final step is to write the best value per average score. Sometimes the a certain configuration value best average score would match the best maximum score, however, sometimes it won't.

In the below images you can see a demonstration of a certain configuration value and its average and maximum score. You could see that not always there is a correlation between average and maximum score.

In the below image a certain algorithm configuration has values 40,50,60 & 70. 

The value 70 appears in the configuration permutation of the maximum score and also has the best average. Having the best average means that setting the value 70 would give the best average score for any other configuration permutations. It implies about the robustness of this configuration.

In the below image we choose another algorithm configuration parameter with the values of 0,1 & 2. 

We did it by dragging the relevant configuration parameter field in the pivot table "Rows" section to the first position.
 You can see that the best average cost function score is for value 0 and the best maximum cost function score  is for the value 1.

Determine the winner with a final contest

At the end of the analysis phase, you would have a configuration set that provides the maximum score and a configuration set that provide the best average score.

This phase will try to minimize any implication the algorithm might have due to the test data, i.e. we want to rule out the possibility the algorithm is biased to a certain test data.

Create a test data with the following:

• 80% positive tests
• 70% positive tests
• 30% positive tests
• 20% positive tests

Execute each test data on each of the maximum and average configuration set.

The winner would be based on your customer need and the given result. If there isn't much variation between the runs score, it means that your algorithm is robust and you can choose the one that gave you maximum score, however, if there is big variation, it means that your algorithm is data dependent and it up to your customer needs whether to choose the best average or the best maximum.

An extra bonus getting using this method:

• Test how your algorithm utilized resources over time (Memory leaks, CPU utilization, disk space,..)
• Enable to accurately measure changes in the algorithm over time, across versions.

Any feedback is highly welcome.


Further good reads:
http://www.chioka.in/class-imbalance-problem/
https://en.wikipedia.org/wiki/Sensitivity_and_specificity

Please keep this link to the original post.

2 Android Studio productivity tips you MUST know

This post will handle frequent tasks one usually do in Android Studio:

• Modifying layout XML files
• Observing logcat output

If you do that actions, make sure you aware of the below tips.

Modifying layout XML files

When modifying XML layout files, one has to decide the ultimate decision: 

Which view to use - Design or Text view?

Design view provides you a good visual presentation of the layout, however, the text view provides faster property control easier to edit. So eventually, one would find themselves switching back and forths between those tabs.

Fortunately, you can have them both and gain productivity!

Choose the Text tab and you would see a collapsed "Preview" text on the right side of the ide (see image below)

It will open a small preview tab slider on the right. Every change that would be done in the Text tab will be reflected immediately in the preview.  Every element you will select in the preview will highlight the relevant element in the Text view.

This is how this dual view might look like:

Android Studio Text and Visual Preview on the same view

Observing logcat output

Logging into logcat is one of the basic fundamentals every android developer use.

It is very useful to trace application output and trace execution.

Often one would compare log output from 2 sessions in order to find differences.

Android Studio has a built-in tool for that, which will make the comparison super-fast.

The first step is to select the first output you want to compare and copy it to the memory (i.e. Cntl+C).

The image below shows a sample selection & copy from a logcat:

The second step is to filter the logcat to present the latest run (you don't have to use filter, just make sure it is visible in the logcat) , select it and right-click on the selection. 

In the context menu you will see an option to compare the selected text with the clipboard.

Right Click context menu on selection

Choose this option and you will see a diff tool.  

Android Studio Diff tool

That's it. Hope you will find it useful.

Please include this link to the original post.

2 ways where gradle overpower maven in Android build

As Android Studio which is the official IDE for Android based project use gradle as it's build platform, it made many of the maven users build their project using gradle.

As I constantly learn new features about gradle, I often reflect on the differences between  these two build systems.

The following observations are mainly focused on my experience in Android development.

3rd party libraries integration -
Assuming you use Maven in Eclipse based project, the process is tedious including adding the entire project source as a project, marking it as library dependency & modifying manually the permissions in the AndroidManifest.xml files.
In gradle, you simply define the dependency.  It has the aar format concept when it will do all the merge automatically into the final apk.

Version control -
In maven, you must either specify a version for a given dependency or use the word SNAPSHOT to use a certain version that is subject to updates.
i.e if you define dependency in spring core:

 org.springframework
 spring-core
 4.2.4.RELEASE

or in its development version:

 org.springframework
 spring-core
 4.2.4.SNAPSHOT

You will always get the 4.2.4 version.

In gradle you can specify a dependency version in a + sign, indicating that you want to receive any new version (this concept is called dynamic version).
so if an artifact is released with a new version (different number) you would get it automatically.
This mechanism has even finer granularity as you can specify 3.+ and get only updates which related to version 3.

The reason maven has real issues with determining which version is last is because it allows setting alphanumeric characters in the version (e.g. alpha, beta, ..).
In gradle, you are encouraged to use numbers only and hence it would automatically recognize which is the latest.

include this link to the original post

10 tips on how to build the perfect SDK

This post was born as a query of a friend of mine who thought there was not enough documentation on writing a good SDK that others can easily use.

In the last decade, SDK usage has become a major part of the development lifecycle. In fact it is so commonly used and integrated into products, that one would say developers need to acquire more knowledge with many frameworks rather than learning deep algorithms to implement by themselves.

This post is mainly addressed those who want to learn how to write the best SDK and supply documentation for it.

The goal/orientation of an SDK is that its documentation should be focused and clear.

If you feel there is more than one focus area - consider splitting it.

Below is a list which I hope would help you construct an SDK in a good way and writing its documentation:

0.     Learn what is out there

Try to see what your competitors or companies in similar domain as yours have done.

This may give you a point of reference. Take what you like and improve what you did not like.

1.     Simplicity

Code - simple code means your consumers find it easy to use. This might include as few as possible way to interact with your code, e.g. expose only one interface class; short method signatures, e.g. small number of input parameters; etc.

Except of initiation, which occurs once and might require some configuration, make the usage of your SDK methods as simple as possible.

As such -  try to keep method signature with as few parameters as possible.

You can achieve this by providing default configuration and default implementation classes, that can be overridden by advanced users.

Hide any class and method consumers don't need to use, i.e. make classes/methods public only if consumers must use them, otherwise use local or private scope. Some IDEs can help you do it automatically via code inspection and cleanup.

Documentation: Make your documentation as simple as possible. This means sometimes to write more explanatory text and sometimes write as less as possible. Inline code samples often help, as most humans learn by example.

2.     Provide an easy start - the way someone can use your code in less than 5 minutes. This is important as consumers want as little integration effort as possible, moreover, sometimes consumers want to evaluate your product, and without an ability to easily experiment with it, they would probably choose to skip it.

3.     Keep it short

This section is mainly relevant for documentation, but is also related to the ways the consumers can interact with the SDK code. In regards to documentation; this can be achieved by providing code samples and self-explanatory method names and using defaults.

4.     Integration - keep in mind the diversity of your consumers development environments.

For example, If you are writing an android library, the integration with it vary if your consumers user Android Studio with gradle (that requires aar artifacts and publishing to a remote repository) or they use Eclipse where you need to provide jar files, instruction about how to change the AndroidManifest.xml and a standalone eclipse project for the SDK.

This would impact your build mechanism & it's artifacts. However, don't try to win it all from day one. Do what fit most to your first client or to most of your predicted consumers.

5.     Sample project

Create the most basic project in GitHub that simulates a client that uses your SDK.

This would demonstrate your consumers how your product can fit their needs as well the easiest way to integrate with your product.  If you want to show an advanced usage, do it in another project. Often your consumers would use it as their main source of documentation, so provide inline comments and write code in a self-explanatory way as possible.

6.     Overview - in the beginning of the documentation or in the README.md file in the GitHub project, provide an overview about your solution in plain English. In this section, I usually like to provide a sample use case that will explain a typical SDK usage. If possible, provide a simple diagram or chart so people that don't like to read manuals, will see the benefit of your SDK quickly.

7.     Initiation - use conventions that are acceptable in the SDK domain.

It may be constructors with overload, build pattern or similar. The initiation should smartly use defaults in order to keep easy start.

8.     Defaults 

Defaults are important to keep code simplicity and reduce configuration (see the simplicity section). The defaults you provide (either configuration or implementation) shall represent the way you think most of the consumers will use your SDK.

You can provide several method overloading, where the simplest signature would call the more advance method signature with the default.

9. Publishing

• Offline use non-editable format - PDF. The advantage is that you can easily create one, store it locally in Dropbox and for each update, the version is updated automatically.
• Online - your corporate website. This is the preferred way, however, it might create a hassle of some IT overhead when you update it.

Hope those guildelines helps. Feedback is more than welcome.

Please keep this link to the original post.

photo credit

Android Gradle - build tips - Lint HTML layout

With the release of gradle 2.10 there is a nice feature of Lint check html reports.

Simply speaking, whenever you run a build, a nice formatted html report will be generated. It is a VERY nice touch to the lint xml reports and makes the report much more readable and hence more actionable. It will save you time and make your code more robust and stable.

This feature was contributed by Sebastian Schuberth .

To check it out do the following:

1. Download gradle 2.10 and extract it to the default Android Studio gradle directory.
It may be:
C:/Program Files/Android/Android Studio/gradle/gradle-2.10

2. locate the file gradle-wrapper.properties via Android Studio and change distributionUrl to: https\://services.gradle.org/distributions/gradle-2.10-all.zip

3. In Android Studio open File->Settings and change gradle location to the one you extracted the gradle 2.10 file.

Android studio - change gradle version

4. Open a terminal for the project folder (either by Android Studio terminal view or explicitly via command line tool) and type: gradlew {module.name}:build

Note: The parameter {module.name} is optional and intended for multi module project where you want to build certqin modules.

That's it.

After the build finishes, scroll via its output and you will see something similar to:

Wrote HTML report to file:///C:/dev/workspace/common/commonlibrary/build/outputs/lint-results.html

Wrote XML report to file:///C:/dev/workspace/common/commonlibrary/build/outputs/lint-results.xml

Open the html link and you would be pleasantly surprised.

Check it out:

Lint Html report - sample

IMHO, the generated report is visually better than the report being generated via Android Studio-> Code Inspection. It is very convenient to read the warnings/errors and correct.

Another advantage is that report will be generated on every build and not on demand.

Note:

The lint html report would be generated from command line and not via Android Studio default build.

Please include a link to the original post.

Applying an Android application into any Application Store is not a walk in the park.
One need to prepare, beside the application binaries of course, several supporting documents (marketing, images, reviewer instructions etc.).

The publication process length might vary from store to store.
In Google Play the publication starts immediatly after the binary is being uploaded.
In Samsung Store it take several long days till the binaries are inspected and pass the entire certification process.
Amazon store submission process is very similar to the Samsung store process but the waiting period might be a bit shorter.
However, in all the stores mentioned above there is a lack of certainty about WHEN the process will end.

Therefore I was very surprised when I saw the submission process at the Korean store - T-Store.
In the submission dashboard there is a counter that specifies the deadline of this review (see image below)

T-Store Review Deadline counter

I like this efficiency and the feedback!

In case you wonder - this is NOT effects the inspection quality. In fact their inspection is one of the best quality I seen.

Geek, Nerd, Dork ?

Some nice venn diagram about what makes a geek, nerd or dork:

http://www.buzzfeed.com/scott/nerd-venn-diagram

Recently a more scientifically approach was taken in a research about the difference between a nerd and a geek.. here are the results:
http://www.makeuseof.com/tag/are-you-a-geek-or-a-nerd-find-out-for-sure/

PixMix - very cool app recommendation

I would like to recommend a cool mobile app I recently installed called PixMix.

You might have came across it in a Jelastic technological post, in their spotlight corner.
http://blog.jelastic.com/2012/12/07/the-jelastic-spotlight-pixmix/


PixMix app allows one to instantly create photo albums and share it with friends.
So every photo me and my friends  take in a certain album, instantly uploaded into an online space which show all the photos.

What distinguish it apart from other apps is the fact that it has cool feature of animated photo gallery.
This feature is a killer feature for me.
Every new photo, of a certain album, is immediately shown in the live animated gallery.

There is also the right amount of balance between social sharing and private albums. It allows you to share what right (and not instantly takes the whole album public) and for who is right.

I tried it in my daughter 2 years birthday party.
I connected my laptop into my living room TV, opened the album via the PixMix site and displayed the animated live gallery.
I started to take few photos vie PixMix, just to see it works, and to my surprise it was instantly appeared in the animated gallery.
That really took the attention of my family, and before I realized, everyone started taking photos and compete each other to see which photo will appear in the animated gallery...

The nice thing is that at the end of the birthday, I was left with many photos, capturing many moments, I wasn't even part of...

Check it out  :-)

DB connection pool - production tips

Every developer knows that DB IO is the bottle neck of most web application.

It is also a common best practice to use an open connection to the DB and manage it in a pool, since opening a DB connection is costly operation.

However, in many DB connection pools tutorials, often the mentioned parameters are the size of the initial pool (min, max size) and how many connection to increment.

I would like to state here additional configuration which is important mainly in production use to efficient pool management -

The following are configuration parameters which are specific to C3P0 pool, however, one can

1. maxIdleTime - Num of seconds a Connection can remain pooled but unused before being discarded. Zero means idle connections never expire. I would recommend not keeping idle connection much time.
2. maxIdleTimeExcessConnections - number of seconds that Connections in excess of minPoolSize should be permitted to remain idle in the pool before being culled. Intended for applications that wish to aggressively minimize the number of open Connections.
3. maxConnectionAge -  A Connection older than maxConnectionAge will be destroyed and purged from the pool. This differs from maxIdleTime in that it refers to absolute age.
4. unreturnedConnectionTimeout - Will destroy open/active connection if  it wasn't returned to the pool within the specified amount of time. This could potentially prevent memory leaks if exception which prevent connection to close occurred. use it with parameter- debugUnreturnedConnectionStackTraces=true so you could debug the and find the reason for such behavior.

 

 

references - C3P0 configuration

How to change logging level in runtime

Changing the log logging level in runtime is important  mainly in production environment where you might want to have debug logging for limited amount of time.

Well, changing the root logger is very simple - assuming you have an input parameter with the wanted logging level simply get the root logger and set by the input logging level, such as:

Logger root = Logger.getRootLogger();

//setting the logging level according to input
if ("FATAL".equalsIgnoreCase(logLevel)) {
    root.setLevel(Level.FATAL);
}else if ("ERROR".equalsIgnoreCase(logLevel)) {
    root.setLevel(Level.ERROR);
}

However - the common case is that we maintain log instance per class, for example:

class SomeClass{

//class level logger
static Logger logger - Logger.getLogger(SomeClass.class);
}

and setting the root logger is not enough, since the class level logger will not be affected.

The trick is to remember get all the loggers in the system and change their logging level too.
For example:

Logger root = Logger.getRootLogger();
Enumeration allLoggers = root.getLoggerRepository().getCurrentCategories();

//set logging level of root and all logging instances in the system
if ("FATAL".equalsIgnoreCase(logLevel)) {
    root.setLevel(Level.FATAL);
    while (allLoggers.hasMoreElements()){
        Category tmpLogger = (Category) allLoggers.nextElement();
        tmpLogger .setLevel(Level.FATAL);
    }
}else if ("ERROR".equalsIgnoreCase(logLevel)) {
    root.setLevel(Level.ERROR);
    while (allLoggers.hasMoreElements()){
        Category tmpLogger = (Category) allLoggers.nextElement();
        tmpLogger .setLevel(Level.ERROR);
    }
}

So just wrap it up in a service class and call it from your controller, with a dynamic logLevel String parameter which represent the logging level you wish to set your system to.

If any of you need the complete solution, please let me know.

Reference to the basic approach is in this link.

Build documentation to last - choose the agile way

Lately I wondered what the best way to document a project is?

My documentation experience vary among different tools and methodologies.
I would like to share some observation I have and a conclusion about the best way to document a project.


The documentation could be classified to the following categories:

Documentation place:

• In line code/ version control system (via code commit description)
• In separate server linked to the code
• In separate server, decoupled from the code (no direct linkage)

Documentation done by:

• Developer
• Product team/ Design team / architects
• Technical writers

Documentation too:

• IDE
• Design documents
• Modeling tool, Process Flow
• Wiki
• version control (e.g. git, svn,..) commits
• Interface documentation

Not surprisingly there is a direct correlation between the tool used , the person who document the code, the amount of documentation, the "distance" of documentation from the code and the accuracy of that documentation.

Given the categories above it could be organized in the following manner:

• Developers
• In line code/ version control system (via code commit description) 
• IDE/ version control

• Product team/ Design team / architects
• In separate server linked to the code 
•  Design documents, Interface documentation

• Technical writers
•  In separate server, decoupled from the code (no direct linkage)
• Modeling tool, Process Flo, wiki

Developers tend to write short inline documentation using IDE, well interface semantics and complementary  well written code commits.
As long as the person who document the functionality has more distance from the code, the documentation would usually be in places more decoupled from where the code exist and more comprehensive.

From my experience, even good design tend to change a bit and even if the documentation is good but is decoupled from the code, most chances are that it won't catch up with code change.

In real life, when requirements keep coming from the business into development, it sometimes brings with it not only additional code to directly support functionality, but often we see the need for some structural or infra change and refactoring.

The inline code documentation is agile and change with minimum effort along the change in functionality. If the developer submit the code grouped by functionality and provide good explanation about changes that were done it would the most updated and accurate  documentation .

I know that some of you might wonder about heavy duty design or complex functionality documentation.
I would recommend tackle these issues as much as possible inline the code, for example, assuming you read some pattern or some bug solution in the web put a link to that solution near the method/class which implement the solution. Try to model your code by known patterns so it would avoid documentation. Try to use conventions so it would reduce amount of configuration and make your code flow more predictable and discoverable.

This approach is even more important when managing a project in agile methodology.
Usually such methodology would rather direct communication with product/business to understand requirements rather than documented PRDs. This makes it even more important to have the code self explanatory easy for orientation. Moreover, frequent changes in design and business change would cause decoupled documentation soon be obsolete (or will drag hard maintenance)

Although it sounds easier said than done and it is not a silver bullet for every project, writing documentation as close as possible to the code itself should be taken as a guideline / philosophy when developing a project.


Acknowledgment - above image was taken from lwiki's Gallery : https://picasaweb.google.com/lh/photo/ScQcKRBjhY7UvnzJ7vNArdMTjNZETYmyPJy0liipFm0

Spring Profile pattern , part 4

Phase 3: using the pattern

As you can recall, in previous steps we defined an interface for configuration data.
Now we will use the interface in a class which needs different data per environment.

Please note that this example is the key differentiator from the example given in  the Spring blog, since now we don't need to create a class for each profile, since in this case we use the same method across profiles and only the data changes.


Step 3.1 - example for using the pattern

@Configuration
@EnableTransactionManagement
//DB connection configuration class 
//(don't tell me you're still using xml... ;-)
public class PersistenceConfig {

 @Autowired
 private SystemStrings systemStrings; //Spring will wire by active profile

 @Bean
 public LocalContainerEntityManagerFactoryBean entityManagerFactoryNg(){
  LocalContainerEntityManagerFactoryBean factoryBean
  = new LocalContainerEntityManagerFactoryBean();
  factoryBean.setDataSource( dataSource() );
  factoryBean.setPersistenceUnitName("my_pu");       
  JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter(){
   {
    // JPA properties
    this.setDatabase( Database.MYSQL);
this.setDatabasePlatform("org.hibernate.dialect.MySQLDialect");
    this.setShowSql(systemStrings.getShowSqlMngHibernate());//is set per environemnt..           
   
   }
  };       
  factoryBean.setJpaVendorAdapter( vendorAdapter );
  factoryBean.setJpaProperties( additionalProperties() );

  return factoryBean;
 }
//...
@Bean
 public ComboPooledDataSource dataSource(){
  ComboPooledDataSource poolDataSource = new ComboPooledDataSource();
  try {
   poolDataSource.setDriverClass( systemStrings.getDriverClassNameMngHibernate() );
  } catch (PropertyVetoException e) {
   e.printStackTrace();
  }       
                 //is set per environemnt..
  poolDataSource.setJdbcUrl(systemStrings.getJdbcUrl());
  poolDataSource.setUser( systemStrings.getDBUsername() );
  poolDataSource.setPassword( systemStrings.getDBPassword() );
  //.. more properties...       
  return poolDataSource;
 }
}

I would appreciate comments and improvements.
Enjoy!

part 1, part 2 , part 3, part 4

Spring Profile pattern , part 3

Phase 2: implementing the profile pattern
This phase utilizes the infra we built before and implements the profile pattern.



Step 2.1 - create a properties interface
Create an interface for the configuration data you have.
In our case, the interface will provide access to the four configuration data  items.
so it would look something like:

public interface SystemStrings {

String getJdbcUrl();
String getDBUsername();
String getDBPassword();
Boolean getHibernateShowSQL();
//..... 

Step 2.2 - create a class for each profile
Example for a development profile:

@Dev //Notice the dev annotation
@Component("systemStrings")
public class SystemStringsDevImpl extends AbstractSystemStrings implements SystemStrings{
      
 public SystemStringsDevImpl() throws IOException {
                //indication on the relevant properties file
  super("/properties/my_company_dev.properties");
 } 
}

Example for a production profile:

@Prouction //Notice the production annotation
@Component("systemStrings")
public class SystemStringsProductionImpl extends AbstractSystemStrings implements SystemStrings{
      
 public SystemStringsProductionImpl() throws IOException {
                //indication on the relevant properties file
  super("/properties/my_company_production.properties");
 } 
}

The two classes above are where the binding between the properties file and the related environment occur.

You've probably noticed that the classes extend an abstract class. This technique is useful so we won't need to define each getter for each Profile, this would not be manageable in the long run, and really, there is no point of doing it.

The sweet and honey lies in the next step, where the abstract class is defined.

Step 2.3 - create an abstract file which holds the entire data

public abstract class AbstractSystemStrings implements SystemStrings{

 //Variables as in configuration properties file
private String jdbcUrl;
private String dBUsername;
private String dBPassword;
private boolean hibernateShowSQL;

public AbstractSystemStrings(String activePropertiesFile) throws IOException {
  //option to override project configuration from externalFile
  loadConfigurationFromExternalFile();//optional..
                //load relevant properties
  loadProjectConfigurationPerEnvironment(activePropertiesFile);  
 }

private void loadProjectConfigurationPerEnvironment(String activePropertiesFile) throws IOException {
  Resource[] resources = new ClassPathResource[ ]  {  new ClassPathResource( activePropertiesFile ) };
  Properties props = null;
  props = PropertiesLoaderUtils.loadProperties(resources[0]);
                jdbcUrl = props.getProperty("jdbc.url");
                dBUsername = props.getProperty("db.username"); 
                dBPassword = props.getProperty("db.password");
                hibernateShowSQL = new Boolean(props.getProperty("hibernate.show_sql"));  
}

//here should come the interface getters....

part 1, part 2 , part 3, part 4, next >>

Spring Profile pattern ,part 2

Spring Profile pattern -  phase 1: infra preparation


This phase will establish the initial infra for using Spring Profile and the configuration files.

Step 1.1  - create a properties file which contains all configuration data
Assuming you have a maven style project, create a file in src/main/resources/properties for each environment, e.g:
my_company_dev.properties
my_company_test.properties
my_company_production.properties

example for my_company_dev.properties content:

jdbc.url=jdbc:mysql://localhost:3306/my_project_db
db.username=dev1
db.password=dev1
hibernate.show_sql=true

example for my_company_production.properties content:


jdbc.url=jdbc:mysql://10.26.26.26:3306/my_project_db
db.username=prod1
db.password=fdasjkladsof8aualwnlulw344uwj9l34
hibernate.show_sql=false


Step 1.2  - create an annotation for each profile
In src.main.java.com.mycompany.annotation create annotation for each Profile, e.g :

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Profile("DEV")
public @interface Dev {
}

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Profile("PRODUCTION")
public @interface Production {
}

Create an enum for each profile:
public interface MyEnums {

public enum Profile{
DEV,
TEST,
PRODUCTION
}

Step 1.3  - make sure the profile is loaded during context loading

• Define a system variable to indicate on which environment the code is running.
  In Tomcat, go to ${tomcat.di}/conf/catalina.properties and insert a line:
  profile=DEV  (according to your environment)
• Define a class to set the active profile
  

  public class ConfigurableApplicationContextInitializer implements
    ApplicationContextInitializer {
  
   @Override
   public void initialize(ConfigurableApplicationContext applicationContext) {
        
    String profile = System.getProperty("profile");
      
    if (profile==null || profile.equalsIgnoreCase(Profile.DEV.name())){
     applicationContext.getEnvironment().setActiveProfiles(Profile.DEV.name());   
    }else if(profile.equalsIgnoreCase(Profile.PRODUCTION.name())){
     applicationContext.getEnvironment().setActiveProfiles(Profile.PRODUCTION.name()); 
    }else if(profile.equalsIgnoreCase(Profile.TEST.name())){
     applicationContext.getEnvironment().setActiveProfiles(Profile.TEST.name()); 
          }
   }
  }
  

• Make sure the class is loaded during context loading
  in the project web.xml, insert the following:

  
       contextInitializerClasses
       com.matomy.conf.ConfigurableApplicationContextInitializer
   
  

part 1, part 2 , part 3, part 4 , next >>

Spring Profile pattern , part 1

Recently we were introduced with the concept of Spring Profiles.
This concept is an easy configuration differentiator  for different deployment environments.
The straight forward use case (which was presented) was to annotate the relevant classes so Spring would load the appropriate class according to the active profile.

However, this approach might not always serve the common use case... often, the configuration keys would be the same and only the values will change per environment.

In this post, I would like to present a pattern to support loading configuration data per environment, without the need to create/maintain multiple classes for each profile (i.e. for each environment).

Throughout the post I would take the DB connection configuration as a sample, assuming we have different DB definitions (e.g. username or connection URL) for each deployment environment.

The main idea is to use one class for loading the configuration (i.e.. one class for DB connection definition) and inject into it the appropriate instance which holds the correct profile configuration data.

For convenience and clarity, the process was divided into 3 phases:

Phase 1: infra preparation
Step 1.1  - create a properties file which contains all configuration data
Step 1.2  - create an annotation for each profile
step 1.3  - make sure the profile is loaded during context loading

Phase 2: implementing the profile pattern
Step 2.1 - create a properties interface
Step 2.2 - create a class for each profile
Step 2.3 - create an abstract file which holds the entire data

Phase 3: using the pattern
Step 3.1 - example for using the pattern

next part >>

Email filtering using Aspect and Spring Profile

During web application development, often the need for sending emails arise.

However, sometimes the database is populated by data from production and there is a risk of sending emails to real customers during email test execution.

This post will explain  how to avoid it without explicitly write code in the send email function.

We would use 2 techniques:

1. Spring Profiles - a mechanism to indicate what the running environment is (i.e. development,  production,..)
2. AOP - in simplified words its a mechanism to write additional logic on methods in decoupled way.

I would assume you already have Profiles set on your projects and focus on the Aspect side.

In that example the class which sends emails is EmailSender with the method send, as specified below:

public class EmailSender {
//empty default constructor is a must due to AOP limitation
public EmailSender() {}

//Sending email function
//EmailEntity - object which contains all data required for email sending (from, to, subject,..)
public void send(EmailEntity emailEntity) {
//logic to send email
}
}

Now, we will add the logic which prevent sending email to customers where code is not running on production.
For this we will use Aspects so we won't have to write it in the send method and by that we would maintain the separation of concern principle.

Create a class that will contain the filtering method:

@Aspect
@Component
public class EmailFilterAspect {

public EmailFilterAspect() {}
}

Then create a PointCut for catching the send method execution:

@Pointcut("execution(public void com.mycompany.util.EmailSender.send(..))")
 public void sendEmail(){}

Since we need to control whether the method should be executed or not, we need to use the Arround annotation.

@Around("sendEmail()")
public void emailFilterAdvice(ProceedingJoinPoint proceedingJoinPoint){
 try {
  proceedingJoinPoint.proceed(); //The send email method execution
 } catch (Throwable e) {                           
  e.printStackTrace();
 }
}

As a last point, we need to access the send method input parameter (i.e. get the EmailEntity) and verify we don't send emails to customers on development.

@Around("sendEmail()")
 public void emailFilterAdvice(ProceedingJoinPoint proceedingJoinPoint){

 //Get current profile
ProfileEnum profile = ApplicationContextProvider.getActiveProfile();

Object[] args = proceedingJoinPoint.getArgs();        //get input parameters
        if (profile != ProfileEnum.PRODUCTION){
            //verify only internal mails are allowed
            for (Object object : args) {
                if (object instanceof EmailEntity){
                    String to = ((EmailEntity)object).getTo();
                    if (to!=null && to.endsWith("@mycompany.com")){//If not internal mail - Dont' continue the method                        
                        try {
                            proceedingJoinPoint.proceed();
                        } catch (Throwable e) {
                            e.printStackTrace();
                        }
                    }
                }
            }
        }else{
            //In production don't restrict emails
            try {
                proceedingJoinPoint.proceed();
            } catch (Throwable e) {
                e.printStackTrace();
            }
        }
}

That's it.
Regarding configuration, you need to include the aspect jars in your project.
In Maven it's look like this:

        org.aspectj
 aspectjrt
 ${org.aspectj.version}


 org.aspectj
 aspectjweaver
 ${org.aspectj.version}
 runtime

and in your spring application configuration xml file, you need to have this:

Good luck!

Seamlessly Static Meta Model generation

I would like to introduce a simple, quick and straight forward way to create Static Meta Model classes.

First, I would like to correct a perception I had in my previous post regarding the place such files are created.
Since the Static Meta Model files are generated classes and should automatically changed for each @Entity modification they should placed in the target folder and not comitted to the repository.

Moreover, creation of static meta model files via Eclipse works correctly if you use the appropriate generation project version and te right plugin.

First step is to get the class generation jar. Put in pom.xml :
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-jpamodelgen</artifactId>
            <version>1.1.1.Final</version>
        </dependency>

Then in eclipse add the annotation generation plugin via Help ->Eclipse Market place -> search for "Apt M2E" and install it.

After the installation right click on your project -> properties -> Java compiler-> Annotation processing -> mark "enable project specific settings" (and actually all the checkboxes on that screen), in the Generated Source Directory put "target\generated-sources" (This will generate the classes in your target folder).

Inside the Annotation processing item there is a Factory Path item, enable this part as well and set the jar we import via maven to generate the classes. You can do it by clicking Add Variable -> M2_REPO -> Extend -> and choose the following: path : /org/hibernate/hibernate-jpamodelgen/1.2.0.Final/hibernate-jpamodelgen-1.2.0.Final.jar

Make sure only that path is checked.

As a final step, please make sure the target\generated-sources folder is on your classpath (right click-> build path -> ad as source folder).

That's it. Every change should trigger automatic static meta model generation.

Spring Data JPA - Limit query size

This one is very informative and short:

If you are using Spring Data JPA, you are probably familiar with the technique of query creation out of methods names.

Following that mechanism, there is no explicit way to use the LIMIT keyword in the query name,
However, there is a simple way to use the SQL LIMIT with these queries.
The implicit way to limit the query result size is to utilize the pagination mechanism.
 
Simply provide and extra paging object with the objects number you need to limit.


Below is a simplified example of how the repository interface should be used:

public interface EntityRepository extends CrudRepository, JpaSpecificationExecutor {

 List findByEntityMemberNameLike(String query, Pageable pageable);//Pageable will limit the query result

}

Below is a simplified example of how the usage of such query could be:

@Autowired
EntityRepository entityRepository;

...

int queryLimit = 10;
List queryResults = entityRepository.findByEntityMemberNameLike(queryString, new PageRequest(0, queryLimit));

LESS is more?

Recently I've been exposed to LESS which is a UI framework.
In a nutshell, this framework extends the standard CSS in a way that insert programming flavor into it, such as inheritance, variables, functions and build.
(By the way - this is not the only framework with such concept)


Using this framework has some pros and cons which I would like to highlight.
Pros:

• Enable css reuse via inheritance, variables and more.
• Build *one* css for the final html to use.

Cons:

• Introduce another development language, which requires strong CSS knowledge (if you use it well, otherwise, no point to use it)
• End-to-end developers (UI to server) might need another domain to maser
• Coupling your project with another framework. Disconnecting from less is not simple
• Produced css are not ideal for development phase and debug
• CSS changes can be done only via this framework/language

At first review this LESS technique sounds like a css simplification magic, however at the end we stopped using it since its cost/benefit was low.

Please let me specify my claims.
The main disadvantage of this framework, IMHO, is that it makes your project too couples with it.
While the generated CSS is ideal for production, in development environment, it's very hard to work with.
In a typical project where several UI frameworks (e.g. jquery) are involved - imagine how long the one output css file will be.
Moreover, not only that it would be very hard to work on the CSS output directly, it would be very hard in the future to separate it to file per framework since LESS does not support it.

Another disadvantage is that in order to use this framework properly, the developer need to learn another language and expertise in a domain which is not necessary the developer strong side.

This tool is mainly used by UI side people, so knowing that tool is like Java developer to know Spring framework, however, this does not work the other way around - for Java developers such tool added complexity. It is one thing to know JS or modify slightly css, and another thing to have experience in CSS inheritance and advanced techniques.

I think it applies for LESS developers as well. How many UI developers do you know who master Spring or Hibernate...

Anyway I hope my experience would help other to estimate whether to use such framework or not.

Eclipse metamodel generation (JPA 2.0) issues

Update:
Please see updated post in that manner which simplifies the process a bit better.

Static Metamodel classes are uses for type safe queries in JPA2.0 standard.

There are few ways to create these classes:
1. Define the IDE (e.g. eclipse) to generate the files automatically (like the IDE automatic build which generates the class files on each Java file change).
2. Define maven to generate the files. The generation will be done when running the maven script.

 I would recommend to avoid the generation via the Eclipse IDE, since I've noticed that the generated static meta model are inaccurate (e.g. some members were missing).

 I'm not sure what the reason is since the missing member has no special attribute that differentiate it form the other class members that are generated, however, the fact is that it is constantly not generated.

Static meta model files generation via maven overcome this issue.


Good luck !