Creating Automation Framework from scratch or use automation testing tool?

Often I receive many requests on Quora regarding when should we build an automation framework from scratch vs. When we should be using an automation tool based likee Katalon or Postman, UFT and the list is unlimited!


We all know that over the last decade, Testing has evolved a lot. First, we had the manual Testing then selenium IDE (Record and Playback) and then WebDriver and API automation coming to picture!

 Also, with the DevOps Culture coming into the picture, a lot of IT companies focus on are giving preferences to test automation like never before.

If you are into the Testing domain, one of the best career decisions would be to be a part of the Automation Testing domain.


With so many Automation tools in the market that promise in helping to get test automation started for the organization, anyone can easily get lost in finding the right tool testing tool that suits your organization’s needs.


I always believe that bringing automation in any project is like fighting a battle. It’s crucial to have the right experienced team, a perfect strategy, and lastly, the right tool that helps you achieve test automation for your project.


I have seen startups focusing on UI Automation from the start of the project, even when UI is still under development. 

Such development often turns out to be a nightmare with those long-running slow UI tests, which doesn’t help in giving a clear picture of whether the feature is ready to go in production or not.

As a result, to get more clarity, teams often end up doing manual Testing for the application again. Oops

But Jatin How can we get better ROI on automation testing from the start of the project?


So, Guys, the best approach to follow is to understand the skill sets of your team and see whether the developers and testers can work on the same programming language.


If this is the situation in your organization in which you have a well-established team and who are comfortable in writing clean code and you want to have full control of the framework, then you should go in creating the automation framework from scratch.


BUT if your team is having different skillsets or your testers are not comfortable in writing test scripts, then the better option would be to rely on testing tools.

I also feel that if you are building an application from scratch, it’s better to rely on tools than to have a team building an automation framework.


But why?

The reason is that during the initial days of development, both the backend code and UI code will be going through many refactoring. 

So there is no point in building a framework because it will be acting as overhead, and ROI of automation will below.


In the early stage of Project Development, you can really can confidently really on tools that will help you achieve automation testing.

There are many automation testing tools in the market that help in achieving test automation. 

I will be comparing three tools that you should consider adding to your organization.


I am listing the pros and cons of these tools [Postman, Katalon, and Test Project] that will help you in selecting the tool for your project. 

The comparison is based o the Cost-Time of Setup- Customization for your project and Community Support!

Let’s get started!

Postman [API Testing Tool]:

  • API Testing Tool
    • Postman has been around helping testers and developers test the API’s are working fine or not. 
    • Just being an API Testing tool is an advantage and disadvantage of Postman because it is limited to the only APIs, and you will need to look for some other means for UI Automation Testing.
  • Language Support:
    • The Test Scripts are written in JavaScripts.
  • Code Reusability:
    • You cannot have code reusability in Postman. Hence sometimes maintaining those APIS requests is a headache.
  • Reporting:
    •  Ahh, we all know how much we love seeing those test reports, but if you want to have a detailed reporting in Postman it’s not possible 🙁
  • Designing a Framework using Postman? : 
    • Not likely, and if you’re going to pass test data from external resources, then you can only use 1 CSV file.
    •  With 1 CSV file maintaining Test Data is often tricky if you are testing a lot of API Request with different parameters and request body.
  • Jenkins Integration: 
    • You cannot have direct integration of Postman with Jenkins. You need to rely on a node library called Newman to do it, which can execute only your collections. 
    • Also, if you make any changes in the collection, you need to export the collection again and replace it with the previous one!.
    •  This activity can get messy in the long run.
  • Learning Curve:
    • Learning Curve Of Postman is not that difficult. If you are majorly testing API requests, I would suggest you give the Postman tool a try!
  • Cost:
    • Postman has both paid and free versions.
    • If you are need services like monitoring and collaborative working, then you can consider having the paid version.

Now, Let’s talk about another testing tool that focuses on both UI and API Testing that is Katalon.



  • UI Testing and API Testing 
    • Katalon is an Automation Testing tool that does API Testing and UI Testing. 
    • If you have postman scripts, you can easily import your postman scripts to Katalon and use it in the Katalon application.
  • I have been getting my hands dirty on Katalon, and based on my experience, User Documentation for Katalon still needs to be updated as sometimes finding the issue is a bit difficult.
  • Language Support:
    • Now unlike Selenium WebDriver that supports multiple programming languages based on the company’s knowledge stack, Katalon heavily relies on one Groovy Programming Language.
    • So just for your UI Automation, you might be working on Groovy language, and your application is written in some other language like Python or Java.
    • Why I feel this as a problem is because you might have one team member working on Groovy scripts, which can increase the dependency on him.
    •  I hope they support more languages in the future. 
  • Data-driven Testing 
    • Data-Driven Testing is possible in Katalon. You can attach a data source file CSV, Excel, or Database Connection.
  • Addons or Plugins:
    • Though there are addons in Katalon, there will be a time you need some specific customize you need to go for the Enterprise version.
  • Small Community: Community Support for Katalon is minimal compared to Selenium.
  • Knowledge Base and Support:
  • The KB articles and support are better for Enterprise version than the Free version. 
  • I hope they can work on support for the Katalon Studio to make it better!  

One piece of advice whenever you are selecting any test automation tool is to look for community support. There will be particular situations in which you need help from the tool team. The next tool is the best in terms of features and support!


  • Test Project is a free automation tool that helps you automate API, Web Application, and Mobile Application.
  • It keeps the complex configuration hidden and so that your primary focus should only be on the Testing of the application.
  • What makes Test Project tool different from Katalon and Postman is that it’s a free cloud platform that can be used to record your UI, Mobile, API Tests on run the tests.
  • Running tests locally or evening on the cloud server is pretty simple. 
  • What are Test Project Agents: 
    • Test Projects Agents keep all the updated Browser versions ensuring your drivers and browsers compatibility issue will never arise.
  • Language Support:
    • Just like Katalon, Language Support is limited to Java. 
  • Continuous Testing
    • You can schedule tests on any browser or device you want. All you need to do is download the Test Project Agent.
    • Achieving Continous Testing with Test Project and any CI tools like Jenkins, Gitlab, Team City is not difficult.
  • Reports:
    • Reports generated compared to Postman or Katalon are much very much detailed in Test Project, and setting up the reports is also not at all tricky.
  • Data-driven Testing 
    • Data-Driven Testing is possible in TestProject. You can attach a data source file (CSV file) to your test/job.
  • Knowledge Base and Support:
  • The KB articles and support are excellent for Test Project. My personal experience. My queries got resolved in 2 hours window.
  • Customization:
    • You can request customization adds-on, too, as it’s a community-based testing tool you can always find the solution for your project.

 That’s it!

My final take away would be that you should try to understand the pros and cons of the tools before you start using one for your automation project.

If you are a start-up or don’t wanna invest a lot in the initial stage for automation, I would advise you to consider Test Project as it eases out a lot of technical glitches Katalon Studio Free version has. 

If you are primary focus would be on the API tests on the initial level, then Postman is your go-to tool.

Note: These suggestions are my suggestions, and I have nothing to do with these tools!

Leave a Comment

Your email address will not be published.