useful Linux commands while developing rails web applications

Rails application best optimization concepts

Optimization is MUST of a Rails application specially for the high traffic Ruby On Rails application and for good UX. Here are some optimization tips which I already applied in my recent Rails projects.


Optimization of Rails Application and application server

• Apply caching(Object cache, fragment_cache, action_cache, page_cache) and use memcached server instead of disc for storing cached data. You can get Rails caching idea from here http://guides.rubyonrails.org/caching_with_rails.html
• If you have complex searching functionality into your project then use search engine like sphinx or solr instead of doing sql like search
• If you use passanger for application server, try to use passenger pool size equal to your server's number of CPUs(cores), for more, see from here http://www.modrails.com/documentation/Users%20guide%20Apache.html#_passengermaxpoolsize_lt_integer_gt
• You could use Amazon S3 to host your assets(including images,compressed javascript and css)
• There is a great ruby gem for Rails https://github.com/AssetSync/asset_sync to sync your assets after immediate deploy.
• You could also use CDN(Content deliver network) for static resources(images,videos, pdf,css,js, etc) to give location wise resource delivery.
• If CDN looks costly then use rails assets host(it is not alternative of CDN) for concurrent static resources download, for more info about asset host check this link http://api.rubyonrails.org/classes/ActionView/Helpers/AssetTagHelper.html
• Use eager loading to avoid N+1 query problem. see from here
  http://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations
• You could use background job(like Resque, Sidekiq) to process things asynchronously. For instance, after a user gets registered, you want to send email via sendgrid/mailchimp/AWS SES, simply execute email sending code inside a worker process to give user faster UX.
• To process big data set, check this link for parallel processing with fork
  http://abdul-barek-rails.blogspot.com/2012/02/parallel-processing-in-ruby-on-rails.html

Optimization of web server

• Use nginx instead of Apache for better static resources(images,videos,css,javascript,text, html, etc) response
• For static resources cached on client side for specific time(for example, 30 days), use this configuration

  	For Apache, use the Location directive into your virtual host(be sure you have included expires and headers modules)
  	Header unset ETag
  	FileETag None
  	ExpiresActive On
  	ExpiresDefault "access plus 30 days"

For nginx location ~ ^/assets/ { expires 30d; add_header Cache-Control public; add_header ETag ""; break; }

view rawgistfile1.txt hosted with ❤ by GitHub

Make sure your web server is responding with gzip data

If your application faces heavy traffic then use load balancing. In one of my project, I used nginx as balancer on front end and 4 apache web servers on backend nodes. If you use cloud server like rackspace then you can use rackspace's builtin load balancer with your node servers or you can create your own load balancer

Use http reverse proxy like varnish cache server so that most of the requests are being served by the cache server instead of hitting your application server, see this link for varnish configuration
http://abdul-barek-rails.blogspot.com/2012/07/integrate-http-reverse-proxy-cache.html

You can use http cache but be careful, its risky! because already requested requests(from a client) will not hit your application server(because of cache validation time), instead, the client will get response from its local cached data. To avoid this, you can change url.

Client Side Optimization

• Try to use more Asynchronous requests using AJAX for better user experience and responsiveness in your application where you can.
• Try to reduce number of requests which is the key optimization on client side
• On production environment, use Rails asset pipeline feature so that your JavaScripts and css files are being combined and minified or compressed
• Use css stripe to avoid loading more images(same type of images) so that you can reduce number of http requests for images
• Try to reduce static resources weight(reduce size in KB or Byte)
• You can use jquery lazy load plugin to load images lazily or you can use jquery appear plugin to load images/contents on when screen appears to view
• To measure your application's load time and performance, use this google tool
  https://developers.google.com/speed/pagespeed/insights
• For better inspection of your server's resources on browser, use firefox's firebug adons in firefox browser and see how much time all resources are taking to load and see how weight of resources are and see what are the unnecessary http requests and then reduce resources weight and load time and reduce unnecessary http requests as you can.
  

Database Optimization

• Apply database indexing on table's those columns which are being used for join, foreign key, ordering or on where clause
• Apply data reporting, I mean instead of pulling big data result set on the fly, prepare them before they are being used, you can use any scheduler gem like rufus scheduler or delayed job to make stuff pre-ready.
• Inspect slow query from your database server log file and optimize it

You can use NewRelic for details servers logs, graphs, database logs and many many stuff to analyze more.

Rails application best optimization concepts

Optimization is MUST of a Rails application specially for the high traffic Ruby On Rails application and for good UX. Here are some optimization tips which I already applied in my recent Rails projects.


Optimization of Rails Application and application server

• Apply caching(Object cache, fragment_cache, action_cache, page_cache) and use dedicated separate Redis, memcached server for storing cached data. You can get Rails caching idea from here http://guides.rubyonrails.org/caching_with_rails.html
• You could use background job(like Resque, Sidekiq) to process things asynchronously. For instance, you wanna process payment through Paypal/Stripe/Braintree API then execute the corresponding payment related code inside a separate worker process asynchronously.
• You could use Amazon S3 to host your assets(including images,compressed javascript and css)
• There is a great ruby gem for Rails https://github.com/AssetSync/asset_sync to sync your assets with AWS S3 after immediate deploy.
• If you have complex searching functionality into your project then use search engine like sphinx or solr instead of doing sql like search
• If you use passanger for application server, try to use passenger pool size equal to your server's number of CPUs(cores), for more, see from here http://www.modrails.com/documentation/Users%20guide%20Apache.html#_passengermaxpoolsize_lt_integer_gt
• You could also use CDN(Content deliver network) for static resources(images,videos, pdf,css,js, etc) to give location wise resource delivery.
• If CDN looks costly then use rails assets host(it is not alternative of CDN) for concurrent static resources download, for more info about asset host check this link http://api.rubyonrails.org/classes/ActionView/Helpers/AssetTagHelper.html
• Use eager loading to avoid N+1 query problem. see from here
  http://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations
• To process big data set, check this link for parallel processing with fork
  http://abdul-barek-rails.blogspot.com/2012/02/parallel-processing-in-ruby-on-rails.html

Optimization of web server

• If your application faces heavy traffic then use load balancing. In one of my project, I used nginx as balancer on front end and 4 apache web servers on backend nodes. If you use cloud server like rackspace then you can use rackspace's builtin load balancer with your node servers or you can create your own load balancer
• Using AWS Load Balancer for many EC2 instances is also a great way to divide traffic.
• AWS Autoscaling is also an AWESOME automated way of scaling your app based on your traffic
• Use nginx instead of Apache for better static resources(images,videos,css,javascript,text, html, etc) response
• For static resources cached on client side for specific time(for example, 30 days), use this configuration
• Make sure your web server is responding with gzip data
• Use http reverse proxy like varnish cache server so that most of the requests are being served by the cache server instead of hitting your application server, see this link for varnish configuration
  http://abdul-barek-rails.blogspot.com/2012/07/integrate-http-reverse-proxy-cache.html
• You can use http cache but be careful, its risky! because already requested requests(from a client) will not hit your application server(because of cache validation time), instead, the client will get response from its local cached data. To avoid this, you can change url.

Client Side Optimization

• Try to use more Asynchronous requests using AJAX for better user experience and responsiveness in your application where you can.
• Try to reduce number of requests which is the key optimization on client side
• On production environment, use Rails asset pipeline feature so that your JavaScripts and css files are being combined and minified or compressed
• Use css stripe to avoid loading more images(same type of images) so that you can reduce number of http requests for images
• Try to reduce static resources weight(reduce size in KB or Byte)
• You can use jquery lazy load plugin to load images lazily or you can use jquery appear plugin to load images/contents on when screen appears to view
• To measure your application's load time and performance, use this google tool
  https://developers.google.com/speed/pagespeed/insights
• For better inspection of your server's resources on browser, use firefox's firebug adons in firefox browser and see how much time all resources are taking to load and see how weight of resources are and see what are the unnecessary http requests and then reduce resources weight and load time and reduce unnecessary http requests as you can.
  

Database Optimization

• PostgreSQL database is better than MySQL for large data sets
• You could use AWS RDS to hold data for better latency and data backups
• Apply database indexing on table's those columns which are being used for join, foreign key, ordering or on where clause
• Apply data reporting, I mean instead of pulling big data result set on the fly, prepare them before they are being used, you can use any scheduler gem like rufus scheduler or delayed job to make stuff pre-ready.
• Inspect slow query from your database server log file and optimize it

You can use NewRelic for details servers logs, graphs, database logs and many many stuff to analyze more.

integrate http reverse proxy cache server with Rails and Varnish

• Setup Apache and make a virtual host to listen it on other port like 8080
• Install varnish and listens it on port 80
• For the Varnish, you need to do configuration on two files like bellow

Here is the requesting flow: Browser ---> Varnish---->Apache2----->Application Server
That means, when u request through browser, it request to Varnish(on port 80) and if varnish has cached then it served directly to client(browser) without interfering apache otherwise Varnish passes the request to apache...In this way you can scale your Rails app heavily and serve a lot of requests concurrently with low memory and low resource consumed.

Some varnish useful commands:

• Restart Varnish - /etc/init.d/varnish restart
• Show Varnish statistics - varnishstat
• Test to see response is coming through Varnish -
  curl -I http://www.your_app_url.com

Parallel Processing in Ruby On Rails

I used ruby's fork when I was sending 60,000 emails from one of Rails applications and it is wise to split the big result set and send emails of each chunk parallely on forks for faster processing! Here is a sample ruby code for processing N length data.

I tested it on Ubuntu 11, ruby 1.9.2, Rails 3.0.10

argument out of range and 0.0 issue on rails

Argument out of range Exception on windows7 with Rails3 and model fields class type is float(0.0). Actually I was trying to use mysql2 gem on windows7 in Rails3.

How I fixed it in my case:

• My system was x86 typed(windows7) and I had 32 bit mysql client installed in my machine with Rails 3.1 and Ruby 1.9.2
  
• I downloaded libmysql.dll (32 bit) and placed it on windows/system32 and ruby/bin folder
• I restarted my laptop and FIXED!

This is nothing but libmysql.dll version related issue with system!

Dynamic Image from text/string in Rails

Image will be generated dynamically from string or text:

Use - require 'RMagick' in your class



Pre-requisite : You must have rmagick and Imagemagick installed.
Details: http://www.simplesystems.org/RMagick/doc/draw.html

How to develop rails plugin from scratch?

Generate Plugin with command: ruby script/generate plugin hello_world
It will create file system as bellow:
- lib
    - hello_world.rb
- tasks
    - hello_world_tasks.rake
- test
    - hello_world_test.rb
- init.rb
- install.rb
- uninstall.rb
- README
- Rakefile
- MIT-LICENSE

init.rb will be executed every time when your application runs. Generally hook code is to be included here like you want to make all methods of your plugin available in your app’s models, controllers, views and helpers

Example:
#All methods in module HelloWorld will be available in all #model’s object
ActiveRecord::Base.class_eval do
    include HelloWorld
end

#All methods in module HelloWorld will be available in all controllers
ActionController::Base.class_eval do
    include HelloWorld
end

#All methods in module HelloWorld will be available in all views and all helpers
ActionView::Base.class_eval do
    include HelloWorld
end

lib/ hello_world.rb :

Example:
# All methods in this library will be available in all models, controllers, views and helpers if you write code as init.rb above

module HelloWorld
    def say_hello
        return "Hello World"
    end
    def hello_text
        return “this is text”
    end
end

Now you need to do unit test your plugin’s methods. To do so follow:
test/hello_world_test.rb:

Example:
require 'test/unit'
require File.join(File.dirname(__FILE__),'../lib/hello_world.rb')

class HelloWorldTest < Test::Unit::TestCase
    include HelloWorld # includes your library methods to test

    def test_this_plugin
        hello_world = say_hello
        assert_equal hello_world, "Hello World"
        assert_not_nil hello_world
    end

    def test_another
        assert_equal true,true
    end
end

To run these test cases, go to your plugin’s directory with command prompt and run like: ruby test/hello_world_test.rb

You are done actually with plugin!

Install.rb: It will be executed only once when the plugin is being installed.

Uninstall.rb: This will be executed when you do uninstall your plugin

README: In this file, you should write easy documentation of your plugin with examples. Also you may highlight yourself here.