Deploying Shiny Server on Amazon EC2

Setting up Shiny Server on Amazon’s Elastic Compute Cloud (EC2) can be a bit of a process if you’re unfamiliar with Linux system administration. Thankfully, we’ve done all the hard work for you! If you’re looking to get Shiny Server running in EC2 without delay, just boot up an instance of our public Amazon Machine Image (AMI): ami-e2a3358b named “ShinyServer.”

This is no longer the supported way to deploy Shiny Server. See www.rstudio.com/shiny/server/ to download an installer that will handle this complexity for you.

If you want to recreate this image yourself, you can follow these steps, but you’ll need a running instance on EC2 first. If you’re unfamiliar with EC2, read up on how to create and access EC2 instances here. Go do that now.

Welcome back! I’ll now assume that you’re able to create and access EC2 instances. For this tutorial, you’ll likely want to create a fairly powerful instance — the process of building R and Node.js from scratch can take 20-30 minutes even on powerful hardware — it could take hours or even be impossible due to memory constraints on a smaller instance. Of course, it should only take an hour or two to get everything installed. At that point, you can create an image of your system and later boot it up with whatever size instance you care to use.

Note that Amazon EC2 instances use volatile storage — this means that any changes you make to the system (including changes written to the hard drive) will be lost the moment you terminate your instance! Cloud instances are treated as disposable and are no more reliable than a physical server (i.e. can crash at a moment’s notice).

Once you’ve created a system you’d like to keep, be sure to create an image of that system so that you can boot up other instances of that system at a later date. If you’re new to EC2, take a few minutes to play around with this practice before diving into this guide — I don’t want to have the hours you lost to an EC2 instance that vanished weighing on my conscience…

Go ahead and create a Large (or bigger) instance and log in. Once we’ve logged in, it’s always a good idea to ensure that we’re using the latest packages Amazon provides. We can update with

sudo yum update -y

Install R & Shiny

We can now install the packages we’ll need in order to be able to build R from source:

sudo yum install gcc gcc-c++ gcc_gfortran readline-devel make texinfo \
   libX11-devel libX11-devel.* libICE-devel.* libSM-devel.* libdmx-devel.* \
   libx* xorg-x11* libFS* libX* tetex mesa-libGLU-devel libjpeg-devel \
   libpng-devel cairo-devel pango-devel -y

Once we have those packages, we’re ready to build R. Download the latest version available from the R Project home page. At the time of writing, that’s 2.15.2 which we can download and untar using the following commands:

wget http://cran.rstudio.com/src/base/R-2/R-2.15.2.tar.gz
tar -xvf R-2.15.2.tar.gz
cd R-2.15.2

We’re now in the untar’d directory containing the source code for R. We can configure it to not expect X, since we’re running on a headless server, then compile it using make:

./configure
make
make check   # technically optional

Depending on the size of instance you chose, this could take quite some time. On a large instance, configuration took a minute or two and compiling using make took 20 minutes. The check process took only 2-3 minutes . If you chose a larger EC2 image which has more than one core and sufficient memory to run both builds simultaneously (“Large” or above), you can actually begin the compilation process of Node.js (see below) so the longest steps run concurrently. Now go get some tea.

If everything was able to build correctly, then you can install R to the regular locations using

sudo make install

Confirm that R installed successfully using

R --version

which should return the version of R you downloaded and installed. We’re now ready to install the Shiny package from CRAN. To install the shiny package into R globally across the system, run the following command:

sudo su --c "R -e \"install.packages('shiny', repos='http://cran.rstudio.com/')\""

to install the Shiny package. It make take a couple of minutes to grab the package and all of its dependencies. When that’s ready, you can exit R so we can  proceed with setting up the Node.js environment.

Setup Node.js

Node.js requires Python in order to install properly; we haven’t installed that yet, so we’ll need to:

sudo yum install python27 -y

Shiny Server expects Node.js with a version of at least 0.8.16. The latest stable release at the time of writing is 0.8.19 which can be downloaded from GitHub to assemble from source.

wget https://github.com/joyent/node/archive/v0.8.19.tar.gz
tar -xvf v0.8.19.tar.gz
cd node-0.8.19

Now you’re ready to configure and compile Node.

./configure
make

Just like R’s compilation, the make step will take some time. On a large instance, it took about 10 minutes . If it was able to finish compiling without errors, go ahead to install the binaries in the usual spots:

sudo make install

You should now be ready to go with Node!

Install Shiny Server

We’ll first need to grab the latest code for Shiny. At this point, RStudio’s just keeping it on GitHub, so we’ll install git and clone the repository. If you’re opposed to installing git, you could just download a version of the code using wget.

sudo yum install git -y

Now we can clone their repository and begin the installation process:

git clone https://github.com/rstudio/shiny-server.git
sudo npm install -g shiny-server

It’s possible that you’ll get an error of “npm: command not found” because npm isn’t on the root user’s path. If this happens, just find npm and run the command after providing the explicit location. For instance, mine looked like this:

[ec2-user@ip-1-1-1-1 ~]$ sudo npm install -g shiny-server/
sudo: npm: command not found
[ec2-user@ip-1-1-1-1 ~]$ whereis npm
npm: /usr/local/bin/npm
[ec2-user@ip-1-1-1-1 ~]$ sudo /usr/local/bin/npm install -g shiny-server

which should reoslve the issue. You now have an Amazon image with Shiny Server installed! There are just a few minor steps to get everything working with the default configuration.

Shiny Server will attempt to run as the user named “shiny” by default. So we’ll need to create that user and the directories Shiny Server will attempt to use by default for logging and web files.

sudo useradd -r shiny
sudo mkdir -p /var/shiny-server/www
sudo mkdir /var/shiny-server/log

Note that you can omit these steps if you plan to configure Shiny Server to run as a different user or use different directories. If you plan to customize the configuration, then copy the provided configuration file to the expected system location:

sudo mkdir -p /etc/shiny-server/
sudo cp shiny-server/config/default.config /etc/shiny-server/shiny-server.conf

Then make whatever configuration changes you’d like to the file at /etc/shiny-server/shiny-server.conf (You can view all of the various configuration options online at https://github.com/rstudio/shiny-server/blob/master/config.html).

You’re now ready to test Shiny Server by deploying a sample application. You can use an application of your own or use some of our sample apps.

cd /var/shiny-server/www
sudo git clone https://github.com/trestletech/shiny-sandbox.git .

(Don’t forget the “.” at the end of line 2.) You should now have a “sampleapp” application, among others, within Shiny Server’s jurisdiction. Start up Shiny Server and make sure everything comes online properly.

sudo shiny-server

This may not work, again due to path issues (“shiny-server command not found” or “node: command not found”). It’s probably easiest to just add whatever path is missing to the PATH variable explicitly for now.

For instance, if “whereis shiny-server” produced a location of “/usr/local/bin/shiny-server”, you’d want to add “/usr/local/bin” to the PATH variable via:

sudo -i
export PATH=\$PATH:/usr/local/bin
#to do this automatically when you login as root
echo "export PATH=\$PATH:/usr/local/bin" >> ~/.bashrc 
shiny-server

Now that shiny-server and node are on your path, you should be able to start the server and see something like the following:

[2013-02-08 03:38:07.115] [INFO] shiny-server - Shiny Server v0.3.0 (Node.js v0.8.19)
[2013-02-08 03:38:07.117] [INFO] shiny-server - Using config file "/usr/local/lib/node_modules/shiny-server/config/default.config"
[2013-02-08 03:38:07.202] [INFO] shiny-server - Starting listener on 0.0.0.0:3838

Which shows that, if you haven’t changed the configuration file, shiny will listen for connections from any machine on port 3838. You can try visiting your Amazon instance on port 3838 by looking up its public DNS (the same one you used when SSHing into the machine) and visiting the URL of the format "http://<your amazon machine name>:3838/" which will show a list of all deployed applications on your Shiny Server. You can click any of them to visit them. Note that, if you’re using our sample apps, there are some additional packages and configuration required to use some of these applications — try “sampleapp” to start off with.

If you are having trouble connecting, first check that your Amazon EC2 Instance is accessible on port 3838 universally (see this site for more details). Also check the shiny-server is currently running on your server.

Start on Boot

The final step is optional, but if you want shiny-server to start automatically when the machine boots up, then you can use the Upstart system in order to make the server start on boot and also refresh itself if it ever crashes. An upstart script comes with Shiny, but some modifications are required in order to make it work with the default Amazon Linux AMI. Below is the file that is working on my instance:

# shiny-server.conf

description "Shiny application server"

#start on stopped networking

start on stopped rc RUNLEVEL=[S3]
stop on runlevel [016]

limit nofile 1000000 1000000

exec /usr/local/bin/shiny-server >> /var/log/shiny-server.log 2>&1
respawn

Copy this text into a file saved at “/etc/init/shiny-server.conf” Then reboot the image to see if shiny-server restarts itself when the machine boots.

sudo reboot

If you’re still able to access you Shiny applications, then you should be in good shape. Remember that Amazon EC2 instances use volatile storage, so all of the files on the hard drive will disappear the moment you terminate this image! If you were hoping to save all of this work to use again later, you can create an AMI of your instance to be able to boot up more instances like this one at a later date. See this page for help dealing with Amazon AMIs.

 

Happy Shiny hosting!

 

20 Comments

  1. mahzad says:

    Hi

    I have a Question : Is ami-e2a3358b (shinyserver)free??

    I am using free usage tier in EC2. Is it possible I setup This Shiny server in My VM (worksatation software) in Local Computer?

    What is advantage of instance of Amazon and My VM?

    • Jeff Allen says:

      Hi Mahzad. Yes and no… The image is free to use, in that you won’t get any bill from me for using this image. But any resources you consume on Amazon are eligible to be billed if you exceed their free usage tier. So if you ran this image as your one Micro server and stayed beneath their usage requirements for the free tier, you should be good.

      You could certainly setup a local VM with this software by following the instructions at the bottom, though you may have to modify them slightly to make them work for your distribution (try using Ubuntu, if you want the most painless experience with installing Shiny Server). I’ve never done this myself, but it is possible to import or export VMs from Amazon to a local instance: http://aws.amazon.com/ec2/vmimport/.

      The main advantage of using the Amazon cloud would be immediate scalability. Using this image (or an image you create similar to this one), you could start up a server with dozens of GB of RAM and multiple cores to run this software within minutes — or 10 such images all at once. Also, if you’re hosting any other data/services on EC2, it may be easier to integrate them in if the server’s running there. The obvious advantage of running it on your own server would be that no one’s going to charge you for running the VM (presumably). So if you have a sufficiently powerful workstation and don’t need to share these apps with anyone in a stable/reliable way, you could certainly get away with just doing it on a VM on your own machine.

      The main

  2. Bin W. says:

    Hi, You have done a great job writing this tutorial!!!
    I have a concern that how can you implement the authentication for a shiny server. Let’s say that we have some inside info to put on the shiny server and I don’t want everyone to see it unless they type in password and username. Thanks a lot!

    • Jeff Allen says:

      Thanks, Bin.

      Unfortunately, that’s not a feature that Shiny Server implements just yet, though they are planning that for their commercial/enterprise version. There have been discussions (such as this one) about how you might accomplish getting user authentication into Shiny Server, but it involves hacking the code.

      If you’re just looking for basic authentication, you can try something like this if you’re not looking to get into the node.js code.

  3. nimmi says:

    This is not visible for me . I tried searching this image in amazon

    • Jeff Allen says:

      Sorry for the delayed response — I just commented on this in the post below. I’ll let you know what I figure out!

    • Jeff Allen says:

      I added a link above, but you should be able to find it here.

      Not sure why it doesn’t show up in the Marketplace searches currently…

  4. Paul says:

    Is the AMI still public? I cannot seem to find it.

    • Jeff Allen says:

      How strange. It’s still listed as public from my end, but I’m not able to find it in the Marketplace, either. It looks like a few other people have experienced this (per Google) so maybe there’s a bug? I’ll try to contact them and see what I can get.

      I also tried to copy the original AMI. I’m told the new ID will be “ami-3da6cb54″ and it’s called “ShinyServer2″ when it becomes available. I’ll let you know what I find!

    • Jeff Allen says:

      I added a link above, but you should be able to find it here.

      Not sure why it doesn’t show up in the Marketplace searches currently…

  5. niekto says:

    gcc-gfortran instead gcc_gfortran in CentOS 5.6

  6. Mike F says:

    I tried using your AMI file, but when I start up shiny-server, I get an error. Any idea what I need to modify?

    [2013-07-11 22:48:11.916] [INFO] shiny-server – Shiny Server v0.3.0 (Node.js v0.8.19)
    [2013-07-11 22:48:11.917] [INFO] shiny-server – Using config file “/etc/shiny-server/shiny-server.conf”
    [2013-07-11 22:48:11.955] [INFO] shiny-server – Starting listener on 0.0.0.0:3838
    [2013-07-11 22:48:11.960] [ERROR] shiny-server – HTTP server error (0.0.0.0:3838): listen EADDRINUSE
    [2013-07-11 22:48:11.961] [INFO] shiny-server – Shutting down worker processes

    • Mike F says:

      I figured out that I didn’t have port 3838 open on my AWS Instance. Problem solved

      • Tyler H says:

        I keep getting the same EADDRINUSE error and I cannot resolve it. Do you have a more descriptive solution by chance?

  7. Joe H says:

    I would like to launch your AMI in the EU region. Have you made this public on S3? If so can you share the address? I am quite new to all of this but the only other option I can think of is to
    1. launch it in US
    2. take a snapshot
    3. Move to EU using EC2 AMI copy

    Any other suggestions welcome!

    • Jeff Allen says:

      Hi Joe.

      I haven’t tried launching in other regions, so your best bet may be to copy the AMI out. Please let me know if you have any success with this, as I’ll be sure to pass it on to others!

      Jeff

  8. Alan says:

    Hi Jeff,

    I get no supported authentication methods available. I created an instance with your AMI without any keys. What is the procedure/credentials for setting up the instance to login?
    Sorry if I’ve missed this in your notes.

  9. SB says:

    Hello,
    Thank you for the elaborate instructions on shiny server install on ec2.
    Being a complete novice in Linux however, I am facing some issues when starting shiny server.
    I am getting this error:

    [2014-01-04 ip] [INFO] shiny-server – Shiny Server v0.3.0 (Node.js v0.8.19)
    [2014-01-04 ip] [INFO] shiny-server – Using config file “/usr/local/lib/node_modules/shiny-server/config/default.config”
    [2014-01-04 ip] [INFO] shiny-server – Starting listener on 0.0.0.0:3838
    [2014-01-04 ip] [ERROR] shiny-server – HTTP server error (0.0.0.0:3838): listen EADDRINUSE
    [2014-01-04 ip] [INFO] shiny-server – Shutting down worker processes

    I have opened the 3838 port on EC2 but the error still continues.

    I also tried lsof -i :3838
    but it gives no list.

    Can you please guide me to correct path?

    Thanks,
    SB

  10. Spencer Boucher says:

    Maybe put an update at the top of the page that mentions that the Node way of doing this is now deprecated?

    • Jeff Allen says:

      Thanks; great point. I just added it.

Trackbacks/Pingbacks

  1. Graphical Tools (rgl) on a Headless Shiny Server | Trestle Technology - [...] on Amazon EC2. We have the base Shiny Server installation on Amazon Linux (described thoroughly in our previous post), ...

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">