Introduction
While working on
apps.berryintl.com’s web service, it appeared from the Windows Azure project templates that I might need several worker roles because they cycle at different times. One worker needed to cycle every day. The other needed to cycle every five minutes. Since the work and the cycle rate were very different, it felt “right” to isolate each role.
In this post, I’ll show you how I combined multiple worker roles into a single, underutilized Azure web role thereby keeping the concept and design of worker roles while not having to pay extra for them. In order to verify the solution, the project uses Azure Diagnostics Trace statements.
The solution file is available for
download.
Combining Multiple Azure Worker Roles into an Azure Web Role
Currently, I use a small Azure instance and there isn’t enough traffic to justify a separate worker role. I implemented
Wayne’s process
for combining one worker role with a web role a while back. Now I needed to add another worker role to the mix. Wayne wrote another post about
multiple worker roles, however these two articles used slightly different methods in that one uses the Run() method and the other uses the OnStart() method of the WebRole.cs file. I needed to come up with a solution that included both concepts.
Note: Wayne Berry is the founding member of the
31a Project. I won’t cover material from Wayne’s two posts. If you haven’t read them, you should. There is a lot of great insight into Azure that I do not duplicate in this post.
Run()
Combining worker and web roles meant adding a Run() method to the Web Project WebRole.cs file. In the standard WebRole.cs generated from a web role project template in Visual Studio, the Run() method is not present.
Original WebRole.cs
public class WebRole : RoleEntryPoint
{
public override bool OnStart()
{
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
return base.OnStart();
}
}
In
this modified WebRole.cs, the Run() method contains the code to be treated as a worker role. A WebRole class created from a Worker Role template project created by Visual Studio has the Run() method overridden. By adding the Run() method to my web role project I can get both a web role and a worker role in the same instances. I wanted to use the Run() to both be 1) consistent with how the worker roles were treated in the role lifecycle and 2) to separate out what was logically background process from web site functionality.
In the new WebRole class below, notice that the base class is no longer RoleEntryPoint but is ThreadedRoleEntryPoint. ThreadedRoleEntryPoint sits between WebRole and RoleEntryPoint and gives the project the threaded workers but calls the RoleEntryPoint so all the roles can behave as expected.
New WebRole.cs
/// <summary>
/// Manages creation, usage and deletion of workers in threads
/// </summary>
public class WebRole : ThreadedRoleEntryPoint
{
/// <summary>
/// Treated as WebRole Start
/// If the OnStart method returns false, the instance is immediately stopped.
/// If the method returns true, then Windows Azure starts the role by calling
/// the Microsoft.WindowsAzure.ServiceRuntime.RoleEntryPoint.Run() method.
/// </summary>
/// <returns>bool success</returns>
public override bool OnStart()
{
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
Trace.TraceInformation("WebRole::OnStart ", "Information");
// Setup Azure Dignostics so tracing is captured in Azure Storage
this.DiagnosticSetup();
return base.OnStart();
}
/// <summary>
/// Treated as WorkerRole Run
/// </summary>
public override void Run()
{
Trace.TraceInformation("WebRole::Run begin", "Information");
List<WorkerEntryPoint> workers = new List<WorkerEntryPoint>();
// ONLY CHANGE SHOULD BE TO ADD OR REMOVE FROM THE NEXT TWO LINES
// MORE OR LESS ADDITIONS
// WITH SAME OR DIFFERENT WORKER CLASSES
workers.Add(new Worker1());
workers.Add(new Worker2());
base.Run(workers.ToArray());
Trace.TraceInformation("WebRole::Run end", "Information");
}
}
By inserting a class between the original child, WebRole, and the original parent, RoleEntryPoint, we have a place to manage the threaded worker roles.
ThreadedRoleEntryPoint()
The ThreadedRoleEntryPoint contains thread creation, adds one worker to its own thread and runs the workers.
ThreadedRoleEntryPoint.cs
/// <summary>
/// Middle class that sits between WebRole and RoleEntryPoint
/// </summary>
public abstract class ThreadedRoleEntryPoint : RoleEntryPoint
{
/// <summary>
/// Threads for workers
/// </summary>
private List<Thread> threads = new List<Thread>();
/// <summary>
/// Worker array passed in from WebRole
/// </summary>
private WorkerEntryPoint[] workers;
/// <summary>
/// Initializes a new instance of the ThreadedRoleEntryPoint class
/// </summary>
public ThreadedRoleEntryPoint()
{
EventWaitHandle = new EventWaitHandle(false, EventResetMode.ManualReset);
}
/// <summary>
/// Gets or sets WaitHandle to deal with stops and exceptions
/// </summary>
protected EventWaitHandle EventWaitHandle { get; set; }
/// <summary>
/// Called from WebRole, bringing in workers to add to threads
/// </summary>
/// <param name="workers">WorkerEntryPoint[] arrayWorkers</param>
public void Run(WorkerEntryPoint[] arrayWorkers)
{
this.workers = arrayWorkers;
foreach (WorkerEntryPoint worker in this.workers)
{
worker.OnStart();
}
foreach (WorkerEntryPoint worker in this.workers)
{
this.threads.Add(new Thread(worker.ProtectedRun));
}
foreach (Thread thread in this.threads)
{
thread.Start();
}
while (!EventWaitHandle.WaitOne(0))
{
// Restart Dead Threads
for (int i = 0; i < this.threads.Count; i++)
{
if (!this.threads[i].IsAlive)
{
this.threads[i] = new Thread(this.workers[i].Run);
this.threads[i].Start();
}
}
EventWaitHandle.WaitOne(1000);
}
}
/// <summary>
/// OnStart override
/// </summary>
/// <returns>book success</returns>
public override bool OnStart()
{
return base.OnStart();
}
/// <summary>
/// OnStop override
/// </summary>
public override void OnStop()
{
EventWaitHandle.Set();
foreach (Thread thread in this.threads)
{
while (thread.IsAlive)
{
thread.Abort();
}
}
// Check To Make Sure The Threads Are
// Not Running Before Continuing
foreach (Thread thread in this.threads)
{
while (thread.IsAlive)
{
Thread.Sleep(10);
}
}
// Tell The Workers To Stop Looping
foreach (WorkerEntryPoint worker in this.workers)
{
worker.OnStop();
}
base.OnStop();
}
}
Now that the WebRole has a Run() passing an array of different worker roles and the intermediate class deals with the threads, we need to develop each worker role we want to run.
The Worker Role
The WorkerEntryPoint class is the base class for each worker role that needs to be created. The ProtectedRun() method allows any system exceptions to get back up to the WebRole class so that the worker role can be restarted.
Note: The important distinction is that
all worker roles in this sample will stop and restart when the system exception bubbles back up to the WebRole.cs class. In the normal Azure Worker Role practice, where each worker role is in its own assembly, only the troubled worker would be stopped and started.
WorkerEntryPoint.cs
/// <summary>
/// Model for Workers
/// </summary>
public class WorkerEntryPoint
{
/// <summary>
/// Cycle rate of 30 seconds
/// </summary>
public readonly int Seconds30 = 30000;
/// <summary>
/// Cycle rate of 45 seconds
/// </summary>
public readonly int Seconds45 = 45000;
/// <summary>
/// OnStart method for workers
/// </summary>
/// <returns>bool for success</returns>
public virtual bool OnStart()
{
return true;
}
/// <summary>
/// Run method
/// </summary>
public virtual void Run()
{
}
/// <summary>
/// OnStop method
/// </summary>
public virtual void OnStop()
{
}
/// <summary>
/// This method prevents unhandled exceptions from being thrown
/// from the worker thread.
/// </summary>
internal void ProtectedRun()
{
try
{
// Call the Workers Run() method
this.Run();
}
catch (SystemException)
{
// Exit Quickly on a System Exception
throw;
}
catch (Exception)
{
}
}
}
Each Worker
Each worker needs to inherit from WorkerEntryPoint with the code unique to that worker. The two workers in the sample app (Worker1 & Worker2) have a couple of details you could change. The first is the milliseconds passed to Thread.Sleep. This controls the cycle rate of the worker thread. The second change should be adding the
work of the thread. In the sample, the workers each print out trace information to the output window as their
work.
Worker1.cs
/// <summary>
/// Worker1 contains the entire cycle of the worker thread
/// </summary>
public class Worker1 : WorkerEntryPoint
{
/// <summary>
/// Run is the function of an working cycle
/// </summary>
public override void Run()
{
Trace.TraceInformation("Worker1:Run begin", "Information");
try
{
while (true)
{
// CHANGE SLEEP TIME
Thread.Sleep(this.Seconds30);
//// ADD CODE HERE
string traceInformation = DateTime.UtcNow.ToString() + " Worker1:Run loop thread=" + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString();
Trace.TraceInformation(traceInformation, "Information");
}
}
catch (SystemException se)
{
Trace.TraceError("RunWorker1:Run SystemException", se.ToString());
throw se;
}
catch (Exception ex)
{
Trace.TraceError("RunWorker1:Run Exception", ex.ToString());
}
Trace.TraceInformation("Worker1:Run end", "Information");
}
}
Using a Separate Library for the logical Worker Role
When you download the sample application and look at the solution, you will notice that the four files that make up the logical Worker Role are in a separate assembly. This is explained in the comment at the bottom of the
Running Multiple Threads post:
Azure seeks the worker role assembly for the first class that derives from RoleEntryPoint, and tries to load the abstract class in this library.
Do not move the files back into the assembly that has the webrole.cs file. It won’t work.
Viewing the Work
In order to make the sample as simple as possible and yet give you something you can
see, the code prints trace statements. This will let us see evidence of the work.
On the local development box, you can see the trace statements in the Visual Studio Output Window or the Azure Compute Emulator. In order to see those statements in the Azure Cloud, I’ll use Azure Diagnostics to insert each statement into an Azure Storage Table. Once the information is in Azure Storage you can use the Visual Studio Server Explorer to view the storage or a third party tool. My current favorite is
Azure Storage Explorer.
TODO: In the WindowsAzureProject1 project’s
ServiceConfiguration.Cloud.cscfg file, change the value of "
Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" to your own Azure Storage account if you plan to deploy this sample to the Azure cloud.
Using Azure Diagnostics to Verify Deployment
The Azure Diagnostics takes the Trace statements and sticks them in an Azure Storage table named WADLogsTable on a timed cycle. In the image below, you can see four lines of tracing where each worker role is spitting out date, time, and thread number.
If you are running the sample, make sure you give Azure Diagnostics enough time to move the traces to Azure Storage. The sample’s rate is 1 minute for all trace statements however you can configure this. The DiagnosticSetup() method below shows those two specific lines in bold.
DiagnosticSetup() called from WebRole.cs OnStart()
/// <summary>
/// This sets up Azure Diagnostics to Azure Storage.
/// </summary>
private void DiagnosticSetup()
{
// Transfer to Azure Storage on this rage
TimeSpan transferTime = TimeSpan.FromMinutes(1);
DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
// Transfer logs to storage every cycle
dmc.Logs.ScheduledTransferPeriod = transferTime;
// Transfer verbose, critical, etc. logs
dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
System.Diagnostics.Trace.Listeners.Add(new Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener());
System.Diagnostics.Trace.AutoFlush = true;
#if DEBUG
var account = CloudStorageAccount.DevelopmentStorageAccount;
#else
// USE CLOUD STORAGE
var account = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"));
#endif
// Start up the diagnostic manager with the given configuration
DiagnosticMonitor.Start(account, dmc);
}
Verify Resources
The idea of combining worker roles into a web role is that the worker roles don’t take down the web role to the point that a customer or web request can’t be answered in an expected time period. For my worker role that I run (in my real app, not this sample) on a 24 hour cycle, I’m not too concerned about over-working my web role. But the worker rule that I run every five minutes is something I need to take a closer look at. In order to verify the combination of web and worker roles is functioning within the limits I set for the response, I need to make sure there are no failed requests. This information is captured in the
Failed Requests Log. The next step is to take a look at the
performance counters.
Summary
This blog post explains how to combine multiple threaded worker roles into a single web role. It is important to understand this should only be done for deployments that under-utilize the resources of the web role. After you download and alter the sample, verify your web role and worker roles are running as expected and that the resources are not over taxed with the new threaded worker roles.
The solution file is available for
download.
Posts
