Thursday 28 January 2016

Sitecore FakeDB and real world testing

One of the biggest issues in my opinion with Sitecore at present, is that it is relatively hard to unit test anything you build for it.  This is because none of the integral pieces (Item, Database, Security, etc.) have interfaces, are often sealed classes or have no parameter-less constructors, making it extremely hard to mock.  There are ways around this, such as stand-alone mocks Microsoft Fakes, but I have never found this particularly easy to implement and they don't work well ReSharper's test runners.

Obviously components that don't integrate with Sitecore directly (controllers for example) or where perhaps you can use Glass Mapper can be mocked out, to some extent.

I recently came across Sitecore FakeDB (https://marketplace.sitecore.net/Modules/Sitecore_FakeDb.aspx) and wondered if this might be the answer to my problems.

I wondered if we could test something substantial like a pipeline processor?

The code examples below assume that:
  • You are familiar with how to set up the configuration for Sitecore FakeDB (see the documentation tab on the above Sitecore Marketplace link).
  • You are familiar with xUnit (https://xunit.codeplex.com/)
  • You are familiar with mocking  and/or duck casting technologies.
  • You have a basic understanding of Sitecore.
I also used a combination of NSubstitute (http://nsubstitute.github.io/) and Moq (https://github.com/Moq/moq4), as Moq had issues setting up the GetItem<T> method found on Glass Mapper's SitecoreService class.

N.B. Moq kept reporting no such extension method - I don't believe that GetItem<T> is an extension method - but I needed to get going so I used NSubstitute instead.

Brief:

I want to write a test for a pipeline processor, without having to make too many changes to the existing code.

I have an item resolver; an HttpRequestProcessor that looks at the current URL passed via arguments, and makes a decision to redirect to another or not.

All HttpRequestProcessors are passed HttpRequestArgs in a Process method which contains contextual information such as the current URL, etc.

The code:

Here is my process method (the starting point for my pipeline processor):

public override void Process(HttpRequestArgs args)
{

   Assert.ArgumentNotNull(args, "args");
   if (Context.Item != null || Context.Database == null || Context.Database.Name.ToLowerInvariant() == "core" || args.Url.ItemPath.Length == 0)
    {
       return;
    }

   if (args.Url.FilePath.Contains("/sku/"))
    {
        GetSkuItem(args);
    }

   if (!args.Url.FilePath.Contains("/brands/")) return;
   Context.Item = GetBucketItem(args);
}

So the first issue I hit was how to construct the HttpRequestArgs, which is in itself is a sealed class (so cannot be mocked) and is constructed using a combination of both HttpRequest (not base) and HttpResponse (not base) both of which are hard to mock out.

This is what I came up with:

private HttpRequestArgs CreateHttpRequestArgs(string url)
{
  _redirect = new StringBuilder();

  var response = new HttpResponse(new StringWriter(_redirect));
  var request =
      new HttpRequestArgs(
          new System.Web.HttpContext(new HttpRequest("", url, ""),
          response), HttpRequestType.End);

  var dynMethod = request.GetType().GetMethod("Initialize", BindingFlags.NonPublic | BindingFlags.Instance);
      dynMethod.Invoke(request, null);
  return request;
}

The important part here is the small piece of reflection at the end, that calls the private Initialize method on the HttpRequestArgs object, which amongst other things populates the URL on the object itself.

The StringWriter above is passed a StringBuilder, which we will use later. 

Now my test methods can use this to construct the HttpRequestArgs and send it through to the processor.

So, lets start with something really simple, no database (anything else will require us to start using Sitecore FakeDB).

[Fact]
public void Null_Database_Item()
{
  var request = CreateHttpRequestArgs("http://myhost/");

  _bucketItemResolver.Process(request);

  Sitecore.Context.Item.Should().BeNull();
}

N.B. The _bucketItemResolver class is initialized in the constructor for this class

Now lets pass in context item, which should also fail at the first if in the above Process method where it checks for a null context item.

[Fact]
public void A_Context_Item()
{
   using (var db = new Db {new DbItem("Home") {{"Title", "Welcome!"}}})
   {

     var request = CreateHttpRequestArgs("http://myhost/");

     var homeItem = db.GetItem("/sitecore/content/home");
     Assert.Equal("Welcome!", homeItem["Title"]);

     Sitecore.Context.Item = homeItem;

     _bucketItemResolver.Process(request);

     Sitecore.Context.Item.Should().Be(homeItem);
    }
}

In the above code, we construct the Fake Db with a single item.  We set the context to be that item, and when the test is run the Process method checks for a null context and halts any further processing immediately because that condition fails.

A more complex test:

One of the URL checks I have, is one that looks for the following pattern: '/sku/'. 

The processor does the following if it finds a URL pattern match in the GetSkuItem method:
  1. Splits up the file-path to determine if there is a SKU number after the pattern.
  2. Does a product search with SKU against a custom index to find the product.
  3. Uses a link provider to build the URL to that product
  4. Redirects to the new URL
private void GetSkuItem(HttpRequestArgs args)
{
   try
   {
      var filepath = args.Url.FilePath;

      var splitPath = filepath.ToLowerInvariant().Split('/').RemoveWhere(p => p.Equals(string.Empty));
      var pathPaths = splitPath as string[] ?? splitPath.ToArray();
      if (!pathPaths.Any() || pathPaths.Count() <= 1) return;
      var productSearchString = pathPaths[1];
      string redirectUrl;
      var originalUrl = args.Url.FilePath;
      lock (RedirectionCache)
      {
          if (!RedirectionCache.TryGetValue(originalUrl, out redirectUrl))
          {
              var product = ProductsSearchIndex.GetProductBySku(productSearchString);
              if (product != null)
              {
                   redirectUrl = new ProductLinkProvider().GetItemUrl(product.GetItem(),
                     (UrlOptions) UrlOptions.DefaultOptions.Clone());

                   if (!string.IsNullOrWhiteSpace(redirectUrl))
                   //Add new url to the cache.
                   if (!RedirectionCache.ContainsKey(originalUrl))
                      RedirectionCache.Add(originalUrl, redirectUrl);
               }
           }
        }

        if (!string.IsNullOrWhiteSpace(redirectUrl)) // There is a redirection match
        {
            try
            {
                args.Context.Response.RedirectPermanent(redirectUrl, true);
            }
            catch (ThreadAbortException)
            {
               Log.Info(string.Format("Permanent Redirection being applied for {0} to {1}", originalUrl, redirectUrl),
                     this);
            }
            catch (NullReferenceException)
            {
               //Used for testing
               args.Context.Response.Write(redirectUrl);
               throw;
            }
         }
      }
      catch (ThreadAbortException)
      {

      }
      catch (Exception ex)
      {
         Log.Error(ex.Message, ex, this);
      }
} 

Here is the test that I wrote:

[Fact]
public void Its_A_Sku()
{
   var index = Substitute.For<Sitecore.ContentSearch.ISearchIndex>();
   Sitecore.ContentSearch.ContentSearchManager.SearchConfiguration.Indexes.Add("product_master_search_index", index);

   using (var db = new Db
           {
              new DbItem("Home") {{"Title", "Welcome!"}},
              new DbItem("Globals/Import/products/10074-test-toy")
                {
                  {"Name", "10074-test-toy"},
                  {"SKU", "10074"},
                  {"Product Name", "10074 Test Toy"},
                  {"Brand", "{989FA9CC-0522-4C6F-AECD-B11F6AC84CFB}"}
                }
            })
    {
        var homeItem = db.GetItem("/sitecore/content/home");
        Assert.Equal("Welcome!", homeItem["Title"]);

        var request = CreateHttpRequestArgs("http://myhost/sku/10074");

        var searchResultItem = Substitute.For<ProductSearchResultItem>();

        searchResultItem.GetItem()
            .Returns(db.GetItem("/sitecore/content/Globals/Import/products/10074-test-toy"));
        searchResultItem.SKU = "10074";
        searchResultItem.Product_Name = "10074 Test Toy";
        searchResultItem.Name = "10074-test-toy";

        var mockglassProduct = new Moq.Mock<GlassProduct>();

        mockglassProduct.Setup(p => p.TemplateId).Returns(IImported_ProductConstants.TemplateId.Guid);
        mockglassProduct.Setup(p => p.Product_Name).Returns(searchResultItem.Product_Name);
        mockglassProduct.Setup(p => p.Name).Returns(searchResultItem.Name);
        mockglassProduct.Setup(p => p.Brand).Returns(new Imported_Brand{Name = "Test Brand"});

        var mockSitecoreService = Substitute.For<ISitecoreService>();

        mockSitecoreService.GetItem<GlassProduct>(Arg.Any<Guid>())
                           .Returns(mockglassProduct.Object);


        ProductLinkProvider.SitecoreService = mockSitecoreService;

        index.CreateSearchContext()
            .GetQueryable<ProductSearchResultItem>()
            .Returns((new[] {searchResultItem}).AsQueryable());

        _bucketItemResolver.Process(request);

        Sitecore.Context.Item.Should().BeNull();

        _redirect.ToString().Should().Be(string.Format("/brands/{0}/{1}", mockglassProduct.Object.Brand.Name, mockglassProduct.Object.Name));
     }
}

  • I recreate the search index using Sitecore FakeDB.
  • Set up the in-memory database with two items.
  • Mock up the Glass Mapper SitecoreService to return the expected search result item.
  • Set the index to return the correct item from the in-memory DB (make sure the values are correct as linQ to Sitecore will be performed against the item).
  • Execute the processor.
  • And finally check the StringBuilder to see if it has the redirected URL in it.

 I had to make a couple of changes to the existing code to get this test to work. 

  1. I had to add the NullReferenceException catch as there was no way to get the HttpResponse which is used to do the redirect to delegate and do something else instead.  If the processor arguments used HttpResponseBase I would have been able to mock out a delegate method to call. 

    Its possible that the redirect might be able to be done with some other Sitecore Utility methods, but as these would undoubtedly be static, again it would be hard to mock them out.

    I took the opportunity here to populate the StringBuilder with a value - not the neatest but at least I can test a result.

  2. In the ProductLinkProvider class we use the SitecoreService.  Unfortunately this is resolved from DI using Kernel.Resolve (found on a static helper class), so I had to add a property to the class, which I could use in testing, to set up my mocked version of the service and have the code check for a valid service in that property. 

    N.B. If I had had time, I may have been able to rewrite the provider so that the SitecoreService was injected in properly.
Conclusions

All in all I am very happy with Sitecore FakeDb and will be using it to write more comprehensive tests in the future.  I give it two thumbs up!

If in the future Sitecore can make some minor changes with testing in mind, then that would be the icing on the cake.

Wednesday 20 January 2016

Building Lucene Search Index - Order of templates does matter!

I've just spent a frustrating 30 minutes trying to figure out why a search of mine wasn't returning the results I expected.

Here's the json that gets passed to my API that is used to search against the search index:

{"pages":
    ["c159f9f7-77b6-42c2-9b35-50279001987e",
     "8c24e1a1-986e-4c01-9c2f-447b89805fd7",
     "b52b3044-f7b0-406d-b3ac-6be5b8148a7f",
     "f2726c5f-3fbc-4b25-8d53-693c0eb50b93",
     "2319ea68-74c6-4ba3-af7b-02a6b766f311",
     "a70eb547-bab6-44c0-97ca-b749ba333561",
     "2685a1f2-7aac-4e99-ba9d-5b05667140a7",
     "6385cf29-2133-4c70-a247-ec0be744db64",
     "37816c10-fb96-45b6-b123-cb815926df97"
    ],"language":"en","PageSize":10}


In my search I pass in a number of GUIDs (pages) which are matched against Ids in the index.  Sounds simple enough!  I had four templates added to my index configuration, three of which where pages and one (the last in the list) was inherited by all the pages:
       
<include hint="list:IncludeTemplate">
  <!--Product Overview Page--> 
  <templateId>{1A1706B9-95AF-408D-ACF6-A158315333F4}</templateId>
  <!--Product Module Detail Page--> 
  <templateId>{1263F902-B309-47B1-BCE5-25305167E15C}</templateId>
  <!--Product Module Landing Page--> 
  <templateId>{ECA7823D-4BA9-4995-836B-85FA90F00481}</templateId>
  <!--Related Product Details--> 
  <templateId>{57F36C1B-9292-4386-96DB-8B87A2FBBE28}</templateId>
</include>


All worked fine until I added another page template id to the list (which also inherited the same template as the other pages).  I added it as the last item in the list:

         
<include hint="list:IncludeTemplate">
  <!--Product Overview Page--> 
  <templateId>{1A1706B9-95AF-408D-ACF6-A158315333F4}</templateId>
  <!--Product Module Detail Page--> 
  <templateId>{1263F902-B309-47B1-BCE5-25305167E15C}</templateId>
  <!--Product Module Landing Page--> 
  <templateId>{ECA7823D-4BA9-4995-836B-85FA90F00481}</templateId>
  <!--Related Product Details--> 
  <templateId>{57F36C1B-9292-4386-96DB-8B87A2FBBE28}</templateId>
  <!--Alternative Text--> 
  <templateId>{82B30165-9698-4E79-8445-9F2BEF315382}</templateId>
</include>


Now if my search json just included page ids for pages that were of the new page template type then my query would return what I expected.  Even if I added some additional page ids but the unique count per template was less than the count for my new one, it still worked as expected.

But as soon as I sent in a request where there were a larger number of one of the original template types, then the result excluded my new template type.  It didn't matter where in the list I put the Ids for new template type they always got excluded. - Weird or what!

As soon as I changed the order of my templates in the index configuration it all worked as expected:

         
<include hint="list:IncludeTemplate">
  <!--Alternative Text--> 
  <templateId>{82B30165-9698-4E79-8445-9F2BEF315382}</templateId>
  <!--Product Overview Page--> 
  <templateId>{1A1706B9-95AF-408D-ACF6-A158315333F4}</templateId>
  <!--Product Module Detail Page--> 
  <templateId>{1263F902-B309-47B1-BCE5-25305167E15C}</templateId>
  <!--Product Module Landing Page--> 
  <templateId>{ECA7823D-4BA9-4995-836B-85FA90F00481}</templateId>
  <!--Related Product Details--> 
  <templateId>{57F36C1B-9292-4386-96DB-8B87A2FBBE28}</templateId>
</include>


For brevity here is the query I used to return the search results:

  using (var searchContext =
                    ContentSearchManager.GetIndex(RelatedProductsSearchIndexName).CreateSearchContext())
            {
                var query = GetBaseQueryForOtherType<RelatedProductSearchResultsItem>(searchContext, language);

                var predicate = PredicateBuilder.False<RelatedProductSearchResultsItem>();

                var predicateCategory = PredicateBuilder.False<RelatedProductSearchResultsItem>();

                if (relatedProductsRequest.Pages.Any())
                {
                    predicateCategory =
                        relatedProductsRequest.Pages.Select(item => new ID(item))
                            .Aggregate(predicateCategory, (current, id) => current.Or(p => p.ItemId == id));

                    predicate = predicate.Or(predicateCategory);
                }

                query = query.Where(predicate);

                if (relatedProductsRequest.ProductId != Guid.Empty)
                {
                    query = query.Where(p => p.ProductPageID == relatedProductsRequest.ProductId);
                }

                count = query.Count();

                return query.Page(relatedProductsRequest.PageNumber, relatedProductsRequest.PageSize)
                    .GetResults()
                    .Hits.OrderByDescending(h => h.Score)
                    .Select(h => h.Document)
                    .ToList();
            }

Friday 8 January 2016

Using Sitecore to manage external images

We have just gone live with a new site where a large number of the product images are hosted by a third party.  In fact the whole product data set is imported and contains image items that just have a URL to the images themselves.

The issue with these images was that they are massive in size and cannot be loaded into the web site without some sort of re-sizing and compression.  Unfortunately the image hosting site, does not have any functionality that would allow us to resize the image at source - i.e. it is not a DAM.

We wanted to leverage Sitecore to do this for two reasons:
  1. Sitecore has a very robust re-sizing mechanism that is easy to invoke and is familiar to all Sitecore developers.

  2. We wanted the images to be cached on the server - which Sitecore does for all media items by default.
To get Sitecore to process the image, we need to leverage the existing media library functionality.  This is done by using a media library item as the hook or kicking off point for the processor.  We then use that media item's URL where we need to display an external image, and we add some additional parameters to the URL that can then be picked up by a custom image processor.  

In our case we have an image in the media library called CustomContent.  

We have also designed our processor so that if there is no external image, the CustomContent image will be shown instead.



Just to give you an idea of what we are talking about here is an image URL:
/~/media/images/CustomContent.ashx?externalImage=https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ9hdJMNqLxS9LO-wqL5S2la1_D9naH77vul12s10qI8PTzCb10F0qypgWMTyU_xGXI_3jV31vziVwLPGcKmb5vajxeWXhXVmfmqpxIalraqVnpilsJHp4_V-DOzAoNVhXz6xSLUo7VuF1/,27da3a5fe74dc9dc/IMG_0030.JPG&h=180&w=180&id=3f36195e-8dbe-45c5-8ee0-18c2ab130eef&hash=7FC9C5E5044978E99335A9BF4FD3E8B56237E17B

Important points to note:
  1. Extra Parameters including externalImage and id

  2. The image URL can point to any external image
The hash in the URL is added by Sitecore for media request protection, this is a new feature in Sitecore 7.5 and above.  If enabled you will need to register your parameters in the Sitecore config file: Sitecore.Media.RequestProtection.config, and generate the hash when building your image URL.  This means that the URL will then become tamper proof.

In your URL helper / resolver do something like:

HashingUtils.ProtectAssetUrl(
  string.Format("/~/media/images/CustomContent.ashx?externalImage={0}&h={1}&w={2}&id={3}", url,
  source.Height, source.Width, source.Item.ID.ToGuid().ToString("D")));

The HashingUtils class can be found by using Sitecore.Resources.Media.

You will need to add your parameters to the protectedMediaQueryParameters section of the Sitecore.Media.RequestProtection.config file, as follows:


 
So now for the code:

First off the processor has an entry method called Process, in here we accept a single argument parameter of GetMediaStreamPipelineArgs, which we can then examine for our custom parameters:

public void Process(GetMediaStreamPipelineArgs args)
{
    if (args.Options.CustomOptions["externalImage"] != null || 
        args.Options.CustomOptions["id"] != null)
    {
             GetExternalImage(args);
    }
} 

If we find the custom parameters, we can call the main piece of functionality which will get the image re-size and/or compress it and render it through the media request pipeline.

private void GetExternalImage(GetMediaStreamPipelineArgs args)
{
    try
    {
         var client = new WebClient();
         var result = client.OpenRead(new 
            Uri(args.Options.CustomOptions["externalImage"]));

         if (result == null)
         {
                args.AbortPipeline();
                return;
         }
         var bm = CreateBitMapFromStream(result);
         var quantizer = new WuQuantizer();
        if (args.Options.Width != 0 && args.Options.Height != 0)
         {
                 using (var newImage = ScaleImage(bm, args.Options.Width, 
                           args.Options.Height))
                 {
                     using (var quantized = quantizer.QuantizeImage(new Bitmap(newImage)))
                     {
                         var stream = new MemoryStream();
                         quantized.Save(stream, ImageFormat.Png);
                         args.OutputStream = new MediaStream(stream, "png", 
                              args.MediaData.MediaItem);
                     }
                 }
          }
          else
          {
                  using (var quantized = quantizer.QuantizeImage(bm))
                  {
                      var stream = new MemoryStream();
                      quantized.Save(stream, ImageFormat.Png);
                      args.OutputStream = new MediaStream(stream, "png", 
                          args.MediaData.MediaItem);
                  }
          }
       }
       catch (Exception ex)
       {
           Log.Error(ex.Message, ex, this);
           args.AbortPipeline();
       }
}

We use WuQuantizer to do the png compression which does a better job than .net's native compression.

https://www.nuget.org/packages/nQuant/ 

Just for completeness here are the two private methods referenced in the above:

private static Bitmap CreateBitMapFromStream(Stream result)
{
    byte[] imageBytes;
    using (var ms = new MemoryStream())
    {
        var count = 0;
        do
        {
            var buf = new byte[1024];
            count = result.Read(buf, 0, 1024);
            ms.Write(buf, 0, count);
        } while (result.CanRead && count > 0);
        imageBytes = ms.ToArray();
    }

    // Now you can use the returned stream to set the image source too
    var image = new MemoryStream(imageBytes);
    var bm = (Bitmap) Image.FromStream(image);
    return bm;
}

public static Image ScaleImage(Image image, int maxWidth, int maxHeight)
{
    var ratioX = (double) maxWidth/image.Width;
    var ratioY = (double) maxHeight/image.Height;
    var ratio = Math.Min(ratioX, ratioY);

    var newWidth = (int) (image.Width*ratio);
    var newHeight = (int) (image.Height*ratio);

    var newImage = new Bitmap(newWidth, newHeight);

    using (var graphics = Graphics.FromImage(newImage))
        graphics.DrawImage(image, 0, 0, newWidth, newHeight);

    return newImage;
}

Finally implementing the processor, we need to add our processor to the getMediaStream pipeline, and we do that we creating the following config file:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <getMediaStream>
        <processor patch:after="processor[@type='Sitecore.Resources.Media.GrayscaleProcessor, Sitecore.Kernel']"
            type="MyProject.Library.Processors.CustomImageProcessor, MyProject.Library" />
      </getMediaStream>
    </pipelines>
  </sitecore>
</configuration>

Saturday 2 January 2016

Testing in software development and parallels with the real world

I have often thought that we can see many parallels to software development in other professions and walks of life.

The other day we had a plumber come out to our house to fit a water purification system (in the crawl).  At some point during the install he mentioned that he would need to fit a pressure reducing valve as the water pressure in the house was too high for his equipment - for the sake expediency I agreed.  At the time I was surprised that after completing the install he never once set foot in the house.  He handed over his invoice and went on his way, the only thing he said was that if there were any issues please call the office.

A little while later I went to run a faucet (tap) in the house and was surprised (but with hindsight perhaps not really!) that the water came out with a dribble.  I tried the showers and again the pressure was so low that there was no way you could wash under flow of water.  I called their office, and the because the plumber was on another job, the owner had to come (1 hour round trip) spend 15 minutes adjusting (and testing!) to finally get it right.  

What struck me here, was that this was not the first time that this pattern of events had happened to me when calling out a trade to the house.  I am also sure after speaking to friends that this has happened frequently for others as well.  I later found out there are regulations that lay out the kind of tests a plumber should be performing - most importantly to ensure that someone doesn't get scolded!  So why were these not followed?  Do they need to be scripted out?

I suspect that the plumber had done this so many times, that he felt that his assumptions were as good as doing the tests.  Sound Familiar? - But as we know assumptions are the mothers of all mistakes!*

But I know from experience that if I have a set tests laid out before me that require a single click to run, I will use them, and more often than not I will add to and update them as and when necessary.  I also know that developers shy away from writing tests, often siting that it takes too much time.  But is that really their call? Feels to me that we need a culture shift...

The implications for my plumber, also apply to developers:
  1. Customer/end-user gets frustrated/annoyed and may not use them again
  2. Business has additional expense to fix the issue
  3. Business owner/key staff are taken away from running his business
  4. Legal implications if injured, and so on.
I have come across many businesses where the software development teams behave just like my plumber.  They churn out code with little or no testing with a subconscious expectation that QA or (more expensively) the business will test their code.   They make assumptions about how things should operate and rarely have any concrete evidence (unit tests) that their code does in fact work!  

Any business that builds software and relies solely on QA/UAT as their only line of defense will end up with a high turn over of senior staff, software that doesn't perform/doesn't work period/fails frequently, and lazy programmers.

If you get:
  • Your software developers to spend the extra time writing unit tests.
  • Implement a CI platform
  • Have a UAT environment that as closely as possible matches production 
  • Have developers/deployment teams prepared to smoke test deploys based on a set of scripts.
  • Have scripts prepared for UAT
Then:
  • Issues will be caught earlier in the process
  • UAT becomes an easy step-by-step process - a sign off.
  • You reduce the cost to the business 
  • You reduce the strain on the most important people in your business.
  • And more importantly less failures once the code is in production
*Paraphrasing the Marcus Penn (Everett McGill) from the movie Under Siege 2